Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 5 out of 60 is failed (threshold=3) [............................................XXXXX...........] duration: 1.836635594s FAIL (2025-08-07 23:25:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== d. err: <nil> Aug 7 23:07:52.874987115 (13h27m52.464332102s) [upload=13971]: Starting upload Aug 7 23:07:52.875039626 (13h27m52.464384603s) [upload=13971]: Done starting upload Aug 7 23:07:52.875078921 (13h27m52.464423908s) [upload=13971]: Uploading segments... Aug 7 23:07:52.875111583 (13h27m52.46445657s) [upload=13971]: Got next segment. Inline: true Aug 7 23:07:52.875115000 (13h27m52.464459987s) [upload=13971]: Waiting for error group managing segments... Aug 7 23:07:52.875144226 (13h27m52.464489203s) [upload=13971]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:52.875147783 (13h27m52.46449277s) [upload=13971]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:07:52.875150268 (13h27m52.464495255s) [upload=13971]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:53.152523417 (13h27m52.741868404s) [upload=13971]: Done uploading segments... Aug 7 23:07:53.152567841 (13h27m52.741912828s) [upload=13971]: Upload finished. err: <nil> Aug 7 23:08:13.160315277 (13h28m12.749660264s) [upload=13978]: Starting upload Aug 7 23:08:13.160372306 (13h28m12.749717293s) [upload=13978]: Done starting upload Aug 7 23:08:13.160419306 (13h28m12.749764293s) [upload=13978]: Uploading segments... Aug 7 23:08:13.160446879 (13h28m12.749791866s) [upload=13978]: Got next segment. Inline: true Aug 7 23:08:13.160450095 (13h28m12.749795082s) [upload=13978]: Waiting for error group managing segments... Aug 7 23:08:13.160481565 (13h28m12.749826542s) [upload=13978]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:08:13.160485202 (13h28m12.749830189s) [upload=13978]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:08:13.160487416 (13h28m12.749832403s) [upload=13978]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:08:13.673421309 (13h28m13.262766296s) [upload=13978]: Done uploading segments... Aug 7 23:08:13.673439774 (13h28m13.262784761s) [upload=13978]: Upload finished. err: <nil> Aug 7 23:09:08.585854499 (13h29m8.175199486s) [upload=13993]: Starting upload Aug 7 23:09:08.585909965 (13h29m8.175254952s) [upload=13993]: Done starting upload Aug 7 23:09:08.585953217 (13h29m8.175298204s) [upload=13993]: Uploading segments... Aug 7 23:09:08.585980830 (13h29m8.175325817s) [upload=13993]: Got next segment. Inline: true Aug 7 23:09:08.585984206 (13h29m8.175329183s) [upload=13993]: Waiting for error group managing segments... Aug 7 23:09:08.586014684 (13h29m8.175359671s) [upload=13993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:08.586018372 (13h29m8.175363359s) [upload=13993]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:08.586020767 (13h29m8.175365754s) [upload=13993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:08.854829559 (13h29m8.444174546s) [upload=13993]: Done uploading segments... Aug 7 23:09:08.854846763 (13h29m8.44419175s) [upload=13993]: Upload finished. err: <nil> Aug 7 23:10:03.068961384 (13h30m2.658306371s) [upload=14012]: Starting upload Aug 7 23:10:03.069021258 (13h30m2.658366245s) [upload=14012]: Done starting upload Aug 7 23:10:03.069094389 (13h30m2.658439376s) [upload=14012]: Uploading segments... Aug 7 23:10:03.069128504 (13h30m2.658473491s) [upload=14012]: Got next segment. Inline: true Aug 7 23:10:03.069132782 (13h30m2.658477779s) [upload=14012]: Waiting for error group managing segments... Aug 7 23:10:03.069170935 (13h30m2.658515932s) [upload=14012]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:03.069175905 (13h30m2.658520892s) [upload=14012]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:03.069178650 (13h30m2.658523647s) [upload=14012]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:03.364714047 (13h30m2.954059064s) [upload=14012]: Done uploading segments... Aug 7 23:10:03.364761748 (13h30m2.954106745s) [upload=14012]: Upload finished. err: <nil> Aug 7 23:10:44.905335214 (13h30m44.494680201s) [upload=14022]: Starting upload Aug 7 23:10:44.905402192 (13h30m44.494747179s) [upload=14022]: Done starting upload Aug 7 23:10:44.905449983 (13h30m44.49479496s) [upload=14022]: Uploading segments... Aug 7 23:10:44.905479650 (13h30m44.494824628s) [upload=14022]: Got next segment. Inline: true Aug 7 23:10:44.905482787 (13h30m44.494827774s) [upload=14022]: Waiting for error group managing segments... Aug 7 23:10:44.905516160 (13h30m44.494861147s) [upload=14022]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:44.905519717 (13h30m44.494864694s) [upload=14022]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:44.905522222 (13h30m44.494867209s) [upload=14022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:45.196197762 (13h30m44.785542749s) [upload=14022]: Done uploading segments... Aug 7 23:10:45.196218401 (13h30m44.785563389s) [upload=14022]: Upload finished. err: <nil> Aug 7 23:11:56.158319777 (13h31m55.747664764s) [upload=14041]: Starting upload Aug 7 23:11:56.158381214 (13h31m55.747726201s) [upload=14041]: Done starting upload Aug 7 23:11:56.158454435 (13h31m55.747799422s) [upload=14041]: Uploading segments... Aug 7 23:11:56.158485985 (13h31m55.747830972s) [upload=14041]: Got next segment. Inline: true Aug 7 23:11:56.158489752 (13h31m55.747834739s) [upload=14041]: Waiting for error group managing segments... Aug 7 23:11:56.158533225 (13h31m55.747878212s) [upload=14041]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:56.158536933 (13h31m55.74788192s) [upload=14041]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:56.158539077 (13h31m55.747884064s) [upload=14041]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:55.512693945 (13h32m55.102038922s) [upload=14041]: Done uploading segments... Aug 7 23:12:55.512714414 (13h32m55.102059401s) [upload=14041]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:04.863194286 (13h33m4.452539303s) [upload=14061]: Starting upload Aug 7 23:13:04.863346336 (13h33m4.452691313s) [upload=14061]: Done starting upload Aug 7 23:13:04.863414577 (13h33m4.452759614s) [upload=14061]: Uploading segments... Aug 7 23:13:04.863490322 (13h33m4.452835309s) [upload=14061]: Got next segment. Inline: true Aug 7 23:13:04.863497876 (13h33m4.452842863s) [upload=14061]: Waiting for error group managing segments... Aug 7 23:13:04.863564063 (13h33m4.45290905s) [upload=14061]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:04.863569523 (13h33m4.45291451s) [upload=14061]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:04.863571978 (13h33m4.452916965s) [upload=14061]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:04.218769699 (13h34m3.808114676s) [upload=14061]: Done uploading segments... Aug 7 23:14:04.218789687 (13h34m3.808134664s) [upload=14061]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:05.097995389 (13h34m4.687340376s) [upload=14076]: Starting upload Aug 7 23:14:05.098050264 (13h34m4.687395251s) [upload=14076]: Done starting upload Aug 7 23:14:05.098107544 (13h34m4.687452521s) [upload=14076]: Uploading segments... Aug 7 23:14:05.098195913 (13h34m4.6875409s) [upload=14076]: Got next segment. Inline: true Aug 7 23:14:05.098203497 (13h34m4.687548484s) [upload=14076]: Waiting for error group managing segments... Aug 7 23:14:05.098273151 (13h34m4.687618138s) [upload=14076]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:05.098281256 (13h34m4.687626253s) [upload=14076]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:05.098285083 (13h34m4.68763007s) [upload=14076]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:04.449929685 (13h35m4.039274672s) [upload=14076]: Done uploading segments... Aug 7 23:15:04.449952750 (13h35m4.039297737s) [upload=14076]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:05.510260535 (13h35m5.099605522s) [upload=14092]: Starting upload Aug 7 23:15:05.510334587 (13h35m5.099679574s) [upload=14092]: Done starting upload Aug 7 23:15:05.510383931 (13h35m5.099728918s) [upload=14092]: Uploading segments... Aug 7 23:15:05.510421683 (13h35m5.09976667s) [upload=14092]: Got next segment. Inline: true Aug 7 23:15:05.510425931 (13h35m5.099770918s) [upload=14092]: Waiting for error group managing segments... Aug 7 23:15:05.510461179 (13h35m5.099806166s) [upload=14092]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:05.510464996 (13h35m5.099809984s) [upload=14092]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:05.510467561 (13h35m5.099812548s) [upload=14092]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:04.861169851 (13h36m4.450514858s) [upload=14092]: Done uploading segments... Aug 7 23:16:04.861192945 (13h36m4.450537942s) [upload=14092]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:05.761665335 (13h36m5.351010322s) [upload=14104]: Starting upload Aug 7 23:16:05.761743936 (13h36m5.351088913s) [upload=14104]: Done starting upload Aug 7 23:16:05.761859217 (13h36m5.351204194s) [upload=14104]: Uploading segments... Aug 7 23:16:05.761910234 (13h36m5.351255221s) [upload=14104]: Got next segment. Inline: true Aug 7 23:16:05.761914483 (13h36m5.35125947s) [upload=14104]: Waiting for error group managing segments... Aug 7 23:16:05.761947837 (13h36m5.351292814s) [upload=14104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:05.761951704 (13h36m5.351296691s) [upload=14104]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:05.761955882 (13h36m5.351300859s) [upload=14104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:05.106651963 (13h37m4.69599695s) [upload=14104]: Done uploading segments... Aug 7 23:17:05.106703883 (13h37m4.69604887s) [upload=14104]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.836658827s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [........................................X..X.XXXX.X.........] duration: 6.699745838s FAIL (2025-08-07 23:25:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 8]: Upload finished. err: context deadline exceeded Aug 7 23:18:32.336054087 (13h38m31.925399094s) [upload=14140]: Starting upload Aug 7 23:18:32.336180319 (13h38m31.925525306s) [upload=14140]: Done starting upload Aug 7 23:18:32.336261133 (13h38m31.92560616s) [upload=14140]: Uploading segments... Aug 7 23:18:32.336590001 (13h38m31.925935029s) [upload=14140]: Got next segment. Inline: false Aug 7 23:18:32.336650628 (13h38m31.925995655s) [upload=14140]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:32.336658754 (13h38m31.926003761s) [upload=14140]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:32.571732430 (13h38m32.161077417s) [seg_pos={0 0}, upload=14140]: Begin upload segment... Aug 7 23:18:32.571774531 (13h38m32.161119528s) [seg_pos={0 0}, upload=14140]: Done begin upload segment. Aug 7 23:18:32.571805400 (13h38m32.161150417s) [noise=true , address=77.90.31.12:4589 , piece_id=ZMKTDC4M5JHWVARXQDQAKRFSXM3OQQZ2IGQASJCK2OKYX3IQXHVA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.571844173 (13h38m32.161189161s) [noise=true , address=104.36.82.9:4007 , piece_id=VZLD7RU4SK5MAQE22D7OBVT6XOQAN75UNDMT7JNPCRZNRKFX36XQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572218380 (13h38m32.161563367s) [noise=true , address=104.36.82.10:4077 , piece_id=WYGEC4FYUPAR45MZIWLZQ2ASF5SUPECS4YI3UTRRHEVAKUEYCGYA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572289677 (13h38m32.161634654s) [noise=true , address=77.90.31.17:4770 , piece_id=ISA43NF4AKPK2SGYDPKT6ZIAD5WJ27A6GT6JZA2GHQUCAQBRRAFA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572494809 (13h38m32.161839796s) [noise=true , address=104.36.82.7:4911 , piece_id=FAMSQZJMALQIEAG5BVR5OV5WHEHN432Y4DZD3DNPC5CKP7X7GMQA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572588257 (13h38m32.161933244s) [noise=true , address=77.90.31.15:4697 , piece_id=6RYZJ22TCTJFRBMQV7M545VHB475CHDV2DBKCGO3MJP65VEMJ5FQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572772450 (13h38m32.162117427s) [noise=true , address=45.91.116.12:4225 , piece_id=22GAFTM5SITGTVJEELA3BJP7W2QZW7BVYJGJMKJOLE2RF2GZ5CGQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572865959 (13h38m32.162210946s) [noise=true , address=45.91.116.12:4227 , piece_id=T3JVU7VMSYBTYXBVGRD7LSNIG3PWF3QDVDNDQ24JQOSV7ZFIODNA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573029050 (13h38m32.162374037s) [noise=true , address=77.90.31.19:4842 , piece_id=3ZRHNMBPQUJH5SZZ75UJWLYIYZ5YI7CC5OMIJJFM7WVKZWJX6R3A, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573227882 (13h38m32.162572869s) [noise=true , address=104.36.82.4:4804 , piece_id=LTSBCPH74YSWH7IHN6MH4NWTBJ5FQ7534VERPCY5NIDV24KYPXMQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573293797 (13h38m32.162638774s) [noise=true , address=104.36.82.15:4259 , piece_id=GFCTNTXSY4WVSTPAD3AZIYCHE4O5EN24WN25VUNCRNWCSUWCYBMA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573504962 (13h38m32.162849949s) [noise=true , address=77.90.31.16:4734 , piece_id=OJPJZ5RF7QY4KNFBZ3MAVTA63HMC4JRG3HQLBN7SVROVKFQZY7BQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573527936 (13h38m32.162872923s) [noise=true , address=45.91.116.7:4009 , piece_id=VKYMGNJEGFLB5M3L3WLD37C5ZILQGLRJROSQ4ZEIBN46CQSY5JZQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.574078970 (13h38m32.163423947s) [noise=true , address=45.91.116.11:4119 , piece_id=UXO65SB4OZ57BCKOWGE2VDV7KCSKCGE7I7K4RTGRYJ7VU5GKFPHQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.574216453 (13h38m32.16356143s) [noise=true , address=104.36.82.12:4149 , piece_id=2K5BCSGU4UN43HF7VHG3MBNMQEGSIZPPHTDHDFI3XPY5DDG6NKOA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.576753347 (13h38m32.166098334s) [noise=true , address=104.36.82.8:4971 , piece_id=DNQVVV7QT2A2NFMIRURJ5IZRFPYZVX2NADRTX2P2YORA2CKMWY6Q, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:34.327895253 (13h38m33.91724024s) [upload=14140]: Next returned nil segment Aug 7 23:18:34.327912295 (13h38m33.917257282s) [upload=14140]: Waiting for error group managing segments... Aug 7 23:18:34.716828436 (13h38m34.306173413s) [noise=true , address=104.36.82.7:4911 , piece_id=FAMSQZJMALQIEAG5BVR5OV5WHEHN432Y4DZD3DNPC5CKP7X7GMQA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.729509878 (13h38m34.318854865s) [noise=true , address=104.36.82.12:4149 , piece_id=2K5BCSGU4UN43HF7VHG3MBNMQEGSIZPPHTDHDFI3XPY5DDG6NKOA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.738858565 (13h38m34.328203592s) [noise=true , address=104.36.82.4:4804 , piece_id=LTSBCPH74YSWH7IHN6MH4NWTBJ5FQ7534VERPCY5NIDV24KYPXMQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.783948702 (13h38m34.373293679s) [noise=true , address=77.90.31.19:4842 , piece_id=3ZRHNMBPQUJH5SZZ75UJWLYIYZ5YI7CC5OMIJJFM7WVKZWJX6R3A, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.824404112 (13h38m34.413749099s) [noise=true , address=104.36.82.15:4259 , piece_id=GFCTNTXSY4WVSTPAD3AZIYCHE4O5EN24WN25VUNCRNWCSUWCYBMA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.863114634 (13h38m34.452459611s) [noise=true , address=104.36.82.9:4007 , piece_id=VZLD7RU4SK5MAQE22D7OBVT6XOQAN75UNDMT7JNPCRZNRKFX36XQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.889984711 (13h38m34.479329698s) [noise=true , address=77.90.31.16:4734 , piece_id=OJPJZ5RF7QY4KNFBZ3MAVTA63HMC4JRG3HQLBN7SVROVKFQZY7BQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.982222673 (13h38m34.57156766s) [noise=true , address=104.36.82.10:4077 , piece_id=WYGEC4FYUPAR45MZIWLZQ2ASF5SUPECS4YI3UTRRHEVAKUEYCGYA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.999676977 (13h38m34.589021964s) [noise=true , address=77.90.31.17:4770 , piece_id=ISA43NF4AKPK2SGYDPKT6ZIAD5WJ27A6GT6JZA2GHQUCAQBRRAFA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.002083712 (13h38m34.591428699s) [noise=true , address=45.91.116.12:4225 , piece_id=22GAFTM5SITGTVJEELA3BJP7W2QZW7BVYJGJMKJOLE2RF2GZ5CGQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.055266482 (13h38m34.644611469s) [noise=true , address=45.91.116.12:4227 , piece_id=T3JVU7VMSYBTYXBVGRD7LSNIG3PWF3QDVDNDQ24JQOSV7ZFIODNA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.081593190 (13h38m34.670938177s) [noise=true , address=45.91.116.11:4119 , piece_id=UXO65SB4OZ57BCKOWGE2VDV7KCSKCGE7I7K4RTGRYJ7VU5GKFPHQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.123608726 (13h38m34.712953703s) [noise=true , address=77.90.31.12:4589 , piece_id=ZMKTDC4M5JHWVARXQDQAKRFSXM3OQQZ2IGQASJCK2OKYX3IQXHVA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.123631779 (13h38m34.712976766s) [seg_pos={0 0}, upload=14140]: Segment reached optimal threshold of 13 pieces. Aug 7 23:18:35.123785944 (13h38m34.713130931s) [noise=true , address=45.91.116.7:4009 , piece_id=VKYMGNJEGFLB5M3L3WLD37C5ZILQGLRJROSQ4ZEIBN46CQSY5JZQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:35.123945530 (13h38m34.713290517s) [noise=true , address=77.90.31.15:4697 , piece_id=6RYZJ22TCTJFRBMQV7M545VHB475CHDV2DBKCGO3MJP65VEMJ5FQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:35.123972431 (13h38m34.713317418s) [noise=true , address=104.36.82.8:4971 , piece_id=DNQVVV7QT2A2NFMIRURJ5IZRFPYZVX2NADRTX2P2YORA2CKMWY6Q, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: read tcp 188.166.225.170:44698->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:18:35.123986628 (13h38m34.713331615s) [seg_pos={0 0}, upload=14140]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:18:35.124083804 (13h38m34.713428791s) [upload=14140]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:18:35.124087611 (13h38m34.713432599s) [upload=14140]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:35.414558916 (13h38m35.003903903s) [upload=14140]: Done uploading segments... Aug 7 23:18:35.414577682 (13h38m35.00392267s) [upload=14140]: Upload finished. err: <nil> Aug 7 23:18:58.905554387 (13h38m58.494899394s) [upload=14146]: Starting upload Aug 7 23:18:58.905620613 (13h38m58.4949656s) [upload=14146]: Done starting upload Aug 7 23:18:58.905683373 (13h38m58.49502836s) [upload=14146]: Uploading segments... Aug 7 23:18:58.905783355 (13h38m58.495128342s) [upload=14146]: Got next segment. Inline: false Aug 7 23:18:58.905832459 (13h38m58.495177446s) [upload=14146]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:58.905836627 (13h38m58.495181614s) [upload=14146]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:19:28.246792926 (13h39m27.836137913s) [upload=14146]: Next segment err: context deadline exceeded Aug 7 23:19:28.246867379 (13h39m27.836212366s) [upload=14146]: Done uploading segments... Aug 7 23:19:28.246884292 (13h39m27.836229269s) [upload=14146]: Upload finished. err: context deadline exceeded Duration: 6.699764493s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................................XX.X.........] duration: 6.938550065s FAIL (2025-08-07 23:25:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== PC5WABH5C4FB5BLC55F4FZQAU4IAHMEA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.821935980 (13h37m27.411280967s) [noise=true , address=104.36.82.8:4983 , piece_id=AVNQRSGZB5XHG3VFYFHRLBLKQ7B4YWXGIPWDK32DYJSMFXUFTSNA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.822230193 (13h37m27.41157518s) [noise=true , address=104.36.82.16:4310 , piece_id=OAD3ECB3B5RXQB362MCLIWSQC4PKS3QUHWVMIMK4XPGH63JKO3LQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.822283705 (13h37m27.411628692s) [noise=true , address=104.36.82.59:4323 , piece_id=WNEN4TRJFQ64H74WZOQGGP3V5F7Y3A7OIMORF7M7WEYOR252JSRA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.822684663 (13h37m27.41202965s) [noise=true , address=104.36.82.39:4747 , piece_id=7OQQP63SGNWWQEYR75RRJRHZSL52NRYH3BCS65XI4SWA24BKERUQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.823034542 (13h37m27.412379529s) [noise=true , address=104.36.82.59:4322 , piece_id=D4THBNGRCWHLJC6HETJRODVC7ZRMOWMIAHGM6GYVU7LFIJNBTOZQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.823366667 (13h37m27.412711654s) [noise=true , address=104.36.82.60:4343 , piece_id=UHOQFZPYXHO56TUWMTSQE5XVWFGO66ATAA3EU7TQ5KX2MIN2ZJQQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.823697740 (13h37m27.413042727s) [noise=true , address=79.127.201.215:4355 , piece_id=KUJDHE66HM6IAFRND7KRYDII2FB7AA3S5K5S244JPQZUD4BYXFWA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.824090602 (13h37m27.413435589s) [noise=true , address=104.36.82.10:4093 , piece_id=VQUPLYS4X67PIYNTDH76KJ3DTBIWIH6UM3TQJ6VEREZIUJIH7A2Q, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.824398111 (13h37m27.413743098s) [noise=true , address=64.49.13.170:4718 , piece_id=VXSII2V5QDMEJ5G5PJWNHXSJ6HF5NLLTGEDN5CHKKRWYHIAKHMVA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.824731088 (13h37m27.414076065s) [noise=true , address=104.36.82.43:4847 , piece_id=5Y56Z5NDMWSATFWNBYDWKBZB7SUSJ3ABAFMOY6PJ3B24SPPGJH3A, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.828362365 (13h37m27.417707362s) [noise=true , address=104.36.82.61:4404 , piece_id=KGEALBUHVOZEFP7V7LPYXQG5APDPXBEWG6UG34QGOCYOHPFHEZNQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.831994514 (13h37m27.421339501s) [noise=true , address=104.36.82.18:4370 , piece_id=HHYIBQ2JWTCPEOVRZQLKEWT7BFUZDXXZP6FYED3MPEWTVODBQUQA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.835602707 (13h37m27.424947694s) [noise=true , address=45.91.116.6:4988 , piece_id=QF7HXJOV5JYZ73FJMUUGT2JJUR3W2PY3C7CE6EQQCYGM72VDSTXQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.837475180 (13h37m27.426820177s) [noise=true , address=104.36.82.34:4547 , piece_id=2XYJPP25OROUVGLPDMGFVMDOD4AYOFE6BRQ4OFMVEO2D73XVPNAQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.841864818 (13h37m27.431209825s) [noise=true , address=104.36.82.62:4439 , piece_id=F4FXK4T3IKUXUGL37K3DEDNWWAUZZFEJVT7WBYDVVEZSXO6QG4OQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:29.010273953 (13h37m28.59961894s) [upload=14126]: Next returned nil segment Aug 7 23:17:29.010293008 (13h37m28.599637996s) [upload=14126]: Waiting for error group managing segments... Aug 7 23:17:29.309514867 (13h37m28.898859844s) [noise=true , address=64.49.13.170:4718 , piece_id=VXSII2V5QDMEJ5G5PJWNHXSJ6HF5NLLTGEDN5CHKKRWYHIAKHMVA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.779221569 (13h37m29.368566556s) [noise=true , address=79.127.201.215:4355 , piece_id=KUJDHE66HM6IAFRND7KRYDII2FB7AA3S5K5S244JPQZUD4BYXFWA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.827123595 (13h37m29.416468582s) [noise=true , address=104.36.82.34:4547 , piece_id=2XYJPP25OROUVGLPDMGFVMDOD4AYOFE6BRQ4OFMVEO2D73XVPNAQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.905373281 (13h37m29.494718258s) [noise=true , address=104.36.82.39:4747 , piece_id=7OQQP63SGNWWQEYR75RRJRHZSL52NRYH3BCS65XI4SWA24BKERUQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.927717560 (13h37m29.517062537s) [noise=true , address=104.36.82.43:4847 , piece_id=5Y56Z5NDMWSATFWNBYDWKBZB7SUSJ3ABAFMOY6PJ3B24SPPGJH3A, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.139039260 (13h37m29.728384247s) [noise=true , address=104.36.82.46:4511 , piece_id=4A43V3IOJTGJNWLKNYB7PC5WABH5C4FB5BLC55F4FZQAU4IAHMEA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.142609049 (13h37m29.731954036s) [noise=true , address=104.36.82.10:4093 , piece_id=VQUPLYS4X67PIYNTDH76KJ3DTBIWIH6UM3TQJ6VEREZIUJIH7A2Q, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.265344466 (13h37m29.854689463s) [noise=true , address=104.36.82.8:4983 , piece_id=AVNQRSGZB5XHG3VFYFHRLBLKQ7B4YWXGIPWDK32DYJSMFXUFTSNA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.341283132 (13h37m29.930628119s) [noise=true , address=45.91.116.6:4988 , piece_id=QF7HXJOV5JYZ73FJMUUGT2JJUR3W2PY3C7CE6EQQCYGM72VDSTXQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.343588482 (13h37m29.932933459s) [noise=true , address=104.36.82.61:4404 , piece_id=KGEALBUHVOZEFP7V7LPYXQG5APDPXBEWG6UG34QGOCYOHPFHEZNQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.538382992 (13h37m30.127727969s) [noise=true , address=104.36.82.37:4666 , piece_id=WXBTFS2SS7DMVYBL5BHW7TBXH7FJODGB5JJDT6WP4LOO3LU4GNDA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.631974070 (13h37m30.221319047s) [noise=true , address=104.36.82.9:4028 , piece_id=HSPPTMR3OOL5DWG25IF3LXYTSOUASUM3BFX7RKHZHCUTBP4B3GQA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:31.962954475 (13h37m31.552299462s) [noise=true , address=104.36.82.59:4322 , piece_id=D4THBNGRCWHLJC6HETJRODVC7ZRMOWMIAHGM6GYVU7LFIJNBTOZQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:32.006950608 (13h37m31.596295635s) [noise=true , address=104.36.82.59:4323 , piece_id=WNEN4TRJFQ64H74WZOQGGP3V5F7Y3A7OIMORF7M7WEYOR252JSRA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:32.006984062 (13h37m31.596329049s) [seg_pos={0 0}, upload=14126]: Segment reached optimal threshold of 14 pieces. Aug 7 23:17:32.007159788 (13h37m31.596504775s) [noise=true , address=104.36.82.18:4370 , piece_id=HHYIBQ2JWTCPEOVRZQLKEWT7BFUZDXXZP6FYED3MPEWTVODBQUQA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SXZN7WJvcHys9BRd4Lq8NAZ3MSgHf9ZVv7SgaiJNRosdmFVpm): protocol: expected piece hash; context canceled; EOF Aug 7 23:17:32.007279266 (13h37m31.596624253s) [noise=true , address=104.36.82.60:4343 , piece_id=UHOQFZPYXHO56TUWMTSQE5XVWFGO66ATAA3EU7TQ5KX2MIN2ZJQQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16EsEMRW53sjKVag2WpSDGLhELyE2bSXyW4aAurCvDFKRNkXuf): protocol: write tcp 188.166.225.170:53332->104.36.82.60:4343: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:17:32.007291060 (13h37m31.596636047s) [noise=true , address=104.36.82.16:4310 , piece_id=OAD3ECB3B5RXQB362MCLIWSQC4PKS3QUHWVMIMK4XPGH63JKO3LQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R): protocol: write tcp 188.166.225.170:46238->104.36.82.16:4310: use of closed network connection; write tcp 188.166.225.170:46238->104.36.82.16:4310: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46238->104.36.82.16:4310: use of closed network connection Aug 7 23:17:32.007409626 (13h37m31.596754613s) [noise=true , address=104.36.82.62:4439 , piece_id=F4FXK4T3IKUXUGL37K3DEDNWWAUZZFEJVT7WBYDVVEZSXO6QG4OQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HRW3L15ka62qA7mWmEJaJ2ydTA5f8sVY1iL3Fn1R7ubisPF3W): protocol: read tcp 188.166.225.170:33134->104.36.82.62:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:17:32.007441958 (13h37m31.596786945s) [seg_pos={0 0}, upload=14126]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:17:32.007533564 (13h37m31.596878551s) [upload=14126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:17:32.007541158 (13h37m31.596886145s) [upload=14126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:39.583862369 (13h37m39.173207346s) [upload=14126]: Done uploading segments... Aug 7 23:17:39.583880514 (13h37m39.173225501s) [upload=14126]: Upload finished. err: <nil> Aug 7 23:17:57.117137796 (13h37m56.706482783s) [upload=14133]: Starting upload Aug 7 23:17:57.117195166 (13h37m56.706540153s) [upload=14133]: Done starting upload Aug 7 23:17:57.117244931 (13h37m56.706589918s) [upload=14133]: Uploading segments... Aug 7 23:17:57.117378968 (13h37m56.706723955s) [upload=14133]: Got next segment. Inline: false Aug 7 23:17:57.117404277 (13h37m56.706749264s) [upload=14133]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:57.117408454 (13h37m56.706753441s) [upload=14133]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:56.470635719 (13h38m56.059980716s) [upload=14133]: Next segment err: context deadline exceeded Aug 7 23:18:56.470717675 (13h38m56.060062703s) [upload=14133]: Done uploading segments... Aug 7 23:18:56.470739287 (13h38m56.060084304s) [upload=14133]: Upload finished. err: context deadline exceeded Duration: 6.938574643s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 4 out of 60 is failed (threshold=3) [.........................................X...X..X.X.........] duration: 1.857199927s FAIL (2025-08-07 23:25:37 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.857229573s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X...................X....................X..............] duration: 9.055973328s FAIL (2025-08-07 23:25:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after e85db788dbeec38ad8084daa355e1bf59b8b718af998fc40c7451ca1ba6627b2 != before 1cad814b0341363008cdb7d22dd66bb6d87f0105cd39850888ced4405a0b4b43 ======LOGS====== upload Aug 7 23:13:43.967401320 (13h33m43.556746307s) [upload=14069]: Uploading segments... Aug 7 23:13:43.967498206 (13h33m43.556843193s) [upload=14069]: Got next segment. Inline: false Aug 7 23:13:43.967560455 (13h33m43.556905442s) [upload=14069]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:43.967568550 (13h33m43.556913577s) [upload=14069]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:13:44.414536988 (13h33m44.003881975s) [seg_pos={0 0}, upload=14069]: Begin upload segment... Aug 7 23:13:44.414587104 (13h33m44.003932091s) [seg_pos={0 0}, upload=14069]: Done begin upload segment. Aug 7 23:13:44.414606933 (13h33m44.00395192s) [noise=true , address=104.36.82.60:4362 , piece_id=LJMWWXQ4NXJY54S7WPOSCO37ALDF5PT2GEC5LKOAKB6YEW2PLEAA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.414666236 (13h33m44.004011223s) [noise=true , address=104.36.82.43:4868 , piece_id=TOYHNT5XLP5YVN6ZFSWOS4UJOSIIOKUKWXP4CNSG56CHGEUZFQDQ, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.414966160 (13h33m44.004311147s) [noise=true , address=104.36.82.52:4086 , piece_id=DRDBLVU5QJKKQC5WCUJROGJYG7OQKTIRVAXDLFHQW3FIC2BV2WIA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.415118381 (13h33m44.004463369s) [noise=true , address=104.36.82.16:4296 , piece_id=HCYT2XGA5RGQQGMWKTDKJIF6OQWK6NR4Z4SUPSG5IOA6ECK3EYPQ, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.415228682 (13h33m44.00457367s) [noise=true , address=79.127.205.233:4621 , piece_id=DSV45XV43BCLRGH7BJOICWBBDBCYFZXDC4CVACYRNT4LI745DK7Q, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.415496494 (13h33m44.004841471s) [noise=true , address=104.36.82.58:4291 , piece_id=DP64RXRSK2NLFM3N7TSZCZL4PEGZVHEQJHPJFXAIQXXLOEPQWY5Q, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.415745370 (13h33m44.005090357s) [noise=true , address=45.91.116.13:4158 , piece_id=62VB7AEIWCPAVJSLX3P2TOQRSALK5A6ZAD3ZFVQINR2IVT3AOXXQ, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.416026788 (13h33m44.005371766s) [noise=true , address=104.36.82.36:4642 , piece_id=3X7E6QOZKZ5KGPLWHIWL6WZRGOFHKZLFIZRKVZM3CGJM6VLA6OAA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.416256569 (13h33m44.005601556s) [noise=true , address=104.36.82.14:4226 , piece_id=CLREJCNEC4WX6IGMUEJIV2IEZXFAVLSXLZVYMWJJBHHDHCQ6DVQA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.416456080 (13h33m44.005801067s) [noise=true , address=64.49.13.137:4499 , piece_id=7D5JZSVLNEFT26CX3TBD4L27WIZU2IWH2JK3O7IHPAVDCUDBZAJQ, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.416552746 (13h33m44.005897753s) [noise=true , address=45.91.116.14:4194 , piece_id=2LO5TLHD4ZILDKAMW6EXNTMDHG4PCRFYS6F5EMLUN4EBP4RWEUPA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.416803596 (13h33m44.006148583s) [noise=true , address=45.91.116.11:4150 , piece_id=2YORTWP2L67BV5ENJQ5CKPY3NK337SK6WM5XL77RUZ6LMQWXVORA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.417042221 (13h33m44.006387208s) [noise=true , address=104.36.82.53:4126 , piece_id=A6LKRBLYVWJ7QN4UJ7JPDXKLCI5ZFAJ6I6ICCXDE7V7G2F5R65UA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.417277101 (13h33m44.006622088s) [noise=true , address=104.36.82.59:4332 , piece_id=K5TGW4E726UEFXZQSKWGYU2PHFX2IYE2BYKHXUG2WQXTZV5GV5ZA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.417316085 (13h33m44.006661072s) [noise=true , address=104.36.82.8:4977 , piece_id=PDXPXV3IJ7Y4P25A6BS3HTKCYLLNMRYIHAMLJCVV3QDMHGN4PXKQ, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.417537459 (13h33m44.006882436s) [noise=true , address=104.36.82.2:4757 , piece_id=LJ3JPH3HYJ7Z5QIWGSKWC3KGH7QJXYT3IDTFEN326A6QOLGAWVZA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.417738975 (13h33m44.007083952s) [noise=true , address=64.49.13.26:4946 , piece_id=VYQT7NJAPN6XN6Z5PKEBJCPSYWRZHR5ZCGG6257JA3Q2BTK6S4FA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.417952763 (13h33m44.00729774s) [noise=true , address=104.36.82.9:4037 , piece_id=RISOPBKZ6Y3UYM3HSBMJT6EI3ZXGQKGANDYNUSDK3IGPCNLC4QTQ, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:45.514309387 (13h33m45.103654374s) [upload=14069]: Next returned nil segment Aug 7 23:13:45.514327020 (13h33m45.103672007s) [upload=14069]: Waiting for error group managing segments... Aug 7 23:13:45.795258156 (13h33m45.384603143s) [noise=true , address=64.49.13.137:4499 , piece_id=7D5JZSVLNEFT26CX3TBD4L27WIZU2IWH2JK3O7IHPAVDCUDBZAJQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:45.802263469 (13h33m45.391608456s) [noise=true , address=79.127.205.233:4621 , piece_id=DSV45XV43BCLRGH7BJOICWBBDBCYFZXDC4CVACYRNT4LI745DK7Q, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:45.887877424 (13h33m45.477222401s) [noise=true , address=64.49.13.26:4946 , piece_id=VYQT7NJAPN6XN6Z5PKEBJCPSYWRZHR5ZCGG6257JA3Q2BTK6S4FA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.350209185 (13h33m45.939554172s) [noise=true , address=104.36.82.60:4362 , piece_id=LJMWWXQ4NXJY54S7WPOSCO37ALDF5PT2GEC5LKOAKB6YEW2PLEAA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.389672887 (13h33m45.979017874s) [noise=true , address=104.36.82.52:4086 , piece_id=DRDBLVU5QJKKQC5WCUJROGJYG7OQKTIRVAXDLFHQW3FIC2BV2WIA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.399409194 (13h33m45.988754171s) [noise=true , address=104.36.82.58:4291 , piece_id=DP64RXRSK2NLFM3N7TSZCZL4PEGZVHEQJHPJFXAIQXXLOEPQWY5Q, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.424971119 (13h33m46.014316106s) [noise=true , address=104.36.82.36:4642 , piece_id=3X7E6QOZKZ5KGPLWHIWL6WZRGOFHKZLFIZRKVZM3CGJM6VLA6OAA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.450918913 (13h33m46.0402639s) [noise=true , address=45.91.116.13:4158 , piece_id=62VB7AEIWCPAVJSLX3P2TOQRSALK5A6ZAD3ZFVQINR2IVT3AOXXQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.492609174 (13h33m46.081954161s) [noise=true , address=104.36.82.53:4126 , piece_id=A6LKRBLYVWJ7QN4UJ7JPDXKLCI5ZFAJ6I6ICCXDE7V7G2F5R65UA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.516716376 (13h33m46.106061353s) [noise=true , address=104.36.82.9:4037 , piece_id=RISOPBKZ6Y3UYM3HSBMJT6EI3ZXGQKGANDYNUSDK3IGPCNLC4QTQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.592315332 (13h33m46.181660319s) [noise=true , address=104.36.82.43:4868 , piece_id=TOYHNT5XLP5YVN6ZFSWOS4UJOSIIOKUKWXP4CNSG56CHGEUZFQDQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.746385819 (13h33m46.335730806s) [noise=true , address=45.91.116.11:4150 , piece_id=2YORTWP2L67BV5ENJQ5CKPY3NK337SK6WM5XL77RUZ6LMQWXVORA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.786539771 (13h33m46.375884758s) [noise=true , address=104.36.82.59:4332 , piece_id=K5TGW4E726UEFXZQSKWGYU2PHFX2IYE2BYKHXUG2WQXTZV5GV5ZA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.838001699 (13h33m46.427346676s) [noise=true , address=45.91.116.14:4194 , piece_id=2LO5TLHD4ZILDKAMW6EXNTMDHG4PCRFYS6F5EMLUN4EBP4RWEUPA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.838017648 (13h33m46.427362636s) [seg_pos={0 0}, upload=14069]: Segment reached optimal threshold of 14 pieces. Aug 7 23:13:46.838217681 (13h33m46.427562658s) [noise=true , address=104.36.82.14:4226 , piece_id=CLREJCNEC4WX6IGMUEJIV2IEZXFAVLSXLZVYMWJJBHHDHCQ6DVQA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126xWe3ELGmYQfSUk3tYtGTSa6LnGfDNWcGT164pR6tHEtFPAFt): protocol: expected piece hash; context canceled; EOF Aug 7 23:13:46.838422624 (13h33m46.427767601s) [noise=true , address=104.36.82.2:4757 , piece_id=LJ3JPH3HYJ7Z5QIWGSKWC3KGH7QJXYT3IDTFEN326A6QOLGAWVZA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q): protocol: expected piece hash; context canceled; EOF Aug 7 23:13:46.838575365 (13h33m46.427920352s) [noise=true , address=104.36.82.16:4296 , piece_id=HCYT2XGA5RGQQGMWKTDKJIF6OQWK6NR4Z4SUPSG5IOA6ECK3EYPQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cKhDbA4ptwux2LYNPNAJ7Qh8jECsGWd1KWaK6T6pD38bJWdt1): protocol: rpc: dial tcp 104.36.82.16:4296: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:13:46.838627195 (13h33m46.427972172s) [noise=true , address=104.36.82.8:4977 , piece_id=PDXPXV3IJ7Y4P25A6BS3HTKCYLLNMRYIHAMLJCVV3QDMHGN4PXKQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WCAm12tXwimFSPqz7Seh6A4oxWqQVxqsF459xux4a5CqjpbAT): protocol: write tcp 188.166.225.170:48060->104.36.82.8:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:13:46.838635551 (13h33m46.427980538s) [seg_pos={0 0}, upload=14069]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:13:46.838702128 (13h33m46.428047115s) [upload=14069]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:13:46.838705895 (13h33m46.428050882s) [upload=14069]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:47.127771955 (13h33m46.717116932s) [upload=14069]: Done uploading segments... Aug 7 23:13:47.127791082 (13h33m46.717136069s) [upload=14069]: Upload finished. err: <nil> Duration: 9.056000369s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [........................................X......X............] duration: 1.827062138s FAIL (2025-08-07 23:25:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== 30.892541890 (13h31m30.481886877s) [upload=14032]: Upload finished. err: <nil> Aug 7 23:11:48.390915893 (13h31m47.98026088s) [upload=14038]: Starting upload Aug 7 23:11:48.390965198 (13h31m47.980310185s) [upload=14038]: Done starting upload Aug 7 23:11:48.391014242 (13h31m47.980359229s) [upload=14038]: Uploading segments... Aug 7 23:11:48.391054007 (13h31m47.980399004s) [upload=14038]: Got next segment. Inline: true Aug 7 23:11:48.391057494 (13h31m47.980402471s) [upload=14038]: Waiting for error group managing segments... Aug 7 23:11:48.391089254 (13h31m47.980434231s) [upload=14038]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:48.391093213 (13h31m47.98043819s) [upload=14038]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:48.391095457 (13h31m47.980440444s) [upload=14038]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:48.701950724 (13h31m48.291295711s) [upload=14038]: Done uploading segments... Aug 7 23:11:48.701969449 (13h31m48.291314436s) [upload=14038]: Upload finished. err: <nil> Aug 7 23:12:06.430245154 (13h32m6.019590141s) [upload=14048]: Starting upload Aug 7 23:12:06.430293076 (13h32m6.019638063s) [upload=14048]: Done starting upload Aug 7 23:12:06.430361757 (13h32m6.019706745s) [upload=14048]: Uploading segments... Aug 7 23:12:06.430459223 (13h32m6.01980421s) [upload=14048]: Got next segment. Inline: true Aug 7 23:12:06.430464543 (13h32m6.01980953s) [upload=14048]: Waiting for error group managing segments... Aug 7 23:12:06.430527555 (13h32m6.019872542s) [upload=14048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:06.430533796 (13h32m6.019878783s) [upload=14048]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:06.430536692 (13h32m6.019881679s) [upload=14048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:06.729952062 (13h32m6.319297049s) [upload=14048]: Done uploading segments... Aug 7 23:12:06.729968844 (13h32m6.319313832s) [upload=14048]: Upload finished. err: <nil> Aug 7 23:12:37.551701338 (13h32m37.141046325s) [upload=14055]: Starting upload Aug 7 23:12:37.551746985 (13h32m37.141091972s) [upload=14055]: Done starting upload Aug 7 23:12:37.551807301 (13h32m37.141152288s) [upload=14055]: Uploading segments... Aug 7 23:12:37.551914826 (13h32m37.141259833s) [upload=14055]: Got next segment. Inline: true Aug 7 23:12:37.551921779 (13h32m37.141266766s) [upload=14055]: Waiting for error group managing segments... Aug 7 23:12:37.551977145 (13h32m37.141322122s) [upload=14055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.551984059 (13h32m37.141329086s) [upload=14055]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.551988918 (13h32m37.141333936s) [upload=14055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.767068918 (13h32m37.356413905s) [upload=14055]: Done uploading segments... Aug 7 23:12:37.767104365 (13h32m37.356449342s) [upload=14055]: Upload finished. err: metaclient: Too Many Requests Aug 7 23:13:47.522489627 (13h33m47.111834614s) [upload=14072]: Starting upload Aug 7 23:13:47.522559681 (13h33m47.111904668s) [upload=14072]: Done starting upload Aug 7 23:13:47.522629674 (13h33m47.111974661s) [upload=14072]: Uploading segments... Aug 7 23:13:47.522726771 (13h33m47.112071778s) [upload=14072]: Got next segment. Inline: true Aug 7 23:13:47.522735617 (13h33m47.112080614s) [upload=14072]: Waiting for error group managing segments... Aug 7 23:13:47.522790582 (13h33m47.112135569s) [upload=14072]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:47.522796493 (13h33m47.112141481s) [upload=14072]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:47.522799320 (13h33m47.112144307s) [upload=14072]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:47.796144342 (13h33m47.385489329s) [upload=14072]: Done uploading segments... Aug 7 23:13:47.796162656 (13h33m47.385507643s) [upload=14072]: Upload finished. err: <nil> Aug 7 23:14:22.239921861 (13h34m21.829266848s) [upload=14081]: Starting upload Aug 7 23:14:22.239973791 (13h34m21.829318778s) [upload=14081]: Done starting upload Aug 7 23:14:22.240087399 (13h34m21.829432396s) [upload=14081]: Uploading segments... Aug 7 23:14:22.240187159 (13h34m21.829532156s) [upload=14081]: Got next segment. Inline: true Aug 7 23:14:22.240194182 (13h34m21.829539179s) [upload=14081]: Waiting for error group managing segments... Aug 7 23:14:22.240255971 (13h34m21.829600958s) [upload=14081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:22.240261311 (13h34m21.829606288s) [upload=14081]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:22.240263816 (13h34m21.829608803s) [upload=14081]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:22.537482123 (13h34m22.12682711s) [upload=14081]: Done uploading segments... Aug 7 23:14:22.537500677 (13h34m22.126845664s) [upload=14081]: Upload finished. err: <nil> Aug 7 23:14:56.968922831 (13h34m56.558267818s) [upload=14090]: Starting upload Aug 7 23:14:56.968974069 (13h34m56.558319056s) [upload=14090]: Done starting upload Aug 7 23:14:56.969028383 (13h34m56.55837337s) [upload=14090]: Uploading segments... Aug 7 23:14:56.969059602 (13h34m56.55840459s) [upload=14090]: Got next segment. Inline: true Aug 7 23:14:56.969063500 (13h34m56.558408487s) [upload=14090]: Waiting for error group managing segments... Aug 7 23:14:56.969102906 (13h34m56.558447893s) [upload=14090]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:56.969107034 (13h34m56.558452021s) [upload=14090]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:56.969109899 (13h34m56.558454886s) [upload=14090]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:57.268863065 (13h34m56.858208052s) [upload=14090]: Done uploading segments... Aug 7 23:14:57.268882201 (13h34m56.858227188s) [upload=14090]: Upload finished. err: <nil> Aug 7 23:16:09.138990771 (13h36m8.728335758s) [upload=14107]: Starting upload Aug 7 23:16:09.139048231 (13h36m8.728393208s) [upload=14107]: Done starting upload Aug 7 23:16:09.139094229 (13h36m8.728439226s) [upload=14107]: Uploading segments... Aug 7 23:16:09.139131069 (13h36m8.728476056s) [upload=14107]: Got next segment. Inline: true Aug 7 23:16:09.139135007 (13h36m8.728479994s) [upload=14107]: Waiting for error group managing segments... Aug 7 23:16:09.139167859 (13h36m8.728512846s) [upload=14107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:09.139171937 (13h36m8.728516924s) [upload=14107]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:09.139199730 (13h36m8.728544707s) [upload=14107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:09.409287150 (13h36m8.998632137s) [upload=14107]: Done uploading segments... Aug 7 23:16:09.409303952 (13h36m8.998648939s) [upload=14107]: Upload finished. err: <nil> Aug 7 23:16:30.665596455 (13h36m30.254941442s) [upload=14111]: Starting upload Aug 7 23:16:30.665642382 (13h36m30.254987369s) [upload=14111]: Done starting upload Aug 7 23:16:30.665687749 (13h36m30.255032726s) [upload=14111]: Uploading segments... Aug 7 23:16:30.665725801 (13h36m30.255070788s) [upload=14111]: Got next segment. Inline: true Aug 7 23:16:30.665729409 (13h36m30.255074386s) [upload=14111]: Waiting for error group managing segments... Aug 7 23:16:30.665763554 (13h36m30.255108541s) [upload=14111]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:30.665768984 (13h36m30.255113971s) [upload=14111]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:30.665771289 (13h36m30.255116276s) [upload=14111]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:30.945395745 (13h36m30.534740732s) [upload=14111]: Done uploading segments... Aug 7 23:16:30.945418538 (13h36m30.534763526s) [upload=14111]: Upload finished. err: <nil> Aug 7 23:17:25.677744155 (13h37m25.267089143s) [upload=14125]: Starting upload Aug 7 23:17:25.677793930 (13h37m25.267138917s) [upload=14125]: Done starting upload Aug 7 23:17:25.677862562 (13h37m25.267207539s) [upload=14125]: Uploading segments... Aug 7 23:17:25.677896967 (13h37m25.267241954s) [upload=14125]: Got next segment. Inline: true Aug 7 23:17:25.677900704 (13h37m25.267245681s) [upload=14125]: Waiting for error group managing segments... Aug 7 23:17:25.677933046 (13h37m25.267278033s) [upload=14125]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:25.677936884 (13h37m25.267281871s) [upload=14125]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:25.677939309 (13h37m25.267284296s) [upload=14125]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:26.502642065 (13h37m26.091987052s) [upload=14125]: Done uploading segments... Aug 7 23:17:26.502660942 (13h37m26.092005919s) [upload=14125]: Upload finished. err: <nil> Aug 7 23:17:53.025114325 (13h37m52.614459313s) [upload=14131]: Starting upload Aug 7 23:17:53.025169641 (13h37m52.614514618s) [upload=14131]: Done starting upload Aug 7 23:17:53.025219857 (13h37m52.614564844s) [upload=14131]: Uploading segments... Aug 7 23:17:53.025267939 (13h37m52.614612936s) [upload=14131]: Got next segment. Inline: true Aug 7 23:17:53.025271916 (13h37m52.614616903s) [upload=14131]: Waiting for error group managing segments... Aug 7 23:17:53.025302495 (13h37m52.614647482s) [upload=14131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:53.025306382 (13h37m52.614651369s) [upload=14131]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:53.025308497 (13h37m52.614653484s) [upload=14131]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:53.302835519 (13h37m52.892180506s) [upload=14131]: Done uploading segments... Aug 7 23:17:53.302854656 (13h37m52.892199633s) [upload=14131]: Upload finished. err: metaclient: Too Many Requests Duration: 1.827092526s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [...............................................X.....X.X....] duration: 2.731112322s FAIL (2025-08-07 23:25:31 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.731128662s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...........................................XXXX.X..XX.......] duration: 6.54550673s FAIL (2025-08-07 23:25:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.545525355s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 4 out of 60 is failed (threshold=3) [..............................................XXXX..........] duration: 1.842230144s FAIL (2025-08-07 23:25:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .175884539 (13h29m53.765229526s) [upload=14007]: Starting upload Aug 7 23:09:54.175937640 (13h29m53.765282618s) [upload=14007]: Done starting upload Aug 7 23:09:54.175979842 (13h29m53.765324829s) [upload=14007]: Uploading segments... Aug 7 23:09:54.176041389 (13h29m53.765386376s) [upload=14007]: Got next segment. Inline: true Aug 7 23:09:54.176045487 (13h29m53.765390474s) [upload=14007]: Waiting for error group managing segments... Aug 7 23:09:54.176077538 (13h29m53.765422525s) [upload=14007]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:54.176082318 (13h29m53.765427305s) [upload=14007]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:54.176084522 (13h29m53.765429509s) [upload=14007]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:54.453731274 (13h29m54.043076261s) [upload=14007]: Done uploading segments... Aug 7 23:09:54.453750932 (13h29m54.043095909s) [upload=14007]: Upload finished. err: <nil> Aug 7 23:10:39.330325211 (13h30m38.919670198s) [upload=14021]: Starting upload Aug 7 23:10:39.330417358 (13h30m38.919762345s) [upload=14021]: Done starting upload Aug 7 23:10:39.330504384 (13h30m38.919849371s) [upload=14021]: Uploading segments... Aug 7 23:10:39.330543368 (13h30m38.919888355s) [upload=14021]: Got next segment. Inline: true Aug 7 23:10:39.330546675 (13h30m38.919891662s) [upload=14021]: Waiting for error group managing segments... Aug 7 23:10:39.330581371 (13h30m38.919926349s) [upload=14021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:39.330585309 (13h30m38.919930286s) [upload=14021]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:39.330587663 (13h30m38.91993265s) [upload=14021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:39.610133078 (13h30m39.199478065s) [upload=14021]: Done uploading segments... Aug 7 23:10:39.610150061 (13h30m39.199495048s) [upload=14021]: Upload finished. err: <nil> Aug 7 23:11:32.695175239 (13h31m32.284520226s) [upload=14034]: Starting upload Aug 7 23:11:32.695285580 (13h31m32.284630567s) [upload=14034]: Done starting upload Aug 7 23:11:32.695336297 (13h31m32.284681284s) [upload=14034]: Uploading segments... Aug 7 23:11:32.695365753 (13h31m32.28471074s) [upload=14034]: Got next segment. Inline: true Aug 7 23:11:32.695369610 (13h31m32.284714598s) [upload=14034]: Waiting for error group managing segments... Aug 7 23:11:32.695401581 (13h31m32.284746569s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:32.695405168 (13h31m32.284750155s) [upload=14034]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:32.695407543 (13h31m32.28475253s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:32.954750627 (13h31m32.544095614s) [upload=14034]: Done uploading segments... Aug 7 23:11:32.954767650 (13h31m32.544112627s) [upload=14034]: Upload finished. err: <nil> Aug 7 23:12:37.502051138 (13h32m37.091396125s) [upload=14054]: Starting upload Aug 7 23:12:37.502104370 (13h32m37.091449357s) [upload=14054]: Done starting upload Aug 7 23:12:37.502150869 (13h32m37.091495856s) [upload=14054]: Uploading segments... Aug 7 23:12:37.502182519 (13h32m37.091527506s) [upload=14054]: Got next segment. Inline: true Aug 7 23:12:37.502185986 (13h32m37.091530963s) [upload=14054]: Waiting for error group managing segments... Aug 7 23:12:37.502216213 (13h32m37.0915612s) [upload=14054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.502219780 (13h32m37.091564767s) [upload=14054]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.502222425 (13h32m37.091567412s) [upload=14054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.772497714 (13h32m37.361842701s) [upload=14054]: Done uploading segments... Aug 7 23:12:37.772511151 (13h32m37.361856138s) [upload=14054]: Upload finished. err: <nil> Aug 7 23:13:44.834965069 (13h33m44.424310046s) [upload=14070]: Starting upload Aug 7 23:13:44.835030293 (13h33m44.42437528s) [upload=14070]: Done starting upload Aug 7 23:13:44.835093484 (13h33m44.424438471s) [upload=14070]: Uploading segments... Aug 7 23:13:44.835133321 (13h33m44.424478308s) [upload=14070]: Got next segment. Inline: true Aug 7 23:13:44.835136998 (13h33m44.424481985s) [upload=14070]: Waiting for error group managing segments... Aug 7 23:13:44.835214376 (13h33m44.424559363s) [upload=14070]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:44.835220728 (13h33m44.424565715s) [upload=14070]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:44.835222972 (13h33m44.424567959s) [upload=14070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:45.220889346 (13h33m44.810234343s) [upload=14070]: Done uploading segments... Aug 7 23:13:45.220937208 (13h33m44.810282205s) [upload=14070]: Upload finished. err: <nil> Aug 7 23:14:39.930436267 (13h34m39.519781254s) [upload=14085]: Starting upload Aug 7 23:14:39.930523634 (13h34m39.519868621s) [upload=14085]: Done starting upload Aug 7 23:14:39.930574131 (13h34m39.519919118s) [upload=14085]: Uploading segments... Aug 7 23:14:39.930614779 (13h34m39.519959766s) [upload=14085]: Got next segment. Inline: true Aug 7 23:14:39.930618847 (13h34m39.519963824s) [upload=14085]: Waiting for error group managing segments... Aug 7 23:14:39.930656519 (13h34m39.520001506s) [upload=14085]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:39.930660767 (13h34m39.520005754s) [upload=14085]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:39.930663112 (13h34m39.520008099s) [upload=14085]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:40.207143612 (13h34m39.796488599s) [upload=14085]: Done uploading segments... Aug 7 23:14:40.207161857 (13h34m39.796506844s) [upload=14085]: Upload finished. err: <nil> Aug 7 23:14:59.171620401 (13h34m58.760965388s) [upload=14091]: Starting upload Aug 7 23:14:59.171676218 (13h34m58.761021205s) [upload=14091]: Done starting upload Aug 7 23:14:59.171718468 (13h34m58.761063455s) [upload=14091]: Uploading segments... Aug 7 23:14:59.171763204 (13h34m58.761108191s) [upload=14091]: Got next segment. Inline: true Aug 7 23:14:59.171776990 (13h34m58.761121977s) [upload=14091]: Waiting for error group managing segments... Aug 7 23:14:59.171816787 (13h34m58.761161774s) [upload=14091]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:59.171820764 (13h34m58.761165741s) [upload=14091]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:59.171823429 (13h34m58.761168416s) [upload=14091]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:28.519847650 (13h35m28.109192637s) [upload=14091]: Done uploading segments... Aug 7 23:15:28.519870794 (13h35m28.109215781s) [upload=14091]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:34.319815009 (13h35m33.909159996s) [upload=14100]: Starting upload Aug 7 23:15:34.319867980 (13h35m33.909212967s) [upload=14100]: Done starting upload Aug 7 23:15:34.319916702 (13h35m33.909261679s) [upload=14100]: Uploading segments... Aug 7 23:15:34.319945739 (13h35m33.909290726s) [upload=14100]: Got next segment. Inline: true Aug 7 23:15:34.319949416 (13h35m33.909294403s) [upload=14100]: Waiting for error group managing segments... Aug 7 23:15:34.319988731 (13h35m33.909333718s) [upload=14100]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:34.319992698 (13h35m33.909337685s) [upload=14100]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:34.319995163 (13h35m33.90934015s) [upload=14100]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:03.672763058 (13h36m3.262108045s) [upload=14100]: Done uploading segments... Aug 7 23:16:03.672786503 (13h36m3.26213149s) [upload=14100]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:43.252280188 (13h36m42.841625175s) [upload=14114]: Starting upload Aug 7 23:16:43.252335193 (13h36m42.84168018s) [upload=14114]: Done starting upload Aug 7 23:16:43.252381041 (13h36m42.841726018s) [upload=14114]: Uploading segments... Aug 7 23:16:43.252412271 (13h36m42.841757258s) [upload=14114]: Got next segment. Inline: true Aug 7 23:16:43.252416208 (13h36m42.841761195s) [upload=14114]: Waiting for error group managing segments... Aug 7 23:16:43.252453641 (13h36m42.841798628s) [upload=14114]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:43.252457388 (13h36m42.841802375s) [upload=14114]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:43.252459622 (13h36m42.841804609s) [upload=14114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:12.607365819 (13h37m12.196710806s) [upload=14114]: Done uploading segments... Aug 7 23:17:12.607385377 (13h37m12.196730364s) [upload=14114]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:56.238720000 (13h37m55.828064987s) [upload=14132]: Starting upload Aug 7 23:17:56.238785927 (13h37m55.828130914s) [upload=14132]: Done starting upload Aug 7 23:17:56.238851903 (13h37m55.82819689s) [upload=14132]: Uploading segments... Aug 7 23:17:56.238885677 (13h37m55.828230664s) [upload=14132]: Got next segment. Inline: true Aug 7 23:17:56.238888933 (13h37m55.82823392s) [upload=14132]: Waiting for error group managing segments... Aug 7 23:17:56.238932207 (13h37m55.828277194s) [upload=14132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:56.238935984 (13h37m55.828280971s) [upload=14132]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:56.238938278 (13h37m55.828283265s) [upload=14132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:25.593341271 (13h38m25.182686258s) [upload=14132]: Done uploading segments... Aug 7 23:18:25.593355268 (13h38m25.182700245s) [upload=14132]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.842253932s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................................XX.X........] duration: 7.502143557s FAIL (2025-08-07 23:25:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== PC5WABH5C4FB5BLC55F4FZQAU4IAHMEA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.821935980 (13h37m27.411280967s) [noise=true , address=104.36.82.8:4983 , piece_id=AVNQRSGZB5XHG3VFYFHRLBLKQ7B4YWXGIPWDK32DYJSMFXUFTSNA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.822230193 (13h37m27.41157518s) [noise=true , address=104.36.82.16:4310 , piece_id=OAD3ECB3B5RXQB362MCLIWSQC4PKS3QUHWVMIMK4XPGH63JKO3LQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.822283705 (13h37m27.411628692s) [noise=true , address=104.36.82.59:4323 , piece_id=WNEN4TRJFQ64H74WZOQGGP3V5F7Y3A7OIMORF7M7WEYOR252JSRA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.822684663 (13h37m27.41202965s) [noise=true , address=104.36.82.39:4747 , piece_id=7OQQP63SGNWWQEYR75RRJRHZSL52NRYH3BCS65XI4SWA24BKERUQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.823034542 (13h37m27.412379529s) [noise=true , address=104.36.82.59:4322 , piece_id=D4THBNGRCWHLJC6HETJRODVC7ZRMOWMIAHGM6GYVU7LFIJNBTOZQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.823366667 (13h37m27.412711654s) [noise=true , address=104.36.82.60:4343 , piece_id=UHOQFZPYXHO56TUWMTSQE5XVWFGO66ATAA3EU7TQ5KX2MIN2ZJQQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.823697740 (13h37m27.413042727s) [noise=true , address=79.127.201.215:4355 , piece_id=KUJDHE66HM6IAFRND7KRYDII2FB7AA3S5K5S244JPQZUD4BYXFWA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.824090602 (13h37m27.413435589s) [noise=true , address=104.36.82.10:4093 , piece_id=VQUPLYS4X67PIYNTDH76KJ3DTBIWIH6UM3TQJ6VEREZIUJIH7A2Q, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.824398111 (13h37m27.413743098s) [noise=true , address=64.49.13.170:4718 , piece_id=VXSII2V5QDMEJ5G5PJWNHXSJ6HF5NLLTGEDN5CHKKRWYHIAKHMVA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.824731088 (13h37m27.414076065s) [noise=true , address=104.36.82.43:4847 , piece_id=5Y56Z5NDMWSATFWNBYDWKBZB7SUSJ3ABAFMOY6PJ3B24SPPGJH3A, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.828362365 (13h37m27.417707362s) [noise=true , address=104.36.82.61:4404 , piece_id=KGEALBUHVOZEFP7V7LPYXQG5APDPXBEWG6UG34QGOCYOHPFHEZNQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.831994514 (13h37m27.421339501s) [noise=true , address=104.36.82.18:4370 , piece_id=HHYIBQ2JWTCPEOVRZQLKEWT7BFUZDXXZP6FYED3MPEWTVODBQUQA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.835602707 (13h37m27.424947694s) [noise=true , address=45.91.116.6:4988 , piece_id=QF7HXJOV5JYZ73FJMUUGT2JJUR3W2PY3C7CE6EQQCYGM72VDSTXQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.837475180 (13h37m27.426820177s) [noise=true , address=104.36.82.34:4547 , piece_id=2XYJPP25OROUVGLPDMGFVMDOD4AYOFE6BRQ4OFMVEO2D73XVPNAQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.841864818 (13h37m27.431209825s) [noise=true , address=104.36.82.62:4439 , piece_id=F4FXK4T3IKUXUGL37K3DEDNWWAUZZFEJVT7WBYDVVEZSXO6QG4OQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:29.010273953 (13h37m28.59961894s) [upload=14126]: Next returned nil segment Aug 7 23:17:29.010293008 (13h37m28.599637996s) [upload=14126]: Waiting for error group managing segments... Aug 7 23:17:29.309514867 (13h37m28.898859844s) [noise=true , address=64.49.13.170:4718 , piece_id=VXSII2V5QDMEJ5G5PJWNHXSJ6HF5NLLTGEDN5CHKKRWYHIAKHMVA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.779221569 (13h37m29.368566556s) [noise=true , address=79.127.201.215:4355 , piece_id=KUJDHE66HM6IAFRND7KRYDII2FB7AA3S5K5S244JPQZUD4BYXFWA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.827123595 (13h37m29.416468582s) [noise=true , address=104.36.82.34:4547 , piece_id=2XYJPP25OROUVGLPDMGFVMDOD4AYOFE6BRQ4OFMVEO2D73XVPNAQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.905373281 (13h37m29.494718258s) [noise=true , address=104.36.82.39:4747 , piece_id=7OQQP63SGNWWQEYR75RRJRHZSL52NRYH3BCS65XI4SWA24BKERUQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.927717560 (13h37m29.517062537s) [noise=true , address=104.36.82.43:4847 , piece_id=5Y56Z5NDMWSATFWNBYDWKBZB7SUSJ3ABAFMOY6PJ3B24SPPGJH3A, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.139039260 (13h37m29.728384247s) [noise=true , address=104.36.82.46:4511 , piece_id=4A43V3IOJTGJNWLKNYB7PC5WABH5C4FB5BLC55F4FZQAU4IAHMEA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.142609049 (13h37m29.731954036s) [noise=true , address=104.36.82.10:4093 , piece_id=VQUPLYS4X67PIYNTDH76KJ3DTBIWIH6UM3TQJ6VEREZIUJIH7A2Q, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.265344466 (13h37m29.854689463s) [noise=true , address=104.36.82.8:4983 , piece_id=AVNQRSGZB5XHG3VFYFHRLBLKQ7B4YWXGIPWDK32DYJSMFXUFTSNA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.341283132 (13h37m29.930628119s) [noise=true , address=45.91.116.6:4988 , piece_id=QF7HXJOV5JYZ73FJMUUGT2JJUR3W2PY3C7CE6EQQCYGM72VDSTXQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.343588482 (13h37m29.932933459s) [noise=true , address=104.36.82.61:4404 , piece_id=KGEALBUHVOZEFP7V7LPYXQG5APDPXBEWG6UG34QGOCYOHPFHEZNQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.538382992 (13h37m30.127727969s) [noise=true , address=104.36.82.37:4666 , piece_id=WXBTFS2SS7DMVYBL5BHW7TBXH7FJODGB5JJDT6WP4LOO3LU4GNDA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.631974070 (13h37m30.221319047s) [noise=true , address=104.36.82.9:4028 , piece_id=HSPPTMR3OOL5DWG25IF3LXYTSOUASUM3BFX7RKHZHCUTBP4B3GQA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:31.962954475 (13h37m31.552299462s) [noise=true , address=104.36.82.59:4322 , piece_id=D4THBNGRCWHLJC6HETJRODVC7ZRMOWMIAHGM6GYVU7LFIJNBTOZQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:32.006950608 (13h37m31.596295635s) [noise=true , address=104.36.82.59:4323 , piece_id=WNEN4TRJFQ64H74WZOQGGP3V5F7Y3A7OIMORF7M7WEYOR252JSRA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:32.006984062 (13h37m31.596329049s) [seg_pos={0 0}, upload=14126]: Segment reached optimal threshold of 14 pieces. Aug 7 23:17:32.007159788 (13h37m31.596504775s) [noise=true , address=104.36.82.18:4370 , piece_id=HHYIBQ2JWTCPEOVRZQLKEWT7BFUZDXXZP6FYED3MPEWTVODBQUQA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SXZN7WJvcHys9BRd4Lq8NAZ3MSgHf9ZVv7SgaiJNRosdmFVpm): protocol: expected piece hash; context canceled; EOF Aug 7 23:17:32.007279266 (13h37m31.596624253s) [noise=true , address=104.36.82.60:4343 , piece_id=UHOQFZPYXHO56TUWMTSQE5XVWFGO66ATAA3EU7TQ5KX2MIN2ZJQQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16EsEMRW53sjKVag2WpSDGLhELyE2bSXyW4aAurCvDFKRNkXuf): protocol: write tcp 188.166.225.170:53332->104.36.82.60:4343: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:17:32.007291060 (13h37m31.596636047s) [noise=true , address=104.36.82.16:4310 , piece_id=OAD3ECB3B5RXQB362MCLIWSQC4PKS3QUHWVMIMK4XPGH63JKO3LQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R): protocol: write tcp 188.166.225.170:46238->104.36.82.16:4310: use of closed network connection; write tcp 188.166.225.170:46238->104.36.82.16:4310: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46238->104.36.82.16:4310: use of closed network connection Aug 7 23:17:32.007409626 (13h37m31.596754613s) [noise=true , address=104.36.82.62:4439 , piece_id=F4FXK4T3IKUXUGL37K3DEDNWWAUZZFEJVT7WBYDVVEZSXO6QG4OQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HRW3L15ka62qA7mWmEJaJ2ydTA5f8sVY1iL3Fn1R7ubisPF3W): protocol: read tcp 188.166.225.170:33134->104.36.82.62:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:17:32.007441958 (13h37m31.596786945s) [seg_pos={0 0}, upload=14126]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:17:32.007533564 (13h37m31.596878551s) [upload=14126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:17:32.007541158 (13h37m31.596886145s) [upload=14126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:39.583862369 (13h37m39.173207346s) [upload=14126]: Done uploading segments... Aug 7 23:17:39.583880514 (13h37m39.173225501s) [upload=14126]: Upload finished. err: <nil> Aug 7 23:17:57.117137796 (13h37m56.706482783s) [upload=14133]: Starting upload Aug 7 23:17:57.117195166 (13h37m56.706540153s) [upload=14133]: Done starting upload Aug 7 23:17:57.117244931 (13h37m56.706589918s) [upload=14133]: Uploading segments... Aug 7 23:17:57.117378968 (13h37m56.706723955s) [upload=14133]: Got next segment. Inline: false Aug 7 23:17:57.117404277 (13h37m56.706749264s) [upload=14133]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:57.117408454 (13h37m56.706753441s) [upload=14133]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:56.470635719 (13h38m56.059980716s) [upload=14133]: Next segment err: context deadline exceeded Aug 7 23:18:56.470717675 (13h38m56.060062703s) [upload=14133]: Done uploading segments... Aug 7 23:18:56.470739287 (13h38m56.060084304s) [upload=14133]: Upload finished. err: context deadline exceeded Duration: 7.502199383s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 655.391145ms target: https://us1.storj.io/login FAIL (2025-08-07 23:25:17 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 655.465037ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.........................................X..X.XXXX.X........] duration: 6.563060553s FAIL (2025-08-07 23:25:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 8]: Upload finished. err: context deadline exceeded Aug 7 23:18:32.336054087 (13h38m31.925399094s) [upload=14140]: Starting upload Aug 7 23:18:32.336180319 (13h38m31.925525306s) [upload=14140]: Done starting upload Aug 7 23:18:32.336261133 (13h38m31.92560616s) [upload=14140]: Uploading segments... Aug 7 23:18:32.336590001 (13h38m31.925935029s) [upload=14140]: Got next segment. Inline: false Aug 7 23:18:32.336650628 (13h38m31.925995655s) [upload=14140]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:32.336658754 (13h38m31.926003761s) [upload=14140]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:32.571732430 (13h38m32.161077417s) [seg_pos={0 0}, upload=14140]: Begin upload segment... Aug 7 23:18:32.571774531 (13h38m32.161119528s) [seg_pos={0 0}, upload=14140]: Done begin upload segment. Aug 7 23:18:32.571805400 (13h38m32.161150417s) [noise=true , address=77.90.31.12:4589 , piece_id=ZMKTDC4M5JHWVARXQDQAKRFSXM3OQQZ2IGQASJCK2OKYX3IQXHVA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.571844173 (13h38m32.161189161s) [noise=true , address=104.36.82.9:4007 , piece_id=VZLD7RU4SK5MAQE22D7OBVT6XOQAN75UNDMT7JNPCRZNRKFX36XQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572218380 (13h38m32.161563367s) [noise=true , address=104.36.82.10:4077 , piece_id=WYGEC4FYUPAR45MZIWLZQ2ASF5SUPECS4YI3UTRRHEVAKUEYCGYA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572289677 (13h38m32.161634654s) [noise=true , address=77.90.31.17:4770 , piece_id=ISA43NF4AKPK2SGYDPKT6ZIAD5WJ27A6GT6JZA2GHQUCAQBRRAFA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572494809 (13h38m32.161839796s) [noise=true , address=104.36.82.7:4911 , piece_id=FAMSQZJMALQIEAG5BVR5OV5WHEHN432Y4DZD3DNPC5CKP7X7GMQA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572588257 (13h38m32.161933244s) [noise=true , address=77.90.31.15:4697 , piece_id=6RYZJ22TCTJFRBMQV7M545VHB475CHDV2DBKCGO3MJP65VEMJ5FQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572772450 (13h38m32.162117427s) [noise=true , address=45.91.116.12:4225 , piece_id=22GAFTM5SITGTVJEELA3BJP7W2QZW7BVYJGJMKJOLE2RF2GZ5CGQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572865959 (13h38m32.162210946s) [noise=true , address=45.91.116.12:4227 , piece_id=T3JVU7VMSYBTYXBVGRD7LSNIG3PWF3QDVDNDQ24JQOSV7ZFIODNA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573029050 (13h38m32.162374037s) [noise=true , address=77.90.31.19:4842 , piece_id=3ZRHNMBPQUJH5SZZ75UJWLYIYZ5YI7CC5OMIJJFM7WVKZWJX6R3A, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573227882 (13h38m32.162572869s) [noise=true , address=104.36.82.4:4804 , piece_id=LTSBCPH74YSWH7IHN6MH4NWTBJ5FQ7534VERPCY5NIDV24KYPXMQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573293797 (13h38m32.162638774s) [noise=true , address=104.36.82.15:4259 , piece_id=GFCTNTXSY4WVSTPAD3AZIYCHE4O5EN24WN25VUNCRNWCSUWCYBMA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573504962 (13h38m32.162849949s) [noise=true , address=77.90.31.16:4734 , piece_id=OJPJZ5RF7QY4KNFBZ3MAVTA63HMC4JRG3HQLBN7SVROVKFQZY7BQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573527936 (13h38m32.162872923s) [noise=true , address=45.91.116.7:4009 , piece_id=VKYMGNJEGFLB5M3L3WLD37C5ZILQGLRJROSQ4ZEIBN46CQSY5JZQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.574078970 (13h38m32.163423947s) [noise=true , address=45.91.116.11:4119 , piece_id=UXO65SB4OZ57BCKOWGE2VDV7KCSKCGE7I7K4RTGRYJ7VU5GKFPHQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.574216453 (13h38m32.16356143s) [noise=true , address=104.36.82.12:4149 , piece_id=2K5BCSGU4UN43HF7VHG3MBNMQEGSIZPPHTDHDFI3XPY5DDG6NKOA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.576753347 (13h38m32.166098334s) [noise=true , address=104.36.82.8:4971 , piece_id=DNQVVV7QT2A2NFMIRURJ5IZRFPYZVX2NADRTX2P2YORA2CKMWY6Q, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:34.327895253 (13h38m33.91724024s) [upload=14140]: Next returned nil segment Aug 7 23:18:34.327912295 (13h38m33.917257282s) [upload=14140]: Waiting for error group managing segments... Aug 7 23:18:34.716828436 (13h38m34.306173413s) [noise=true , address=104.36.82.7:4911 , piece_id=FAMSQZJMALQIEAG5BVR5OV5WHEHN432Y4DZD3DNPC5CKP7X7GMQA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.729509878 (13h38m34.318854865s) [noise=true , address=104.36.82.12:4149 , piece_id=2K5BCSGU4UN43HF7VHG3MBNMQEGSIZPPHTDHDFI3XPY5DDG6NKOA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.738858565 (13h38m34.328203592s) [noise=true , address=104.36.82.4:4804 , piece_id=LTSBCPH74YSWH7IHN6MH4NWTBJ5FQ7534VERPCY5NIDV24KYPXMQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.783948702 (13h38m34.373293679s) [noise=true , address=77.90.31.19:4842 , piece_id=3ZRHNMBPQUJH5SZZ75UJWLYIYZ5YI7CC5OMIJJFM7WVKZWJX6R3A, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.824404112 (13h38m34.413749099s) [noise=true , address=104.36.82.15:4259 , piece_id=GFCTNTXSY4WVSTPAD3AZIYCHE4O5EN24WN25VUNCRNWCSUWCYBMA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.863114634 (13h38m34.452459611s) [noise=true , address=104.36.82.9:4007 , piece_id=VZLD7RU4SK5MAQE22D7OBVT6XOQAN75UNDMT7JNPCRZNRKFX36XQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.889984711 (13h38m34.479329698s) [noise=true , address=77.90.31.16:4734 , piece_id=OJPJZ5RF7QY4KNFBZ3MAVTA63HMC4JRG3HQLBN7SVROVKFQZY7BQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.982222673 (13h38m34.57156766s) [noise=true , address=104.36.82.10:4077 , piece_id=WYGEC4FYUPAR45MZIWLZQ2ASF5SUPECS4YI3UTRRHEVAKUEYCGYA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.999676977 (13h38m34.589021964s) [noise=true , address=77.90.31.17:4770 , piece_id=ISA43NF4AKPK2SGYDPKT6ZIAD5WJ27A6GT6JZA2GHQUCAQBRRAFA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.002083712 (13h38m34.591428699s) [noise=true , address=45.91.116.12:4225 , piece_id=22GAFTM5SITGTVJEELA3BJP7W2QZW7BVYJGJMKJOLE2RF2GZ5CGQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.055266482 (13h38m34.644611469s) [noise=true , address=45.91.116.12:4227 , piece_id=T3JVU7VMSYBTYXBVGRD7LSNIG3PWF3QDVDNDQ24JQOSV7ZFIODNA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.081593190 (13h38m34.670938177s) [noise=true , address=45.91.116.11:4119 , piece_id=UXO65SB4OZ57BCKOWGE2VDV7KCSKCGE7I7K4RTGRYJ7VU5GKFPHQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.123608726 (13h38m34.712953703s) [noise=true , address=77.90.31.12:4589 , piece_id=ZMKTDC4M5JHWVARXQDQAKRFSXM3OQQZ2IGQASJCK2OKYX3IQXHVA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.123631779 (13h38m34.712976766s) [seg_pos={0 0}, upload=14140]: Segment reached optimal threshold of 13 pieces. Aug 7 23:18:35.123785944 (13h38m34.713130931s) [noise=true , address=45.91.116.7:4009 , piece_id=VKYMGNJEGFLB5M3L3WLD37C5ZILQGLRJROSQ4ZEIBN46CQSY5JZQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:35.123945530 (13h38m34.713290517s) [noise=true , address=77.90.31.15:4697 , piece_id=6RYZJ22TCTJFRBMQV7M545VHB475CHDV2DBKCGO3MJP65VEMJ5FQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:35.123972431 (13h38m34.713317418s) [noise=true , address=104.36.82.8:4971 , piece_id=DNQVVV7QT2A2NFMIRURJ5IZRFPYZVX2NADRTX2P2YORA2CKMWY6Q, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: read tcp 188.166.225.170:44698->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:18:35.123986628 (13h38m34.713331615s) [seg_pos={0 0}, upload=14140]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:18:35.124083804 (13h38m34.713428791s) [upload=14140]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:18:35.124087611 (13h38m34.713432599s) [upload=14140]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:35.414558916 (13h38m35.003903903s) [upload=14140]: Done uploading segments... Aug 7 23:18:35.414577682 (13h38m35.00392267s) [upload=14140]: Upload finished. err: <nil> Aug 7 23:18:58.905554387 (13h38m58.494899394s) [upload=14146]: Starting upload Aug 7 23:18:58.905620613 (13h38m58.4949656s) [upload=14146]: Done starting upload Aug 7 23:18:58.905683373 (13h38m58.49502836s) [upload=14146]: Uploading segments... Aug 7 23:18:58.905783355 (13h38m58.495128342s) [upload=14146]: Got next segment. Inline: false Aug 7 23:18:58.905832459 (13h38m58.495177446s) [upload=14146]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:58.905836627 (13h38m58.495181614s) [upload=14146]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:19:28.246792926 (13h39m27.836137913s) [upload=14146]: Next segment err: context deadline exceeded Aug 7 23:19:28.246867379 (13h39m27.836212366s) [upload=14146]: Done uploading segments... Aug 7 23:19:28.246884292 (13h39m27.836229269s) [upload=14146]: Upload finished. err: context deadline exceeded Duration: 6.563079119s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 4 out of 60 is failed (threshold=3) [..........................................X...X..X.X........] duration: 1.868319743s FAIL (2025-08-07 23:24:59 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.86833917s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.834033465s FAIL (2025-08-07 23:24:58 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.834060176s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X...................X....................X.............] duration: 7.663916276s FAIL (2025-08-07 23:24:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after e85db788dbeec38ad8084daa355e1bf59b8b718af998fc40c7451ca1ba6627b2 != before 1cad814b0341363008cdb7d22dd66bb6d87f0105cd39850888ced4405a0b4b43 ======LOGS====== upload Aug 7 23:13:43.967401320 (13h33m43.556746307s) [upload=14069]: Uploading segments... Aug 7 23:13:43.967498206 (13h33m43.556843193s) [upload=14069]: Got next segment. Inline: false Aug 7 23:13:43.967560455 (13h33m43.556905442s) [upload=14069]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:43.967568550 (13h33m43.556913577s) [upload=14069]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:13:44.414536988 (13h33m44.003881975s) [seg_pos={0 0}, upload=14069]: Begin upload segment... Aug 7 23:13:44.414587104 (13h33m44.003932091s) [seg_pos={0 0}, upload=14069]: Done begin upload segment. Aug 7 23:13:44.414606933 (13h33m44.00395192s) [noise=true , address=104.36.82.60:4362 , piece_id=LJMWWXQ4NXJY54S7WPOSCO37ALDF5PT2GEC5LKOAKB6YEW2PLEAA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.414666236 (13h33m44.004011223s) [noise=true , address=104.36.82.43:4868 , piece_id=TOYHNT5XLP5YVN6ZFSWOS4UJOSIIOKUKWXP4CNSG56CHGEUZFQDQ, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.414966160 (13h33m44.004311147s) [noise=true , address=104.36.82.52:4086 , piece_id=DRDBLVU5QJKKQC5WCUJROGJYG7OQKTIRVAXDLFHQW3FIC2BV2WIA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.415118381 (13h33m44.004463369s) [noise=true , address=104.36.82.16:4296 , piece_id=HCYT2XGA5RGQQGMWKTDKJIF6OQWK6NR4Z4SUPSG5IOA6ECK3EYPQ, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.415228682 (13h33m44.00457367s) [noise=true , address=79.127.205.233:4621 , piece_id=DSV45XV43BCLRGH7BJOICWBBDBCYFZXDC4CVACYRNT4LI745DK7Q, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.415496494 (13h33m44.004841471s) [noise=true , address=104.36.82.58:4291 , piece_id=DP64RXRSK2NLFM3N7TSZCZL4PEGZVHEQJHPJFXAIQXXLOEPQWY5Q, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.415745370 (13h33m44.005090357s) [noise=true , address=45.91.116.13:4158 , piece_id=62VB7AEIWCPAVJSLX3P2TOQRSALK5A6ZAD3ZFVQINR2IVT3AOXXQ, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.416026788 (13h33m44.005371766s) [noise=true , address=104.36.82.36:4642 , piece_id=3X7E6QOZKZ5KGPLWHIWL6WZRGOFHKZLFIZRKVZM3CGJM6VLA6OAA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.416256569 (13h33m44.005601556s) [noise=true , address=104.36.82.14:4226 , piece_id=CLREJCNEC4WX6IGMUEJIV2IEZXFAVLSXLZVYMWJJBHHDHCQ6DVQA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.416456080 (13h33m44.005801067s) [noise=true , address=64.49.13.137:4499 , piece_id=7D5JZSVLNEFT26CX3TBD4L27WIZU2IWH2JK3O7IHPAVDCUDBZAJQ, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.416552746 (13h33m44.005897753s) [noise=true , address=45.91.116.14:4194 , piece_id=2LO5TLHD4ZILDKAMW6EXNTMDHG4PCRFYS6F5EMLUN4EBP4RWEUPA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.416803596 (13h33m44.006148583s) [noise=true , address=45.91.116.11:4150 , piece_id=2YORTWP2L67BV5ENJQ5CKPY3NK337SK6WM5XL77RUZ6LMQWXVORA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.417042221 (13h33m44.006387208s) [noise=true , address=104.36.82.53:4126 , piece_id=A6LKRBLYVWJ7QN4UJ7JPDXKLCI5ZFAJ6I6ICCXDE7V7G2F5R65UA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.417277101 (13h33m44.006622088s) [noise=true , address=104.36.82.59:4332 , piece_id=K5TGW4E726UEFXZQSKWGYU2PHFX2IYE2BYKHXUG2WQXTZV5GV5ZA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.417316085 (13h33m44.006661072s) [noise=true , address=104.36.82.8:4977 , piece_id=PDXPXV3IJ7Y4P25A6BS3HTKCYLLNMRYIHAMLJCVV3QDMHGN4PXKQ, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.417537459 (13h33m44.006882436s) [noise=true , address=104.36.82.2:4757 , piece_id=LJ3JPH3HYJ7Z5QIWGSKWC3KGH7QJXYT3IDTFEN326A6QOLGAWVZA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.417738975 (13h33m44.007083952s) [noise=true , address=64.49.13.26:4946 , piece_id=VYQT7NJAPN6XN6Z5PKEBJCPSYWRZHR5ZCGG6257JA3Q2BTK6S4FA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.417952763 (13h33m44.00729774s) [noise=true , address=104.36.82.9:4037 , piece_id=RISOPBKZ6Y3UYM3HSBMJT6EI3ZXGQKGANDYNUSDK3IGPCNLC4QTQ, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:45.514309387 (13h33m45.103654374s) [upload=14069]: Next returned nil segment Aug 7 23:13:45.514327020 (13h33m45.103672007s) [upload=14069]: Waiting for error group managing segments... Aug 7 23:13:45.795258156 (13h33m45.384603143s) [noise=true , address=64.49.13.137:4499 , piece_id=7D5JZSVLNEFT26CX3TBD4L27WIZU2IWH2JK3O7IHPAVDCUDBZAJQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:45.802263469 (13h33m45.391608456s) [noise=true , address=79.127.205.233:4621 , piece_id=DSV45XV43BCLRGH7BJOICWBBDBCYFZXDC4CVACYRNT4LI745DK7Q, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:45.887877424 (13h33m45.477222401s) [noise=true , address=64.49.13.26:4946 , piece_id=VYQT7NJAPN6XN6Z5PKEBJCPSYWRZHR5ZCGG6257JA3Q2BTK6S4FA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.350209185 (13h33m45.939554172s) [noise=true , address=104.36.82.60:4362 , piece_id=LJMWWXQ4NXJY54S7WPOSCO37ALDF5PT2GEC5LKOAKB6YEW2PLEAA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.389672887 (13h33m45.979017874s) [noise=true , address=104.36.82.52:4086 , piece_id=DRDBLVU5QJKKQC5WCUJROGJYG7OQKTIRVAXDLFHQW3FIC2BV2WIA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.399409194 (13h33m45.988754171s) [noise=true , address=104.36.82.58:4291 , piece_id=DP64RXRSK2NLFM3N7TSZCZL4PEGZVHEQJHPJFXAIQXXLOEPQWY5Q, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.424971119 (13h33m46.014316106s) [noise=true , address=104.36.82.36:4642 , piece_id=3X7E6QOZKZ5KGPLWHIWL6WZRGOFHKZLFIZRKVZM3CGJM6VLA6OAA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.450918913 (13h33m46.0402639s) [noise=true , address=45.91.116.13:4158 , piece_id=62VB7AEIWCPAVJSLX3P2TOQRSALK5A6ZAD3ZFVQINR2IVT3AOXXQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.492609174 (13h33m46.081954161s) [noise=true , address=104.36.82.53:4126 , piece_id=A6LKRBLYVWJ7QN4UJ7JPDXKLCI5ZFAJ6I6ICCXDE7V7G2F5R65UA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.516716376 (13h33m46.106061353s) [noise=true , address=104.36.82.9:4037 , piece_id=RISOPBKZ6Y3UYM3HSBMJT6EI3ZXGQKGANDYNUSDK3IGPCNLC4QTQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.592315332 (13h33m46.181660319s) [noise=true , address=104.36.82.43:4868 , piece_id=TOYHNT5XLP5YVN6ZFSWOS4UJOSIIOKUKWXP4CNSG56CHGEUZFQDQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.746385819 (13h33m46.335730806s) [noise=true , address=45.91.116.11:4150 , piece_id=2YORTWP2L67BV5ENJQ5CKPY3NK337SK6WM5XL77RUZ6LMQWXVORA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.786539771 (13h33m46.375884758s) [noise=true , address=104.36.82.59:4332 , piece_id=K5TGW4E726UEFXZQSKWGYU2PHFX2IYE2BYKHXUG2WQXTZV5GV5ZA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.838001699 (13h33m46.427346676s) [noise=true , address=45.91.116.14:4194 , piece_id=2LO5TLHD4ZILDKAMW6EXNTMDHG4PCRFYS6F5EMLUN4EBP4RWEUPA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.838017648 (13h33m46.427362636s) [seg_pos={0 0}, upload=14069]: Segment reached optimal threshold of 14 pieces. Aug 7 23:13:46.838217681 (13h33m46.427562658s) [noise=true , address=104.36.82.14:4226 , piece_id=CLREJCNEC4WX6IGMUEJIV2IEZXFAVLSXLZVYMWJJBHHDHCQ6DVQA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126xWe3ELGmYQfSUk3tYtGTSa6LnGfDNWcGT164pR6tHEtFPAFt): protocol: expected piece hash; context canceled; EOF Aug 7 23:13:46.838422624 (13h33m46.427767601s) [noise=true , address=104.36.82.2:4757 , piece_id=LJ3JPH3HYJ7Z5QIWGSKWC3KGH7QJXYT3IDTFEN326A6QOLGAWVZA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q): protocol: expected piece hash; context canceled; EOF Aug 7 23:13:46.838575365 (13h33m46.427920352s) [noise=true , address=104.36.82.16:4296 , piece_id=HCYT2XGA5RGQQGMWKTDKJIF6OQWK6NR4Z4SUPSG5IOA6ECK3EYPQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cKhDbA4ptwux2LYNPNAJ7Qh8jECsGWd1KWaK6T6pD38bJWdt1): protocol: rpc: dial tcp 104.36.82.16:4296: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:13:46.838627195 (13h33m46.427972172s) [noise=true , address=104.36.82.8:4977 , piece_id=PDXPXV3IJ7Y4P25A6BS3HTKCYLLNMRYIHAMLJCVV3QDMHGN4PXKQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WCAm12tXwimFSPqz7Seh6A4oxWqQVxqsF459xux4a5CqjpbAT): protocol: write tcp 188.166.225.170:48060->104.36.82.8:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:13:46.838635551 (13h33m46.427980538s) [seg_pos={0 0}, upload=14069]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:13:46.838702128 (13h33m46.428047115s) [upload=14069]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:13:46.838705895 (13h33m46.428050882s) [upload=14069]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:47.127771955 (13h33m46.717116932s) [upload=14069]: Done uploading segments... Aug 7 23:13:47.127791082 (13h33m46.717136069s) [upload=14069]: Upload finished. err: <nil> Duration: 7.66394002s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [..............................................XXXXXX.X.X....] duration: 5.584283436s FAIL (2025-08-07 23:24:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.584307782s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 5 out of 60 is failed (threshold=3) [.............................................XXXXX..........] duration: 1.853901746s FAIL (2025-08-07 23:24:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== d. err: <nil> Aug 7 23:07:52.874987115 (13h27m52.464332102s) [upload=13971]: Starting upload Aug 7 23:07:52.875039626 (13h27m52.464384603s) [upload=13971]: Done starting upload Aug 7 23:07:52.875078921 (13h27m52.464423908s) [upload=13971]: Uploading segments... Aug 7 23:07:52.875111583 (13h27m52.46445657s) [upload=13971]: Got next segment. Inline: true Aug 7 23:07:52.875115000 (13h27m52.464459987s) [upload=13971]: Waiting for error group managing segments... Aug 7 23:07:52.875144226 (13h27m52.464489203s) [upload=13971]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:52.875147783 (13h27m52.46449277s) [upload=13971]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:07:52.875150268 (13h27m52.464495255s) [upload=13971]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:53.152523417 (13h27m52.741868404s) [upload=13971]: Done uploading segments... Aug 7 23:07:53.152567841 (13h27m52.741912828s) [upload=13971]: Upload finished. err: <nil> Aug 7 23:08:13.160315277 (13h28m12.749660264s) [upload=13978]: Starting upload Aug 7 23:08:13.160372306 (13h28m12.749717293s) [upload=13978]: Done starting upload Aug 7 23:08:13.160419306 (13h28m12.749764293s) [upload=13978]: Uploading segments... Aug 7 23:08:13.160446879 (13h28m12.749791866s) [upload=13978]: Got next segment. Inline: true Aug 7 23:08:13.160450095 (13h28m12.749795082s) [upload=13978]: Waiting for error group managing segments... Aug 7 23:08:13.160481565 (13h28m12.749826542s) [upload=13978]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:08:13.160485202 (13h28m12.749830189s) [upload=13978]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:08:13.160487416 (13h28m12.749832403s) [upload=13978]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:08:13.673421309 (13h28m13.262766296s) [upload=13978]: Done uploading segments... Aug 7 23:08:13.673439774 (13h28m13.262784761s) [upload=13978]: Upload finished. err: <nil> Aug 7 23:09:08.585854499 (13h29m8.175199486s) [upload=13993]: Starting upload Aug 7 23:09:08.585909965 (13h29m8.175254952s) [upload=13993]: Done starting upload Aug 7 23:09:08.585953217 (13h29m8.175298204s) [upload=13993]: Uploading segments... Aug 7 23:09:08.585980830 (13h29m8.175325817s) [upload=13993]: Got next segment. Inline: true Aug 7 23:09:08.585984206 (13h29m8.175329183s) [upload=13993]: Waiting for error group managing segments... Aug 7 23:09:08.586014684 (13h29m8.175359671s) [upload=13993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:08.586018372 (13h29m8.175363359s) [upload=13993]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:08.586020767 (13h29m8.175365754s) [upload=13993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:08.854829559 (13h29m8.444174546s) [upload=13993]: Done uploading segments... Aug 7 23:09:08.854846763 (13h29m8.44419175s) [upload=13993]: Upload finished. err: <nil> Aug 7 23:10:03.068961384 (13h30m2.658306371s) [upload=14012]: Starting upload Aug 7 23:10:03.069021258 (13h30m2.658366245s) [upload=14012]: Done starting upload Aug 7 23:10:03.069094389 (13h30m2.658439376s) [upload=14012]: Uploading segments... Aug 7 23:10:03.069128504 (13h30m2.658473491s) [upload=14012]: Got next segment. Inline: true Aug 7 23:10:03.069132782 (13h30m2.658477779s) [upload=14012]: Waiting for error group managing segments... Aug 7 23:10:03.069170935 (13h30m2.658515932s) [upload=14012]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:03.069175905 (13h30m2.658520892s) [upload=14012]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:03.069178650 (13h30m2.658523647s) [upload=14012]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:03.364714047 (13h30m2.954059064s) [upload=14012]: Done uploading segments... Aug 7 23:10:03.364761748 (13h30m2.954106745s) [upload=14012]: Upload finished. err: <nil> Aug 7 23:10:44.905335214 (13h30m44.494680201s) [upload=14022]: Starting upload Aug 7 23:10:44.905402192 (13h30m44.494747179s) [upload=14022]: Done starting upload Aug 7 23:10:44.905449983 (13h30m44.49479496s) [upload=14022]: Uploading segments... Aug 7 23:10:44.905479650 (13h30m44.494824628s) [upload=14022]: Got next segment. Inline: true Aug 7 23:10:44.905482787 (13h30m44.494827774s) [upload=14022]: Waiting for error group managing segments... Aug 7 23:10:44.905516160 (13h30m44.494861147s) [upload=14022]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:44.905519717 (13h30m44.494864694s) [upload=14022]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:44.905522222 (13h30m44.494867209s) [upload=14022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:45.196197762 (13h30m44.785542749s) [upload=14022]: Done uploading segments... Aug 7 23:10:45.196218401 (13h30m44.785563389s) [upload=14022]: Upload finished. err: <nil> Aug 7 23:11:56.158319777 (13h31m55.747664764s) [upload=14041]: Starting upload Aug 7 23:11:56.158381214 (13h31m55.747726201s) [upload=14041]: Done starting upload Aug 7 23:11:56.158454435 (13h31m55.747799422s) [upload=14041]: Uploading segments... Aug 7 23:11:56.158485985 (13h31m55.747830972s) [upload=14041]: Got next segment. Inline: true Aug 7 23:11:56.158489752 (13h31m55.747834739s) [upload=14041]: Waiting for error group managing segments... Aug 7 23:11:56.158533225 (13h31m55.747878212s) [upload=14041]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:56.158536933 (13h31m55.74788192s) [upload=14041]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:56.158539077 (13h31m55.747884064s) [upload=14041]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:55.512693945 (13h32m55.102038922s) [upload=14041]: Done uploading segments... Aug 7 23:12:55.512714414 (13h32m55.102059401s) [upload=14041]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:04.863194286 (13h33m4.452539303s) [upload=14061]: Starting upload Aug 7 23:13:04.863346336 (13h33m4.452691313s) [upload=14061]: Done starting upload Aug 7 23:13:04.863414577 (13h33m4.452759614s) [upload=14061]: Uploading segments... Aug 7 23:13:04.863490322 (13h33m4.452835309s) [upload=14061]: Got next segment. Inline: true Aug 7 23:13:04.863497876 (13h33m4.452842863s) [upload=14061]: Waiting for error group managing segments... Aug 7 23:13:04.863564063 (13h33m4.45290905s) [upload=14061]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:04.863569523 (13h33m4.45291451s) [upload=14061]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:04.863571978 (13h33m4.452916965s) [upload=14061]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:04.218769699 (13h34m3.808114676s) [upload=14061]: Done uploading segments... Aug 7 23:14:04.218789687 (13h34m3.808134664s) [upload=14061]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:05.097995389 (13h34m4.687340376s) [upload=14076]: Starting upload Aug 7 23:14:05.098050264 (13h34m4.687395251s) [upload=14076]: Done starting upload Aug 7 23:14:05.098107544 (13h34m4.687452521s) [upload=14076]: Uploading segments... Aug 7 23:14:05.098195913 (13h34m4.6875409s) [upload=14076]: Got next segment. Inline: true Aug 7 23:14:05.098203497 (13h34m4.687548484s) [upload=14076]: Waiting for error group managing segments... Aug 7 23:14:05.098273151 (13h34m4.687618138s) [upload=14076]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:05.098281256 (13h34m4.687626253s) [upload=14076]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:05.098285083 (13h34m4.68763007s) [upload=14076]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:04.449929685 (13h35m4.039274672s) [upload=14076]: Done uploading segments... Aug 7 23:15:04.449952750 (13h35m4.039297737s) [upload=14076]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:05.510260535 (13h35m5.099605522s) [upload=14092]: Starting upload Aug 7 23:15:05.510334587 (13h35m5.099679574s) [upload=14092]: Done starting upload Aug 7 23:15:05.510383931 (13h35m5.099728918s) [upload=14092]: Uploading segments... Aug 7 23:15:05.510421683 (13h35m5.09976667s) [upload=14092]: Got next segment. Inline: true Aug 7 23:15:05.510425931 (13h35m5.099770918s) [upload=14092]: Waiting for error group managing segments... Aug 7 23:15:05.510461179 (13h35m5.099806166s) [upload=14092]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:05.510464996 (13h35m5.099809984s) [upload=14092]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:05.510467561 (13h35m5.099812548s) [upload=14092]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:04.861169851 (13h36m4.450514858s) [upload=14092]: Done uploading segments... Aug 7 23:16:04.861192945 (13h36m4.450537942s) [upload=14092]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:05.761665335 (13h36m5.351010322s) [upload=14104]: Starting upload Aug 7 23:16:05.761743936 (13h36m5.351088913s) [upload=14104]: Done starting upload Aug 7 23:16:05.761859217 (13h36m5.351204194s) [upload=14104]: Uploading segments... Aug 7 23:16:05.761910234 (13h36m5.351255221s) [upload=14104]: Got next segment. Inline: true Aug 7 23:16:05.761914483 (13h36m5.35125947s) [upload=14104]: Waiting for error group managing segments... Aug 7 23:16:05.761947837 (13h36m5.351292814s) [upload=14104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:05.761951704 (13h36m5.351296691s) [upload=14104]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:05.761955882 (13h36m5.351300859s) [upload=14104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:05.106651963 (13h37m4.69599695s) [upload=14104]: Done uploading segments... Aug 7 23:17:05.106703883 (13h37m4.69604887s) [upload=14104]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.853946372s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [................................................X.....X.X...] duration: 2.364595736s FAIL (2025-08-07 23:24:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.364618458s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..........................................X..X.XXXX.X.......] duration: 6.768865448s FAIL (2025-08-07 23:24:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 8]: Upload finished. err: context deadline exceeded Aug 7 23:18:32.336054087 (13h38m31.925399094s) [upload=14140]: Starting upload Aug 7 23:18:32.336180319 (13h38m31.925525306s) [upload=14140]: Done starting upload Aug 7 23:18:32.336261133 (13h38m31.92560616s) [upload=14140]: Uploading segments... Aug 7 23:18:32.336590001 (13h38m31.925935029s) [upload=14140]: Got next segment. Inline: false Aug 7 23:18:32.336650628 (13h38m31.925995655s) [upload=14140]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:32.336658754 (13h38m31.926003761s) [upload=14140]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:32.571732430 (13h38m32.161077417s) [seg_pos={0 0}, upload=14140]: Begin upload segment... Aug 7 23:18:32.571774531 (13h38m32.161119528s) [seg_pos={0 0}, upload=14140]: Done begin upload segment. Aug 7 23:18:32.571805400 (13h38m32.161150417s) [noise=true , address=77.90.31.12:4589 , piece_id=ZMKTDC4M5JHWVARXQDQAKRFSXM3OQQZ2IGQASJCK2OKYX3IQXHVA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.571844173 (13h38m32.161189161s) [noise=true , address=104.36.82.9:4007 , piece_id=VZLD7RU4SK5MAQE22D7OBVT6XOQAN75UNDMT7JNPCRZNRKFX36XQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572218380 (13h38m32.161563367s) [noise=true , address=104.36.82.10:4077 , piece_id=WYGEC4FYUPAR45MZIWLZQ2ASF5SUPECS4YI3UTRRHEVAKUEYCGYA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572289677 (13h38m32.161634654s) [noise=true , address=77.90.31.17:4770 , piece_id=ISA43NF4AKPK2SGYDPKT6ZIAD5WJ27A6GT6JZA2GHQUCAQBRRAFA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572494809 (13h38m32.161839796s) [noise=true , address=104.36.82.7:4911 , piece_id=FAMSQZJMALQIEAG5BVR5OV5WHEHN432Y4DZD3DNPC5CKP7X7GMQA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572588257 (13h38m32.161933244s) [noise=true , address=77.90.31.15:4697 , piece_id=6RYZJ22TCTJFRBMQV7M545VHB475CHDV2DBKCGO3MJP65VEMJ5FQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572772450 (13h38m32.162117427s) [noise=true , address=45.91.116.12:4225 , piece_id=22GAFTM5SITGTVJEELA3BJP7W2QZW7BVYJGJMKJOLE2RF2GZ5CGQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572865959 (13h38m32.162210946s) [noise=true , address=45.91.116.12:4227 , piece_id=T3JVU7VMSYBTYXBVGRD7LSNIG3PWF3QDVDNDQ24JQOSV7ZFIODNA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573029050 (13h38m32.162374037s) [noise=true , address=77.90.31.19:4842 , piece_id=3ZRHNMBPQUJH5SZZ75UJWLYIYZ5YI7CC5OMIJJFM7WVKZWJX6R3A, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573227882 (13h38m32.162572869s) [noise=true , address=104.36.82.4:4804 , piece_id=LTSBCPH74YSWH7IHN6MH4NWTBJ5FQ7534VERPCY5NIDV24KYPXMQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573293797 (13h38m32.162638774s) [noise=true , address=104.36.82.15:4259 , piece_id=GFCTNTXSY4WVSTPAD3AZIYCHE4O5EN24WN25VUNCRNWCSUWCYBMA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573504962 (13h38m32.162849949s) [noise=true , address=77.90.31.16:4734 , piece_id=OJPJZ5RF7QY4KNFBZ3MAVTA63HMC4JRG3HQLBN7SVROVKFQZY7BQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573527936 (13h38m32.162872923s) [noise=true , address=45.91.116.7:4009 , piece_id=VKYMGNJEGFLB5M3L3WLD37C5ZILQGLRJROSQ4ZEIBN46CQSY5JZQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.574078970 (13h38m32.163423947s) [noise=true , address=45.91.116.11:4119 , piece_id=UXO65SB4OZ57BCKOWGE2VDV7KCSKCGE7I7K4RTGRYJ7VU5GKFPHQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.574216453 (13h38m32.16356143s) [noise=true , address=104.36.82.12:4149 , piece_id=2K5BCSGU4UN43HF7VHG3MBNMQEGSIZPPHTDHDFI3XPY5DDG6NKOA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.576753347 (13h38m32.166098334s) [noise=true , address=104.36.82.8:4971 , piece_id=DNQVVV7QT2A2NFMIRURJ5IZRFPYZVX2NADRTX2P2YORA2CKMWY6Q, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:34.327895253 (13h38m33.91724024s) [upload=14140]: Next returned nil segment Aug 7 23:18:34.327912295 (13h38m33.917257282s) [upload=14140]: Waiting for error group managing segments... Aug 7 23:18:34.716828436 (13h38m34.306173413s) [noise=true , address=104.36.82.7:4911 , piece_id=FAMSQZJMALQIEAG5BVR5OV5WHEHN432Y4DZD3DNPC5CKP7X7GMQA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.729509878 (13h38m34.318854865s) [noise=true , address=104.36.82.12:4149 , piece_id=2K5BCSGU4UN43HF7VHG3MBNMQEGSIZPPHTDHDFI3XPY5DDG6NKOA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.738858565 (13h38m34.328203592s) [noise=true , address=104.36.82.4:4804 , piece_id=LTSBCPH74YSWH7IHN6MH4NWTBJ5FQ7534VERPCY5NIDV24KYPXMQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.783948702 (13h38m34.373293679s) [noise=true , address=77.90.31.19:4842 , piece_id=3ZRHNMBPQUJH5SZZ75UJWLYIYZ5YI7CC5OMIJJFM7WVKZWJX6R3A, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.824404112 (13h38m34.413749099s) [noise=true , address=104.36.82.15:4259 , piece_id=GFCTNTXSY4WVSTPAD3AZIYCHE4O5EN24WN25VUNCRNWCSUWCYBMA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.863114634 (13h38m34.452459611s) [noise=true , address=104.36.82.9:4007 , piece_id=VZLD7RU4SK5MAQE22D7OBVT6XOQAN75UNDMT7JNPCRZNRKFX36XQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.889984711 (13h38m34.479329698s) [noise=true , address=77.90.31.16:4734 , piece_id=OJPJZ5RF7QY4KNFBZ3MAVTA63HMC4JRG3HQLBN7SVROVKFQZY7BQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.982222673 (13h38m34.57156766s) [noise=true , address=104.36.82.10:4077 , piece_id=WYGEC4FYUPAR45MZIWLZQ2ASF5SUPECS4YI3UTRRHEVAKUEYCGYA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.999676977 (13h38m34.589021964s) [noise=true , address=77.90.31.17:4770 , piece_id=ISA43NF4AKPK2SGYDPKT6ZIAD5WJ27A6GT6JZA2GHQUCAQBRRAFA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.002083712 (13h38m34.591428699s) [noise=true , address=45.91.116.12:4225 , piece_id=22GAFTM5SITGTVJEELA3BJP7W2QZW7BVYJGJMKJOLE2RF2GZ5CGQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.055266482 (13h38m34.644611469s) [noise=true , address=45.91.116.12:4227 , piece_id=T3JVU7VMSYBTYXBVGRD7LSNIG3PWF3QDVDNDQ24JQOSV7ZFIODNA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.081593190 (13h38m34.670938177s) [noise=true , address=45.91.116.11:4119 , piece_id=UXO65SB4OZ57BCKOWGE2VDV7KCSKCGE7I7K4RTGRYJ7VU5GKFPHQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.123608726 (13h38m34.712953703s) [noise=true , address=77.90.31.12:4589 , piece_id=ZMKTDC4M5JHWVARXQDQAKRFSXM3OQQZ2IGQASJCK2OKYX3IQXHVA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.123631779 (13h38m34.712976766s) [seg_pos={0 0}, upload=14140]: Segment reached optimal threshold of 13 pieces. Aug 7 23:18:35.123785944 (13h38m34.713130931s) [noise=true , address=45.91.116.7:4009 , piece_id=VKYMGNJEGFLB5M3L3WLD37C5ZILQGLRJROSQ4ZEIBN46CQSY5JZQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:35.123945530 (13h38m34.713290517s) [noise=true , address=77.90.31.15:4697 , piece_id=6RYZJ22TCTJFRBMQV7M545VHB475CHDV2DBKCGO3MJP65VEMJ5FQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:35.123972431 (13h38m34.713317418s) [noise=true , address=104.36.82.8:4971 , piece_id=DNQVVV7QT2A2NFMIRURJ5IZRFPYZVX2NADRTX2P2YORA2CKMWY6Q, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: read tcp 188.166.225.170:44698->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:18:35.123986628 (13h38m34.713331615s) [seg_pos={0 0}, upload=14140]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:18:35.124083804 (13h38m34.713428791s) [upload=14140]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:18:35.124087611 (13h38m34.713432599s) [upload=14140]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:35.414558916 (13h38m35.003903903s) [upload=14140]: Done uploading segments... Aug 7 23:18:35.414577682 (13h38m35.00392267s) [upload=14140]: Upload finished. err: <nil> Aug 7 23:18:58.905554387 (13h38m58.494899394s) [upload=14146]: Starting upload Aug 7 23:18:58.905620613 (13h38m58.4949656s) [upload=14146]: Done starting upload Aug 7 23:18:58.905683373 (13h38m58.49502836s) [upload=14146]: Uploading segments... Aug 7 23:18:58.905783355 (13h38m58.495128342s) [upload=14146]: Got next segment. Inline: false Aug 7 23:18:58.905832459 (13h38m58.495177446s) [upload=14146]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:58.905836627 (13h38m58.495181614s) [upload=14146]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:19:28.246792926 (13h39m27.836137913s) [upload=14146]: Next segment err: context deadline exceeded Aug 7 23:19:28.246867379 (13h39m27.836212366s) [upload=14146]: Done uploading segments... Aug 7 23:19:28.246884292 (13h39m27.836229269s) [upload=14146]: Upload finished. err: context deadline exceeded Duration: 6.768889152s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 649.675581ms target: https://us1.storj.io/login FAIL (2025-08-07 23:24:31 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 649.699275ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 4 out of 60 is failed (threshold=3) [...............................................XXXX.........] duration: 1.933470468s FAIL (2025-08-07 23:24:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .175884539 (13h29m53.765229526s) [upload=14007]: Starting upload Aug 7 23:09:54.175937640 (13h29m53.765282618s) [upload=14007]: Done starting upload Aug 7 23:09:54.175979842 (13h29m53.765324829s) [upload=14007]: Uploading segments... Aug 7 23:09:54.176041389 (13h29m53.765386376s) [upload=14007]: Got next segment. Inline: true Aug 7 23:09:54.176045487 (13h29m53.765390474s) [upload=14007]: Waiting for error group managing segments... Aug 7 23:09:54.176077538 (13h29m53.765422525s) [upload=14007]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:54.176082318 (13h29m53.765427305s) [upload=14007]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:54.176084522 (13h29m53.765429509s) [upload=14007]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:54.453731274 (13h29m54.043076261s) [upload=14007]: Done uploading segments... Aug 7 23:09:54.453750932 (13h29m54.043095909s) [upload=14007]: Upload finished. err: <nil> Aug 7 23:10:39.330325211 (13h30m38.919670198s) [upload=14021]: Starting upload Aug 7 23:10:39.330417358 (13h30m38.919762345s) [upload=14021]: Done starting upload Aug 7 23:10:39.330504384 (13h30m38.919849371s) [upload=14021]: Uploading segments... Aug 7 23:10:39.330543368 (13h30m38.919888355s) [upload=14021]: Got next segment. Inline: true Aug 7 23:10:39.330546675 (13h30m38.919891662s) [upload=14021]: Waiting for error group managing segments... Aug 7 23:10:39.330581371 (13h30m38.919926349s) [upload=14021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:39.330585309 (13h30m38.919930286s) [upload=14021]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:39.330587663 (13h30m38.91993265s) [upload=14021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:39.610133078 (13h30m39.199478065s) [upload=14021]: Done uploading segments... Aug 7 23:10:39.610150061 (13h30m39.199495048s) [upload=14021]: Upload finished. err: <nil> Aug 7 23:11:32.695175239 (13h31m32.284520226s) [upload=14034]: Starting upload Aug 7 23:11:32.695285580 (13h31m32.284630567s) [upload=14034]: Done starting upload Aug 7 23:11:32.695336297 (13h31m32.284681284s) [upload=14034]: Uploading segments... Aug 7 23:11:32.695365753 (13h31m32.28471074s) [upload=14034]: Got next segment. Inline: true Aug 7 23:11:32.695369610 (13h31m32.284714598s) [upload=14034]: Waiting for error group managing segments... Aug 7 23:11:32.695401581 (13h31m32.284746569s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:32.695405168 (13h31m32.284750155s) [upload=14034]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:32.695407543 (13h31m32.28475253s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:32.954750627 (13h31m32.544095614s) [upload=14034]: Done uploading segments... Aug 7 23:11:32.954767650 (13h31m32.544112627s) [upload=14034]: Upload finished. err: <nil> Aug 7 23:12:37.502051138 (13h32m37.091396125s) [upload=14054]: Starting upload Aug 7 23:12:37.502104370 (13h32m37.091449357s) [upload=14054]: Done starting upload Aug 7 23:12:37.502150869 (13h32m37.091495856s) [upload=14054]: Uploading segments... Aug 7 23:12:37.502182519 (13h32m37.091527506s) [upload=14054]: Got next segment. Inline: true Aug 7 23:12:37.502185986 (13h32m37.091530963s) [upload=14054]: Waiting for error group managing segments... Aug 7 23:12:37.502216213 (13h32m37.0915612s) [upload=14054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.502219780 (13h32m37.091564767s) [upload=14054]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.502222425 (13h32m37.091567412s) [upload=14054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.772497714 (13h32m37.361842701s) [upload=14054]: Done uploading segments... Aug 7 23:12:37.772511151 (13h32m37.361856138s) [upload=14054]: Upload finished. err: <nil> Aug 7 23:13:44.834965069 (13h33m44.424310046s) [upload=14070]: Starting upload Aug 7 23:13:44.835030293 (13h33m44.42437528s) [upload=14070]: Done starting upload Aug 7 23:13:44.835093484 (13h33m44.424438471s) [upload=14070]: Uploading segments... Aug 7 23:13:44.835133321 (13h33m44.424478308s) [upload=14070]: Got next segment. Inline: true Aug 7 23:13:44.835136998 (13h33m44.424481985s) [upload=14070]: Waiting for error group managing segments... Aug 7 23:13:44.835214376 (13h33m44.424559363s) [upload=14070]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:44.835220728 (13h33m44.424565715s) [upload=14070]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:44.835222972 (13h33m44.424567959s) [upload=14070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:45.220889346 (13h33m44.810234343s) [upload=14070]: Done uploading segments... Aug 7 23:13:45.220937208 (13h33m44.810282205s) [upload=14070]: Upload finished. err: <nil> Aug 7 23:14:39.930436267 (13h34m39.519781254s) [upload=14085]: Starting upload Aug 7 23:14:39.930523634 (13h34m39.519868621s) [upload=14085]: Done starting upload Aug 7 23:14:39.930574131 (13h34m39.519919118s) [upload=14085]: Uploading segments... Aug 7 23:14:39.930614779 (13h34m39.519959766s) [upload=14085]: Got next segment. Inline: true Aug 7 23:14:39.930618847 (13h34m39.519963824s) [upload=14085]: Waiting for error group managing segments... Aug 7 23:14:39.930656519 (13h34m39.520001506s) [upload=14085]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:39.930660767 (13h34m39.520005754s) [upload=14085]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:39.930663112 (13h34m39.520008099s) [upload=14085]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:40.207143612 (13h34m39.796488599s) [upload=14085]: Done uploading segments... Aug 7 23:14:40.207161857 (13h34m39.796506844s) [upload=14085]: Upload finished. err: <nil> Aug 7 23:14:59.171620401 (13h34m58.760965388s) [upload=14091]: Starting upload Aug 7 23:14:59.171676218 (13h34m58.761021205s) [upload=14091]: Done starting upload Aug 7 23:14:59.171718468 (13h34m58.761063455s) [upload=14091]: Uploading segments... Aug 7 23:14:59.171763204 (13h34m58.761108191s) [upload=14091]: Got next segment. Inline: true Aug 7 23:14:59.171776990 (13h34m58.761121977s) [upload=14091]: Waiting for error group managing segments... Aug 7 23:14:59.171816787 (13h34m58.761161774s) [upload=14091]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:59.171820764 (13h34m58.761165741s) [upload=14091]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:59.171823429 (13h34m58.761168416s) [upload=14091]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:28.519847650 (13h35m28.109192637s) [upload=14091]: Done uploading segments... Aug 7 23:15:28.519870794 (13h35m28.109215781s) [upload=14091]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:34.319815009 (13h35m33.909159996s) [upload=14100]: Starting upload Aug 7 23:15:34.319867980 (13h35m33.909212967s) [upload=14100]: Done starting upload Aug 7 23:15:34.319916702 (13h35m33.909261679s) [upload=14100]: Uploading segments... Aug 7 23:15:34.319945739 (13h35m33.909290726s) [upload=14100]: Got next segment. Inline: true Aug 7 23:15:34.319949416 (13h35m33.909294403s) [upload=14100]: Waiting for error group managing segments... Aug 7 23:15:34.319988731 (13h35m33.909333718s) [upload=14100]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:34.319992698 (13h35m33.909337685s) [upload=14100]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:34.319995163 (13h35m33.90934015s) [upload=14100]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:03.672763058 (13h36m3.262108045s) [upload=14100]: Done uploading segments... Aug 7 23:16:03.672786503 (13h36m3.26213149s) [upload=14100]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:43.252280188 (13h36m42.841625175s) [upload=14114]: Starting upload Aug 7 23:16:43.252335193 (13h36m42.84168018s) [upload=14114]: Done starting upload Aug 7 23:16:43.252381041 (13h36m42.841726018s) [upload=14114]: Uploading segments... Aug 7 23:16:43.252412271 (13h36m42.841757258s) [upload=14114]: Got next segment. Inline: true Aug 7 23:16:43.252416208 (13h36m42.841761195s) [upload=14114]: Waiting for error group managing segments... Aug 7 23:16:43.252453641 (13h36m42.841798628s) [upload=14114]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:43.252457388 (13h36m42.841802375s) [upload=14114]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:43.252459622 (13h36m42.841804609s) [upload=14114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:12.607365819 (13h37m12.196710806s) [upload=14114]: Done uploading segments... Aug 7 23:17:12.607385377 (13h37m12.196730364s) [upload=14114]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:56.238720000 (13h37m55.828064987s) [upload=14132]: Starting upload Aug 7 23:17:56.238785927 (13h37m55.828130914s) [upload=14132]: Done starting upload Aug 7 23:17:56.238851903 (13h37m55.82819689s) [upload=14132]: Uploading segments... Aug 7 23:17:56.238885677 (13h37m55.828230664s) [upload=14132]: Got next segment. Inline: true Aug 7 23:17:56.238888933 (13h37m55.82823392s) [upload=14132]: Waiting for error group managing segments... Aug 7 23:17:56.238932207 (13h37m55.828277194s) [upload=14132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:56.238935984 (13h37m55.828280971s) [upload=14132]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:56.238938278 (13h37m55.828283265s) [upload=14132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:25.593341271 (13h38m25.182686258s) [upload=14132]: Done uploading segments... Aug 7 23:18:25.593355268 (13h38m25.182700245s) [upload=14132]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.93350801s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................................XX.X.......] duration: 7.118105342s FAIL (2025-08-07 23:24:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== PC5WABH5C4FB5BLC55F4FZQAU4IAHMEA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.821935980 (13h37m27.411280967s) [noise=true , address=104.36.82.8:4983 , piece_id=AVNQRSGZB5XHG3VFYFHRLBLKQ7B4YWXGIPWDK32DYJSMFXUFTSNA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.822230193 (13h37m27.41157518s) [noise=true , address=104.36.82.16:4310 , piece_id=OAD3ECB3B5RXQB362MCLIWSQC4PKS3QUHWVMIMK4XPGH63JKO3LQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.822283705 (13h37m27.411628692s) [noise=true , address=104.36.82.59:4323 , piece_id=WNEN4TRJFQ64H74WZOQGGP3V5F7Y3A7OIMORF7M7WEYOR252JSRA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.822684663 (13h37m27.41202965s) [noise=true , address=104.36.82.39:4747 , piece_id=7OQQP63SGNWWQEYR75RRJRHZSL52NRYH3BCS65XI4SWA24BKERUQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.823034542 (13h37m27.412379529s) [noise=true , address=104.36.82.59:4322 , piece_id=D4THBNGRCWHLJC6HETJRODVC7ZRMOWMIAHGM6GYVU7LFIJNBTOZQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.823366667 (13h37m27.412711654s) [noise=true , address=104.36.82.60:4343 , piece_id=UHOQFZPYXHO56TUWMTSQE5XVWFGO66ATAA3EU7TQ5KX2MIN2ZJQQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.823697740 (13h37m27.413042727s) [noise=true , address=79.127.201.215:4355 , piece_id=KUJDHE66HM6IAFRND7KRYDII2FB7AA3S5K5S244JPQZUD4BYXFWA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.824090602 (13h37m27.413435589s) [noise=true , address=104.36.82.10:4093 , piece_id=VQUPLYS4X67PIYNTDH76KJ3DTBIWIH6UM3TQJ6VEREZIUJIH7A2Q, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.824398111 (13h37m27.413743098s) [noise=true , address=64.49.13.170:4718 , piece_id=VXSII2V5QDMEJ5G5PJWNHXSJ6HF5NLLTGEDN5CHKKRWYHIAKHMVA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.824731088 (13h37m27.414076065s) [noise=true , address=104.36.82.43:4847 , piece_id=5Y56Z5NDMWSATFWNBYDWKBZB7SUSJ3ABAFMOY6PJ3B24SPPGJH3A, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.828362365 (13h37m27.417707362s) [noise=true , address=104.36.82.61:4404 , piece_id=KGEALBUHVOZEFP7V7LPYXQG5APDPXBEWG6UG34QGOCYOHPFHEZNQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.831994514 (13h37m27.421339501s) [noise=true , address=104.36.82.18:4370 , piece_id=HHYIBQ2JWTCPEOVRZQLKEWT7BFUZDXXZP6FYED3MPEWTVODBQUQA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.835602707 (13h37m27.424947694s) [noise=true , address=45.91.116.6:4988 , piece_id=QF7HXJOV5JYZ73FJMUUGT2JJUR3W2PY3C7CE6EQQCYGM72VDSTXQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.837475180 (13h37m27.426820177s) [noise=true , address=104.36.82.34:4547 , piece_id=2XYJPP25OROUVGLPDMGFVMDOD4AYOFE6BRQ4OFMVEO2D73XVPNAQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.841864818 (13h37m27.431209825s) [noise=true , address=104.36.82.62:4439 , piece_id=F4FXK4T3IKUXUGL37K3DEDNWWAUZZFEJVT7WBYDVVEZSXO6QG4OQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:29.010273953 (13h37m28.59961894s) [upload=14126]: Next returned nil segment Aug 7 23:17:29.010293008 (13h37m28.599637996s) [upload=14126]: Waiting for error group managing segments... Aug 7 23:17:29.309514867 (13h37m28.898859844s) [noise=true , address=64.49.13.170:4718 , piece_id=VXSII2V5QDMEJ5G5PJWNHXSJ6HF5NLLTGEDN5CHKKRWYHIAKHMVA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.779221569 (13h37m29.368566556s) [noise=true , address=79.127.201.215:4355 , piece_id=KUJDHE66HM6IAFRND7KRYDII2FB7AA3S5K5S244JPQZUD4BYXFWA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.827123595 (13h37m29.416468582s) [noise=true , address=104.36.82.34:4547 , piece_id=2XYJPP25OROUVGLPDMGFVMDOD4AYOFE6BRQ4OFMVEO2D73XVPNAQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.905373281 (13h37m29.494718258s) [noise=true , address=104.36.82.39:4747 , piece_id=7OQQP63SGNWWQEYR75RRJRHZSL52NRYH3BCS65XI4SWA24BKERUQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.927717560 (13h37m29.517062537s) [noise=true , address=104.36.82.43:4847 , piece_id=5Y56Z5NDMWSATFWNBYDWKBZB7SUSJ3ABAFMOY6PJ3B24SPPGJH3A, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.139039260 (13h37m29.728384247s) [noise=true , address=104.36.82.46:4511 , piece_id=4A43V3IOJTGJNWLKNYB7PC5WABH5C4FB5BLC55F4FZQAU4IAHMEA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.142609049 (13h37m29.731954036s) [noise=true , address=104.36.82.10:4093 , piece_id=VQUPLYS4X67PIYNTDH76KJ3DTBIWIH6UM3TQJ6VEREZIUJIH7A2Q, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.265344466 (13h37m29.854689463s) [noise=true , address=104.36.82.8:4983 , piece_id=AVNQRSGZB5XHG3VFYFHRLBLKQ7B4YWXGIPWDK32DYJSMFXUFTSNA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.341283132 (13h37m29.930628119s) [noise=true , address=45.91.116.6:4988 , piece_id=QF7HXJOV5JYZ73FJMUUGT2JJUR3W2PY3C7CE6EQQCYGM72VDSTXQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.343588482 (13h37m29.932933459s) [noise=true , address=104.36.82.61:4404 , piece_id=KGEALBUHVOZEFP7V7LPYXQG5APDPXBEWG6UG34QGOCYOHPFHEZNQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.538382992 (13h37m30.127727969s) [noise=true , address=104.36.82.37:4666 , piece_id=WXBTFS2SS7DMVYBL5BHW7TBXH7FJODGB5JJDT6WP4LOO3LU4GNDA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.631974070 (13h37m30.221319047s) [noise=true , address=104.36.82.9:4028 , piece_id=HSPPTMR3OOL5DWG25IF3LXYTSOUASUM3BFX7RKHZHCUTBP4B3GQA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:31.962954475 (13h37m31.552299462s) [noise=true , address=104.36.82.59:4322 , piece_id=D4THBNGRCWHLJC6HETJRODVC7ZRMOWMIAHGM6GYVU7LFIJNBTOZQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:32.006950608 (13h37m31.596295635s) [noise=true , address=104.36.82.59:4323 , piece_id=WNEN4TRJFQ64H74WZOQGGP3V5F7Y3A7OIMORF7M7WEYOR252JSRA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:32.006984062 (13h37m31.596329049s) [seg_pos={0 0}, upload=14126]: Segment reached optimal threshold of 14 pieces. Aug 7 23:17:32.007159788 (13h37m31.596504775s) [noise=true , address=104.36.82.18:4370 , piece_id=HHYIBQ2JWTCPEOVRZQLKEWT7BFUZDXXZP6FYED3MPEWTVODBQUQA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SXZN7WJvcHys9BRd4Lq8NAZ3MSgHf9ZVv7SgaiJNRosdmFVpm): protocol: expected piece hash; context canceled; EOF Aug 7 23:17:32.007279266 (13h37m31.596624253s) [noise=true , address=104.36.82.60:4343 , piece_id=UHOQFZPYXHO56TUWMTSQE5XVWFGO66ATAA3EU7TQ5KX2MIN2ZJQQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16EsEMRW53sjKVag2WpSDGLhELyE2bSXyW4aAurCvDFKRNkXuf): protocol: write tcp 188.166.225.170:53332->104.36.82.60:4343: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:17:32.007291060 (13h37m31.596636047s) [noise=true , address=104.36.82.16:4310 , piece_id=OAD3ECB3B5RXQB362MCLIWSQC4PKS3QUHWVMIMK4XPGH63JKO3LQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R): protocol: write tcp 188.166.225.170:46238->104.36.82.16:4310: use of closed network connection; write tcp 188.166.225.170:46238->104.36.82.16:4310: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46238->104.36.82.16:4310: use of closed network connection Aug 7 23:17:32.007409626 (13h37m31.596754613s) [noise=true , address=104.36.82.62:4439 , piece_id=F4FXK4T3IKUXUGL37K3DEDNWWAUZZFEJVT7WBYDVVEZSXO6QG4OQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HRW3L15ka62qA7mWmEJaJ2ydTA5f8sVY1iL3Fn1R7ubisPF3W): protocol: read tcp 188.166.225.170:33134->104.36.82.62:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:17:32.007441958 (13h37m31.596786945s) [seg_pos={0 0}, upload=14126]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:17:32.007533564 (13h37m31.596878551s) [upload=14126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:17:32.007541158 (13h37m31.596886145s) [upload=14126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:39.583862369 (13h37m39.173207346s) [upload=14126]: Done uploading segments... Aug 7 23:17:39.583880514 (13h37m39.173225501s) [upload=14126]: Upload finished. err: <nil> Aug 7 23:17:57.117137796 (13h37m56.706482783s) [upload=14133]: Starting upload Aug 7 23:17:57.117195166 (13h37m56.706540153s) [upload=14133]: Done starting upload Aug 7 23:17:57.117244931 (13h37m56.706589918s) [upload=14133]: Uploading segments... Aug 7 23:17:57.117378968 (13h37m56.706723955s) [upload=14133]: Got next segment. Inline: false Aug 7 23:17:57.117404277 (13h37m56.706749264s) [upload=14133]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:57.117408454 (13h37m56.706753441s) [upload=14133]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:56.470635719 (13h38m56.059980716s) [upload=14133]: Next segment err: context deadline exceeded Aug 7 23:18:56.470717675 (13h38m56.060062703s) [upload=14133]: Done uploading segments... Aug 7 23:18:56.470739287 (13h38m56.060084304s) [upload=14133]: Upload finished. err: context deadline exceeded Duration: 7.118127554s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X......X...........] duration: 4.606456533s FAIL (2025-08-07 23:24:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== 30.892541890 (13h31m30.481886877s) [upload=14032]: Upload finished. err: <nil> Aug 7 23:11:48.390915893 (13h31m47.98026088s) [upload=14038]: Starting upload Aug 7 23:11:48.390965198 (13h31m47.980310185s) [upload=14038]: Done starting upload Aug 7 23:11:48.391014242 (13h31m47.980359229s) [upload=14038]: Uploading segments... Aug 7 23:11:48.391054007 (13h31m47.980399004s) [upload=14038]: Got next segment. Inline: true Aug 7 23:11:48.391057494 (13h31m47.980402471s) [upload=14038]: Waiting for error group managing segments... Aug 7 23:11:48.391089254 (13h31m47.980434231s) [upload=14038]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:48.391093213 (13h31m47.98043819s) [upload=14038]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:48.391095457 (13h31m47.980440444s) [upload=14038]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:48.701950724 (13h31m48.291295711s) [upload=14038]: Done uploading segments... Aug 7 23:11:48.701969449 (13h31m48.291314436s) [upload=14038]: Upload finished. err: <nil> Aug 7 23:12:06.430245154 (13h32m6.019590141s) [upload=14048]: Starting upload Aug 7 23:12:06.430293076 (13h32m6.019638063s) [upload=14048]: Done starting upload Aug 7 23:12:06.430361757 (13h32m6.019706745s) [upload=14048]: Uploading segments... Aug 7 23:12:06.430459223 (13h32m6.01980421s) [upload=14048]: Got next segment. Inline: true Aug 7 23:12:06.430464543 (13h32m6.01980953s) [upload=14048]: Waiting for error group managing segments... Aug 7 23:12:06.430527555 (13h32m6.019872542s) [upload=14048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:06.430533796 (13h32m6.019878783s) [upload=14048]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:06.430536692 (13h32m6.019881679s) [upload=14048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:06.729952062 (13h32m6.319297049s) [upload=14048]: Done uploading segments... Aug 7 23:12:06.729968844 (13h32m6.319313832s) [upload=14048]: Upload finished. err: <nil> Aug 7 23:12:37.551701338 (13h32m37.141046325s) [upload=14055]: Starting upload Aug 7 23:12:37.551746985 (13h32m37.141091972s) [upload=14055]: Done starting upload Aug 7 23:12:37.551807301 (13h32m37.141152288s) [upload=14055]: Uploading segments... Aug 7 23:12:37.551914826 (13h32m37.141259833s) [upload=14055]: Got next segment. Inline: true Aug 7 23:12:37.551921779 (13h32m37.141266766s) [upload=14055]: Waiting for error group managing segments... Aug 7 23:12:37.551977145 (13h32m37.141322122s) [upload=14055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.551984059 (13h32m37.141329086s) [upload=14055]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.551988918 (13h32m37.141333936s) [upload=14055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.767068918 (13h32m37.356413905s) [upload=14055]: Done uploading segments... Aug 7 23:12:37.767104365 (13h32m37.356449342s) [upload=14055]: Upload finished. err: metaclient: Too Many Requests Aug 7 23:13:47.522489627 (13h33m47.111834614s) [upload=14072]: Starting upload Aug 7 23:13:47.522559681 (13h33m47.111904668s) [upload=14072]: Done starting upload Aug 7 23:13:47.522629674 (13h33m47.111974661s) [upload=14072]: Uploading segments... Aug 7 23:13:47.522726771 (13h33m47.112071778s) [upload=14072]: Got next segment. Inline: true Aug 7 23:13:47.522735617 (13h33m47.112080614s) [upload=14072]: Waiting for error group managing segments... Aug 7 23:13:47.522790582 (13h33m47.112135569s) [upload=14072]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:47.522796493 (13h33m47.112141481s) [upload=14072]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:47.522799320 (13h33m47.112144307s) [upload=14072]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:47.796144342 (13h33m47.385489329s) [upload=14072]: Done uploading segments... Aug 7 23:13:47.796162656 (13h33m47.385507643s) [upload=14072]: Upload finished. err: <nil> Aug 7 23:14:22.239921861 (13h34m21.829266848s) [upload=14081]: Starting upload Aug 7 23:14:22.239973791 (13h34m21.829318778s) [upload=14081]: Done starting upload Aug 7 23:14:22.240087399 (13h34m21.829432396s) [upload=14081]: Uploading segments... Aug 7 23:14:22.240187159 (13h34m21.829532156s) [upload=14081]: Got next segment. Inline: true Aug 7 23:14:22.240194182 (13h34m21.829539179s) [upload=14081]: Waiting for error group managing segments... Aug 7 23:14:22.240255971 (13h34m21.829600958s) [upload=14081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:22.240261311 (13h34m21.829606288s) [upload=14081]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:22.240263816 (13h34m21.829608803s) [upload=14081]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:22.537482123 (13h34m22.12682711s) [upload=14081]: Done uploading segments... Aug 7 23:14:22.537500677 (13h34m22.126845664s) [upload=14081]: Upload finished. err: <nil> Aug 7 23:14:56.968922831 (13h34m56.558267818s) [upload=14090]: Starting upload Aug 7 23:14:56.968974069 (13h34m56.558319056s) [upload=14090]: Done starting upload Aug 7 23:14:56.969028383 (13h34m56.55837337s) [upload=14090]: Uploading segments... Aug 7 23:14:56.969059602 (13h34m56.55840459s) [upload=14090]: Got next segment. Inline: true Aug 7 23:14:56.969063500 (13h34m56.558408487s) [upload=14090]: Waiting for error group managing segments... Aug 7 23:14:56.969102906 (13h34m56.558447893s) [upload=14090]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:56.969107034 (13h34m56.558452021s) [upload=14090]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:56.969109899 (13h34m56.558454886s) [upload=14090]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:57.268863065 (13h34m56.858208052s) [upload=14090]: Done uploading segments... Aug 7 23:14:57.268882201 (13h34m56.858227188s) [upload=14090]: Upload finished. err: <nil> Aug 7 23:16:09.138990771 (13h36m8.728335758s) [upload=14107]: Starting upload Aug 7 23:16:09.139048231 (13h36m8.728393208s) [upload=14107]: Done starting upload Aug 7 23:16:09.139094229 (13h36m8.728439226s) [upload=14107]: Uploading segments... Aug 7 23:16:09.139131069 (13h36m8.728476056s) [upload=14107]: Got next segment. Inline: true Aug 7 23:16:09.139135007 (13h36m8.728479994s) [upload=14107]: Waiting for error group managing segments... Aug 7 23:16:09.139167859 (13h36m8.728512846s) [upload=14107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:09.139171937 (13h36m8.728516924s) [upload=14107]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:09.139199730 (13h36m8.728544707s) [upload=14107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:09.409287150 (13h36m8.998632137s) [upload=14107]: Done uploading segments... Aug 7 23:16:09.409303952 (13h36m8.998648939s) [upload=14107]: Upload finished. err: <nil> Aug 7 23:16:30.665596455 (13h36m30.254941442s) [upload=14111]: Starting upload Aug 7 23:16:30.665642382 (13h36m30.254987369s) [upload=14111]: Done starting upload Aug 7 23:16:30.665687749 (13h36m30.255032726s) [upload=14111]: Uploading segments... Aug 7 23:16:30.665725801 (13h36m30.255070788s) [upload=14111]: Got next segment. Inline: true Aug 7 23:16:30.665729409 (13h36m30.255074386s) [upload=14111]: Waiting for error group managing segments... Aug 7 23:16:30.665763554 (13h36m30.255108541s) [upload=14111]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:30.665768984 (13h36m30.255113971s) [upload=14111]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:30.665771289 (13h36m30.255116276s) [upload=14111]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:30.945395745 (13h36m30.534740732s) [upload=14111]: Done uploading segments... Aug 7 23:16:30.945418538 (13h36m30.534763526s) [upload=14111]: Upload finished. err: <nil> Aug 7 23:17:25.677744155 (13h37m25.267089143s) [upload=14125]: Starting upload Aug 7 23:17:25.677793930 (13h37m25.267138917s) [upload=14125]: Done starting upload Aug 7 23:17:25.677862562 (13h37m25.267207539s) [upload=14125]: Uploading segments... Aug 7 23:17:25.677896967 (13h37m25.267241954s) [upload=14125]: Got next segment. Inline: true Aug 7 23:17:25.677900704 (13h37m25.267245681s) [upload=14125]: Waiting for error group managing segments... Aug 7 23:17:25.677933046 (13h37m25.267278033s) [upload=14125]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:25.677936884 (13h37m25.267281871s) [upload=14125]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:25.677939309 (13h37m25.267284296s) [upload=14125]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:26.502642065 (13h37m26.091987052s) [upload=14125]: Done uploading segments... Aug 7 23:17:26.502660942 (13h37m26.092005919s) [upload=14125]: Upload finished. err: <nil> Aug 7 23:17:53.025114325 (13h37m52.614459313s) [upload=14131]: Starting upload Aug 7 23:17:53.025169641 (13h37m52.614514618s) [upload=14131]: Done starting upload Aug 7 23:17:53.025219857 (13h37m52.614564844s) [upload=14131]: Uploading segments... Aug 7 23:17:53.025267939 (13h37m52.614612936s) [upload=14131]: Got next segment. Inline: true Aug 7 23:17:53.025271916 (13h37m52.614616903s) [upload=14131]: Waiting for error group managing segments... Aug 7 23:17:53.025302495 (13h37m52.614647482s) [upload=14131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:53.025306382 (13h37m52.614651369s) [upload=14131]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:53.025308497 (13h37m52.614653484s) [upload=14131]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:53.302835519 (13h37m52.892180506s) [upload=14131]: Done uploading segments... Aug 7 23:17:53.302854656 (13h37m52.892199633s) [upload=14131]: Upload finished. err: metaclient: Too Many Requests Duration: 4.606480009s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [............................................XXXX.X..XX......] duration: 6.544708349s FAIL (2025-08-07 23:24:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.544720143s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.738430104s FAIL (2025-08-07 23:24:09 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.73846416s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 5 out of 60 is failed (threshold=3) [..............................................XXXXX.........] duration: 1.88241502s FAIL (2025-08-07 23:24:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== d. err: <nil> Aug 7 23:07:52.874987115 (13h27m52.464332102s) [upload=13971]: Starting upload Aug 7 23:07:52.875039626 (13h27m52.464384603s) [upload=13971]: Done starting upload Aug 7 23:07:52.875078921 (13h27m52.464423908s) [upload=13971]: Uploading segments... Aug 7 23:07:52.875111583 (13h27m52.46445657s) [upload=13971]: Got next segment. Inline: true Aug 7 23:07:52.875115000 (13h27m52.464459987s) [upload=13971]: Waiting for error group managing segments... Aug 7 23:07:52.875144226 (13h27m52.464489203s) [upload=13971]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:52.875147783 (13h27m52.46449277s) [upload=13971]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:07:52.875150268 (13h27m52.464495255s) [upload=13971]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:53.152523417 (13h27m52.741868404s) [upload=13971]: Done uploading segments... Aug 7 23:07:53.152567841 (13h27m52.741912828s) [upload=13971]: Upload finished. err: <nil> Aug 7 23:08:13.160315277 (13h28m12.749660264s) [upload=13978]: Starting upload Aug 7 23:08:13.160372306 (13h28m12.749717293s) [upload=13978]: Done starting upload Aug 7 23:08:13.160419306 (13h28m12.749764293s) [upload=13978]: Uploading segments... Aug 7 23:08:13.160446879 (13h28m12.749791866s) [upload=13978]: Got next segment. Inline: true Aug 7 23:08:13.160450095 (13h28m12.749795082s) [upload=13978]: Waiting for error group managing segments... Aug 7 23:08:13.160481565 (13h28m12.749826542s) [upload=13978]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:08:13.160485202 (13h28m12.749830189s) [upload=13978]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:08:13.160487416 (13h28m12.749832403s) [upload=13978]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:08:13.673421309 (13h28m13.262766296s) [upload=13978]: Done uploading segments... Aug 7 23:08:13.673439774 (13h28m13.262784761s) [upload=13978]: Upload finished. err: <nil> Aug 7 23:09:08.585854499 (13h29m8.175199486s) [upload=13993]: Starting upload Aug 7 23:09:08.585909965 (13h29m8.175254952s) [upload=13993]: Done starting upload Aug 7 23:09:08.585953217 (13h29m8.175298204s) [upload=13993]: Uploading segments... Aug 7 23:09:08.585980830 (13h29m8.175325817s) [upload=13993]: Got next segment. Inline: true Aug 7 23:09:08.585984206 (13h29m8.175329183s) [upload=13993]: Waiting for error group managing segments... Aug 7 23:09:08.586014684 (13h29m8.175359671s) [upload=13993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:08.586018372 (13h29m8.175363359s) [upload=13993]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:08.586020767 (13h29m8.175365754s) [upload=13993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:08.854829559 (13h29m8.444174546s) [upload=13993]: Done uploading segments... Aug 7 23:09:08.854846763 (13h29m8.44419175s) [upload=13993]: Upload finished. err: <nil> Aug 7 23:10:03.068961384 (13h30m2.658306371s) [upload=14012]: Starting upload Aug 7 23:10:03.069021258 (13h30m2.658366245s) [upload=14012]: Done starting upload Aug 7 23:10:03.069094389 (13h30m2.658439376s) [upload=14012]: Uploading segments... Aug 7 23:10:03.069128504 (13h30m2.658473491s) [upload=14012]: Got next segment. Inline: true Aug 7 23:10:03.069132782 (13h30m2.658477779s) [upload=14012]: Waiting for error group managing segments... Aug 7 23:10:03.069170935 (13h30m2.658515932s) [upload=14012]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:03.069175905 (13h30m2.658520892s) [upload=14012]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:03.069178650 (13h30m2.658523647s) [upload=14012]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:03.364714047 (13h30m2.954059064s) [upload=14012]: Done uploading segments... Aug 7 23:10:03.364761748 (13h30m2.954106745s) [upload=14012]: Upload finished. err: <nil> Aug 7 23:10:44.905335214 (13h30m44.494680201s) [upload=14022]: Starting upload Aug 7 23:10:44.905402192 (13h30m44.494747179s) [upload=14022]: Done starting upload Aug 7 23:10:44.905449983 (13h30m44.49479496s) [upload=14022]: Uploading segments... Aug 7 23:10:44.905479650 (13h30m44.494824628s) [upload=14022]: Got next segment. Inline: true Aug 7 23:10:44.905482787 (13h30m44.494827774s) [upload=14022]: Waiting for error group managing segments... Aug 7 23:10:44.905516160 (13h30m44.494861147s) [upload=14022]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:44.905519717 (13h30m44.494864694s) [upload=14022]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:44.905522222 (13h30m44.494867209s) [upload=14022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:45.196197762 (13h30m44.785542749s) [upload=14022]: Done uploading segments... Aug 7 23:10:45.196218401 (13h30m44.785563389s) [upload=14022]: Upload finished. err: <nil> Aug 7 23:11:56.158319777 (13h31m55.747664764s) [upload=14041]: Starting upload Aug 7 23:11:56.158381214 (13h31m55.747726201s) [upload=14041]: Done starting upload Aug 7 23:11:56.158454435 (13h31m55.747799422s) [upload=14041]: Uploading segments... Aug 7 23:11:56.158485985 (13h31m55.747830972s) [upload=14041]: Got next segment. Inline: true Aug 7 23:11:56.158489752 (13h31m55.747834739s) [upload=14041]: Waiting for error group managing segments... Aug 7 23:11:56.158533225 (13h31m55.747878212s) [upload=14041]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:56.158536933 (13h31m55.74788192s) [upload=14041]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:56.158539077 (13h31m55.747884064s) [upload=14041]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:55.512693945 (13h32m55.102038922s) [upload=14041]: Done uploading segments... Aug 7 23:12:55.512714414 (13h32m55.102059401s) [upload=14041]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:04.863194286 (13h33m4.452539303s) [upload=14061]: Starting upload Aug 7 23:13:04.863346336 (13h33m4.452691313s) [upload=14061]: Done starting upload Aug 7 23:13:04.863414577 (13h33m4.452759614s) [upload=14061]: Uploading segments... Aug 7 23:13:04.863490322 (13h33m4.452835309s) [upload=14061]: Got next segment. Inline: true Aug 7 23:13:04.863497876 (13h33m4.452842863s) [upload=14061]: Waiting for error group managing segments... Aug 7 23:13:04.863564063 (13h33m4.45290905s) [upload=14061]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:04.863569523 (13h33m4.45291451s) [upload=14061]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:04.863571978 (13h33m4.452916965s) [upload=14061]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:04.218769699 (13h34m3.808114676s) [upload=14061]: Done uploading segments... Aug 7 23:14:04.218789687 (13h34m3.808134664s) [upload=14061]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:05.097995389 (13h34m4.687340376s) [upload=14076]: Starting upload Aug 7 23:14:05.098050264 (13h34m4.687395251s) [upload=14076]: Done starting upload Aug 7 23:14:05.098107544 (13h34m4.687452521s) [upload=14076]: Uploading segments... Aug 7 23:14:05.098195913 (13h34m4.6875409s) [upload=14076]: Got next segment. Inline: true Aug 7 23:14:05.098203497 (13h34m4.687548484s) [upload=14076]: Waiting for error group managing segments... Aug 7 23:14:05.098273151 (13h34m4.687618138s) [upload=14076]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:05.098281256 (13h34m4.687626253s) [upload=14076]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:05.098285083 (13h34m4.68763007s) [upload=14076]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:04.449929685 (13h35m4.039274672s) [upload=14076]: Done uploading segments... Aug 7 23:15:04.449952750 (13h35m4.039297737s) [upload=14076]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:05.510260535 (13h35m5.099605522s) [upload=14092]: Starting upload Aug 7 23:15:05.510334587 (13h35m5.099679574s) [upload=14092]: Done starting upload Aug 7 23:15:05.510383931 (13h35m5.099728918s) [upload=14092]: Uploading segments... Aug 7 23:15:05.510421683 (13h35m5.09976667s) [upload=14092]: Got next segment. Inline: true Aug 7 23:15:05.510425931 (13h35m5.099770918s) [upload=14092]: Waiting for error group managing segments... Aug 7 23:15:05.510461179 (13h35m5.099806166s) [upload=14092]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:05.510464996 (13h35m5.099809984s) [upload=14092]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:05.510467561 (13h35m5.099812548s) [upload=14092]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:04.861169851 (13h36m4.450514858s) [upload=14092]: Done uploading segments... Aug 7 23:16:04.861192945 (13h36m4.450537942s) [upload=14092]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:05.761665335 (13h36m5.351010322s) [upload=14104]: Starting upload Aug 7 23:16:05.761743936 (13h36m5.351088913s) [upload=14104]: Done starting upload Aug 7 23:16:05.761859217 (13h36m5.351204194s) [upload=14104]: Uploading segments... Aug 7 23:16:05.761910234 (13h36m5.351255221s) [upload=14104]: Got next segment. Inline: true Aug 7 23:16:05.761914483 (13h36m5.35125947s) [upload=14104]: Waiting for error group managing segments... Aug 7 23:16:05.761947837 (13h36m5.351292814s) [upload=14104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:05.761951704 (13h36m5.351296691s) [upload=14104]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:05.761955882 (13h36m5.351300859s) [upload=14104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:05.106651963 (13h37m4.69599695s) [upload=14104]: Done uploading segments... Aug 7 23:17:05.106703883 (13h37m4.69604887s) [upload=14104]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.882437604s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [...............................................XXXXXX.X.X...] duration: 6.059009691s FAIL (2025-08-07 23:24:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.059031625s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 4 out of 60 is failed (threshold=3) [...........................................X...X..X.X.......] duration: 1.871456983s FAIL (2025-08-07 23:24:01 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.871484297s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 5 out of 60 is failed (threshold=3) [...............................................XXXXX........] duration: 47.900011711s FAIL (2025-08-07 23:23:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== d. err: <nil> Aug 7 23:07:52.874987115 (13h27m52.464332102s) [upload=13971]: Starting upload Aug 7 23:07:52.875039626 (13h27m52.464384603s) [upload=13971]: Done starting upload Aug 7 23:07:52.875078921 (13h27m52.464423908s) [upload=13971]: Uploading segments... Aug 7 23:07:52.875111583 (13h27m52.46445657s) [upload=13971]: Got next segment. Inline: true Aug 7 23:07:52.875115000 (13h27m52.464459987s) [upload=13971]: Waiting for error group managing segments... Aug 7 23:07:52.875144226 (13h27m52.464489203s) [upload=13971]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:52.875147783 (13h27m52.46449277s) [upload=13971]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:07:52.875150268 (13h27m52.464495255s) [upload=13971]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:53.152523417 (13h27m52.741868404s) [upload=13971]: Done uploading segments... Aug 7 23:07:53.152567841 (13h27m52.741912828s) [upload=13971]: Upload finished. err: <nil> Aug 7 23:08:13.160315277 (13h28m12.749660264s) [upload=13978]: Starting upload Aug 7 23:08:13.160372306 (13h28m12.749717293s) [upload=13978]: Done starting upload Aug 7 23:08:13.160419306 (13h28m12.749764293s) [upload=13978]: Uploading segments... Aug 7 23:08:13.160446879 (13h28m12.749791866s) [upload=13978]: Got next segment. Inline: true Aug 7 23:08:13.160450095 (13h28m12.749795082s) [upload=13978]: Waiting for error group managing segments... Aug 7 23:08:13.160481565 (13h28m12.749826542s) [upload=13978]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:08:13.160485202 (13h28m12.749830189s) [upload=13978]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:08:13.160487416 (13h28m12.749832403s) [upload=13978]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:08:13.673421309 (13h28m13.262766296s) [upload=13978]: Done uploading segments... Aug 7 23:08:13.673439774 (13h28m13.262784761s) [upload=13978]: Upload finished. err: <nil> Aug 7 23:09:08.585854499 (13h29m8.175199486s) [upload=13993]: Starting upload Aug 7 23:09:08.585909965 (13h29m8.175254952s) [upload=13993]: Done starting upload Aug 7 23:09:08.585953217 (13h29m8.175298204s) [upload=13993]: Uploading segments... Aug 7 23:09:08.585980830 (13h29m8.175325817s) [upload=13993]: Got next segment. Inline: true Aug 7 23:09:08.585984206 (13h29m8.175329183s) [upload=13993]: Waiting for error group managing segments... Aug 7 23:09:08.586014684 (13h29m8.175359671s) [upload=13993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:08.586018372 (13h29m8.175363359s) [upload=13993]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:08.586020767 (13h29m8.175365754s) [upload=13993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:08.854829559 (13h29m8.444174546s) [upload=13993]: Done uploading segments... Aug 7 23:09:08.854846763 (13h29m8.44419175s) [upload=13993]: Upload finished. err: <nil> Aug 7 23:10:03.068961384 (13h30m2.658306371s) [upload=14012]: Starting upload Aug 7 23:10:03.069021258 (13h30m2.658366245s) [upload=14012]: Done starting upload Aug 7 23:10:03.069094389 (13h30m2.658439376s) [upload=14012]: Uploading segments... Aug 7 23:10:03.069128504 (13h30m2.658473491s) [upload=14012]: Got next segment. Inline: true Aug 7 23:10:03.069132782 (13h30m2.658477779s) [upload=14012]: Waiting for error group managing segments... Aug 7 23:10:03.069170935 (13h30m2.658515932s) [upload=14012]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:03.069175905 (13h30m2.658520892s) [upload=14012]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:03.069178650 (13h30m2.658523647s) [upload=14012]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:03.364714047 (13h30m2.954059064s) [upload=14012]: Done uploading segments... Aug 7 23:10:03.364761748 (13h30m2.954106745s) [upload=14012]: Upload finished. err: <nil> Aug 7 23:10:44.905335214 (13h30m44.494680201s) [upload=14022]: Starting upload Aug 7 23:10:44.905402192 (13h30m44.494747179s) [upload=14022]: Done starting upload Aug 7 23:10:44.905449983 (13h30m44.49479496s) [upload=14022]: Uploading segments... Aug 7 23:10:44.905479650 (13h30m44.494824628s) [upload=14022]: Got next segment. Inline: true Aug 7 23:10:44.905482787 (13h30m44.494827774s) [upload=14022]: Waiting for error group managing segments... Aug 7 23:10:44.905516160 (13h30m44.494861147s) [upload=14022]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:44.905519717 (13h30m44.494864694s) [upload=14022]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:44.905522222 (13h30m44.494867209s) [upload=14022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:45.196197762 (13h30m44.785542749s) [upload=14022]: Done uploading segments... Aug 7 23:10:45.196218401 (13h30m44.785563389s) [upload=14022]: Upload finished. err: <nil> Aug 7 23:11:56.158319777 (13h31m55.747664764s) [upload=14041]: Starting upload Aug 7 23:11:56.158381214 (13h31m55.747726201s) [upload=14041]: Done starting upload Aug 7 23:11:56.158454435 (13h31m55.747799422s) [upload=14041]: Uploading segments... Aug 7 23:11:56.158485985 (13h31m55.747830972s) [upload=14041]: Got next segment. Inline: true Aug 7 23:11:56.158489752 (13h31m55.747834739s) [upload=14041]: Waiting for error group managing segments... Aug 7 23:11:56.158533225 (13h31m55.747878212s) [upload=14041]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:56.158536933 (13h31m55.74788192s) [upload=14041]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:56.158539077 (13h31m55.747884064s) [upload=14041]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:55.512693945 (13h32m55.102038922s) [upload=14041]: Done uploading segments... Aug 7 23:12:55.512714414 (13h32m55.102059401s) [upload=14041]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:04.863194286 (13h33m4.452539303s) [upload=14061]: Starting upload Aug 7 23:13:04.863346336 (13h33m4.452691313s) [upload=14061]: Done starting upload Aug 7 23:13:04.863414577 (13h33m4.452759614s) [upload=14061]: Uploading segments... Aug 7 23:13:04.863490322 (13h33m4.452835309s) [upload=14061]: Got next segment. Inline: true Aug 7 23:13:04.863497876 (13h33m4.452842863s) [upload=14061]: Waiting for error group managing segments... Aug 7 23:13:04.863564063 (13h33m4.45290905s) [upload=14061]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:04.863569523 (13h33m4.45291451s) [upload=14061]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:04.863571978 (13h33m4.452916965s) [upload=14061]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:04.218769699 (13h34m3.808114676s) [upload=14061]: Done uploading segments... Aug 7 23:14:04.218789687 (13h34m3.808134664s) [upload=14061]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:05.097995389 (13h34m4.687340376s) [upload=14076]: Starting upload Aug 7 23:14:05.098050264 (13h34m4.687395251s) [upload=14076]: Done starting upload Aug 7 23:14:05.098107544 (13h34m4.687452521s) [upload=14076]: Uploading segments... Aug 7 23:14:05.098195913 (13h34m4.6875409s) [upload=14076]: Got next segment. Inline: true Aug 7 23:14:05.098203497 (13h34m4.687548484s) [upload=14076]: Waiting for error group managing segments... Aug 7 23:14:05.098273151 (13h34m4.687618138s) [upload=14076]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:05.098281256 (13h34m4.687626253s) [upload=14076]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:05.098285083 (13h34m4.68763007s) [upload=14076]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:04.449929685 (13h35m4.039274672s) [upload=14076]: Done uploading segments... Aug 7 23:15:04.449952750 (13h35m4.039297737s) [upload=14076]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:05.510260535 (13h35m5.099605522s) [upload=14092]: Starting upload Aug 7 23:15:05.510334587 (13h35m5.099679574s) [upload=14092]: Done starting upload Aug 7 23:15:05.510383931 (13h35m5.099728918s) [upload=14092]: Uploading segments... Aug 7 23:15:05.510421683 (13h35m5.09976667s) [upload=14092]: Got next segment. Inline: true Aug 7 23:15:05.510425931 (13h35m5.099770918s) [upload=14092]: Waiting for error group managing segments... Aug 7 23:15:05.510461179 (13h35m5.099806166s) [upload=14092]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:05.510464996 (13h35m5.099809984s) [upload=14092]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:05.510467561 (13h35m5.099812548s) [upload=14092]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:04.861169851 (13h36m4.450514858s) [upload=14092]: Done uploading segments... Aug 7 23:16:04.861192945 (13h36m4.450537942s) [upload=14092]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:05.761665335 (13h36m5.351010322s) [upload=14104]: Starting upload Aug 7 23:16:05.761743936 (13h36m5.351088913s) [upload=14104]: Done starting upload Aug 7 23:16:05.761859217 (13h36m5.351204194s) [upload=14104]: Uploading segments... Aug 7 23:16:05.761910234 (13h36m5.351255221s) [upload=14104]: Got next segment. Inline: true Aug 7 23:16:05.761914483 (13h36m5.35125947s) [upload=14104]: Waiting for error group managing segments... Aug 7 23:16:05.761947837 (13h36m5.351292814s) [upload=14104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:05.761951704 (13h36m5.351296691s) [upload=14104]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:05.761955882 (13h36m5.351300859s) [upload=14104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:05.106651963 (13h37m4.69599695s) [upload=14104]: Done uploading segments... Aug 7 23:17:05.106703883 (13h37m4.69604887s) [upload=14104]: Upload finished. err: metaclient: context deadline exceeded Duration: 47.900039815s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X......X..........] duration: 1.84379972s FAIL (2025-08-07 23:23:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== 30.892541890 (13h31m30.481886877s) [upload=14032]: Upload finished. err: <nil> Aug 7 23:11:48.390915893 (13h31m47.98026088s) [upload=14038]: Starting upload Aug 7 23:11:48.390965198 (13h31m47.980310185s) [upload=14038]: Done starting upload Aug 7 23:11:48.391014242 (13h31m47.980359229s) [upload=14038]: Uploading segments... Aug 7 23:11:48.391054007 (13h31m47.980399004s) [upload=14038]: Got next segment. Inline: true Aug 7 23:11:48.391057494 (13h31m47.980402471s) [upload=14038]: Waiting for error group managing segments... Aug 7 23:11:48.391089254 (13h31m47.980434231s) [upload=14038]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:48.391093213 (13h31m47.98043819s) [upload=14038]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:48.391095457 (13h31m47.980440444s) [upload=14038]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:48.701950724 (13h31m48.291295711s) [upload=14038]: Done uploading segments... Aug 7 23:11:48.701969449 (13h31m48.291314436s) [upload=14038]: Upload finished. err: <nil> Aug 7 23:12:06.430245154 (13h32m6.019590141s) [upload=14048]: Starting upload Aug 7 23:12:06.430293076 (13h32m6.019638063s) [upload=14048]: Done starting upload Aug 7 23:12:06.430361757 (13h32m6.019706745s) [upload=14048]: Uploading segments... Aug 7 23:12:06.430459223 (13h32m6.01980421s) [upload=14048]: Got next segment. Inline: true Aug 7 23:12:06.430464543 (13h32m6.01980953s) [upload=14048]: Waiting for error group managing segments... Aug 7 23:12:06.430527555 (13h32m6.019872542s) [upload=14048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:06.430533796 (13h32m6.019878783s) [upload=14048]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:06.430536692 (13h32m6.019881679s) [upload=14048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:06.729952062 (13h32m6.319297049s) [upload=14048]: Done uploading segments... Aug 7 23:12:06.729968844 (13h32m6.319313832s) [upload=14048]: Upload finished. err: <nil> Aug 7 23:12:37.551701338 (13h32m37.141046325s) [upload=14055]: Starting upload Aug 7 23:12:37.551746985 (13h32m37.141091972s) [upload=14055]: Done starting upload Aug 7 23:12:37.551807301 (13h32m37.141152288s) [upload=14055]: Uploading segments... Aug 7 23:12:37.551914826 (13h32m37.141259833s) [upload=14055]: Got next segment. Inline: true Aug 7 23:12:37.551921779 (13h32m37.141266766s) [upload=14055]: Waiting for error group managing segments... Aug 7 23:12:37.551977145 (13h32m37.141322122s) [upload=14055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.551984059 (13h32m37.141329086s) [upload=14055]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.551988918 (13h32m37.141333936s) [upload=14055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.767068918 (13h32m37.356413905s) [upload=14055]: Done uploading segments... Aug 7 23:12:37.767104365 (13h32m37.356449342s) [upload=14055]: Upload finished. err: metaclient: Too Many Requests Aug 7 23:13:47.522489627 (13h33m47.111834614s) [upload=14072]: Starting upload Aug 7 23:13:47.522559681 (13h33m47.111904668s) [upload=14072]: Done starting upload Aug 7 23:13:47.522629674 (13h33m47.111974661s) [upload=14072]: Uploading segments... Aug 7 23:13:47.522726771 (13h33m47.112071778s) [upload=14072]: Got next segment. Inline: true Aug 7 23:13:47.522735617 (13h33m47.112080614s) [upload=14072]: Waiting for error group managing segments... Aug 7 23:13:47.522790582 (13h33m47.112135569s) [upload=14072]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:47.522796493 (13h33m47.112141481s) [upload=14072]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:47.522799320 (13h33m47.112144307s) [upload=14072]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:47.796144342 (13h33m47.385489329s) [upload=14072]: Done uploading segments... Aug 7 23:13:47.796162656 (13h33m47.385507643s) [upload=14072]: Upload finished. err: <nil> Aug 7 23:14:22.239921861 (13h34m21.829266848s) [upload=14081]: Starting upload Aug 7 23:14:22.239973791 (13h34m21.829318778s) [upload=14081]: Done starting upload Aug 7 23:14:22.240087399 (13h34m21.829432396s) [upload=14081]: Uploading segments... Aug 7 23:14:22.240187159 (13h34m21.829532156s) [upload=14081]: Got next segment. Inline: true Aug 7 23:14:22.240194182 (13h34m21.829539179s) [upload=14081]: Waiting for error group managing segments... Aug 7 23:14:22.240255971 (13h34m21.829600958s) [upload=14081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:22.240261311 (13h34m21.829606288s) [upload=14081]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:22.240263816 (13h34m21.829608803s) [upload=14081]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:22.537482123 (13h34m22.12682711s) [upload=14081]: Done uploading segments... Aug 7 23:14:22.537500677 (13h34m22.126845664s) [upload=14081]: Upload finished. err: <nil> Aug 7 23:14:56.968922831 (13h34m56.558267818s) [upload=14090]: Starting upload Aug 7 23:14:56.968974069 (13h34m56.558319056s) [upload=14090]: Done starting upload Aug 7 23:14:56.969028383 (13h34m56.55837337s) [upload=14090]: Uploading segments... Aug 7 23:14:56.969059602 (13h34m56.55840459s) [upload=14090]: Got next segment. Inline: true Aug 7 23:14:56.969063500 (13h34m56.558408487s) [upload=14090]: Waiting for error group managing segments... Aug 7 23:14:56.969102906 (13h34m56.558447893s) [upload=14090]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:56.969107034 (13h34m56.558452021s) [upload=14090]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:56.969109899 (13h34m56.558454886s) [upload=14090]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:57.268863065 (13h34m56.858208052s) [upload=14090]: Done uploading segments... Aug 7 23:14:57.268882201 (13h34m56.858227188s) [upload=14090]: Upload finished. err: <nil> Aug 7 23:16:09.138990771 (13h36m8.728335758s) [upload=14107]: Starting upload Aug 7 23:16:09.139048231 (13h36m8.728393208s) [upload=14107]: Done starting upload Aug 7 23:16:09.139094229 (13h36m8.728439226s) [upload=14107]: Uploading segments... Aug 7 23:16:09.139131069 (13h36m8.728476056s) [upload=14107]: Got next segment. Inline: true Aug 7 23:16:09.139135007 (13h36m8.728479994s) [upload=14107]: Waiting for error group managing segments... Aug 7 23:16:09.139167859 (13h36m8.728512846s) [upload=14107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:09.139171937 (13h36m8.728516924s) [upload=14107]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:09.139199730 (13h36m8.728544707s) [upload=14107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:09.409287150 (13h36m8.998632137s) [upload=14107]: Done uploading segments... Aug 7 23:16:09.409303952 (13h36m8.998648939s) [upload=14107]: Upload finished. err: <nil> Aug 7 23:16:30.665596455 (13h36m30.254941442s) [upload=14111]: Starting upload Aug 7 23:16:30.665642382 (13h36m30.254987369s) [upload=14111]: Done starting upload Aug 7 23:16:30.665687749 (13h36m30.255032726s) [upload=14111]: Uploading segments... Aug 7 23:16:30.665725801 (13h36m30.255070788s) [upload=14111]: Got next segment. Inline: true Aug 7 23:16:30.665729409 (13h36m30.255074386s) [upload=14111]: Waiting for error group managing segments... Aug 7 23:16:30.665763554 (13h36m30.255108541s) [upload=14111]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:30.665768984 (13h36m30.255113971s) [upload=14111]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:30.665771289 (13h36m30.255116276s) [upload=14111]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:30.945395745 (13h36m30.534740732s) [upload=14111]: Done uploading segments... Aug 7 23:16:30.945418538 (13h36m30.534763526s) [upload=14111]: Upload finished. err: <nil> Aug 7 23:17:25.677744155 (13h37m25.267089143s) [upload=14125]: Starting upload Aug 7 23:17:25.677793930 (13h37m25.267138917s) [upload=14125]: Done starting upload Aug 7 23:17:25.677862562 (13h37m25.267207539s) [upload=14125]: Uploading segments... Aug 7 23:17:25.677896967 (13h37m25.267241954s) [upload=14125]: Got next segment. Inline: true Aug 7 23:17:25.677900704 (13h37m25.267245681s) [upload=14125]: Waiting for error group managing segments... Aug 7 23:17:25.677933046 (13h37m25.267278033s) [upload=14125]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:25.677936884 (13h37m25.267281871s) [upload=14125]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:25.677939309 (13h37m25.267284296s) [upload=14125]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:26.502642065 (13h37m26.091987052s) [upload=14125]: Done uploading segments... Aug 7 23:17:26.502660942 (13h37m26.092005919s) [upload=14125]: Upload finished. err: <nil> Aug 7 23:17:53.025114325 (13h37m52.614459313s) [upload=14131]: Starting upload Aug 7 23:17:53.025169641 (13h37m52.614514618s) [upload=14131]: Done starting upload Aug 7 23:17:53.025219857 (13h37m52.614564844s) [upload=14131]: Uploading segments... Aug 7 23:17:53.025267939 (13h37m52.614612936s) [upload=14131]: Got next segment. Inline: true Aug 7 23:17:53.025271916 (13h37m52.614616903s) [upload=14131]: Waiting for error group managing segments... Aug 7 23:17:53.025302495 (13h37m52.614647482s) [upload=14131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:53.025306382 (13h37m52.614651369s) [upload=14131]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:53.025308497 (13h37m52.614653484s) [upload=14131]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:53.302835519 (13h37m52.892180506s) [upload=14131]: Done uploading segments... Aug 7 23:17:53.302854656 (13h37m52.892199633s) [upload=14131]: Upload finished. err: metaclient: Too Many Requests Duration: 1.843828094s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.................................................X.....X.X..] duration: 2.264264728s FAIL (2025-08-07 23:23:53 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.26428645s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.............................................XXXX.X..XX.....] duration: 6.245234965s FAIL (2025-08-07 23:23:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.245256136s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X...................X....................X............] duration: 7.640460897s FAIL (2025-08-07 23:23:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after e85db788dbeec38ad8084daa355e1bf59b8b718af998fc40c7451ca1ba6627b2 != before 1cad814b0341363008cdb7d22dd66bb6d87f0105cd39850888ced4405a0b4b43 ======LOGS====== upload Aug 7 23:13:43.967401320 (13h33m43.556746307s) [upload=14069]: Uploading segments... Aug 7 23:13:43.967498206 (13h33m43.556843193s) [upload=14069]: Got next segment. Inline: false Aug 7 23:13:43.967560455 (13h33m43.556905442s) [upload=14069]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:43.967568550 (13h33m43.556913577s) [upload=14069]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:13:44.414536988 (13h33m44.003881975s) [seg_pos={0 0}, upload=14069]: Begin upload segment... Aug 7 23:13:44.414587104 (13h33m44.003932091s) [seg_pos={0 0}, upload=14069]: Done begin upload segment. Aug 7 23:13:44.414606933 (13h33m44.00395192s) [noise=true , address=104.36.82.60:4362 , piece_id=LJMWWXQ4NXJY54S7WPOSCO37ALDF5PT2GEC5LKOAKB6YEW2PLEAA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.414666236 (13h33m44.004011223s) [noise=true , address=104.36.82.43:4868 , piece_id=TOYHNT5XLP5YVN6ZFSWOS4UJOSIIOKUKWXP4CNSG56CHGEUZFQDQ, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.414966160 (13h33m44.004311147s) [noise=true , address=104.36.82.52:4086 , piece_id=DRDBLVU5QJKKQC5WCUJROGJYG7OQKTIRVAXDLFHQW3FIC2BV2WIA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.415118381 (13h33m44.004463369s) [noise=true , address=104.36.82.16:4296 , piece_id=HCYT2XGA5RGQQGMWKTDKJIF6OQWK6NR4Z4SUPSG5IOA6ECK3EYPQ, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.415228682 (13h33m44.00457367s) [noise=true , address=79.127.205.233:4621 , piece_id=DSV45XV43BCLRGH7BJOICWBBDBCYFZXDC4CVACYRNT4LI745DK7Q, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.415496494 (13h33m44.004841471s) [noise=true , address=104.36.82.58:4291 , piece_id=DP64RXRSK2NLFM3N7TSZCZL4PEGZVHEQJHPJFXAIQXXLOEPQWY5Q, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.415745370 (13h33m44.005090357s) [noise=true , address=45.91.116.13:4158 , piece_id=62VB7AEIWCPAVJSLX3P2TOQRSALK5A6ZAD3ZFVQINR2IVT3AOXXQ, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.416026788 (13h33m44.005371766s) [noise=true , address=104.36.82.36:4642 , piece_id=3X7E6QOZKZ5KGPLWHIWL6WZRGOFHKZLFIZRKVZM3CGJM6VLA6OAA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.416256569 (13h33m44.005601556s) [noise=true , address=104.36.82.14:4226 , piece_id=CLREJCNEC4WX6IGMUEJIV2IEZXFAVLSXLZVYMWJJBHHDHCQ6DVQA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.416456080 (13h33m44.005801067s) [noise=true , address=64.49.13.137:4499 , piece_id=7D5JZSVLNEFT26CX3TBD4L27WIZU2IWH2JK3O7IHPAVDCUDBZAJQ, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.416552746 (13h33m44.005897753s) [noise=true , address=45.91.116.14:4194 , piece_id=2LO5TLHD4ZILDKAMW6EXNTMDHG4PCRFYS6F5EMLUN4EBP4RWEUPA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.416803596 (13h33m44.006148583s) [noise=true , address=45.91.116.11:4150 , piece_id=2YORTWP2L67BV5ENJQ5CKPY3NK337SK6WM5XL77RUZ6LMQWXVORA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.417042221 (13h33m44.006387208s) [noise=true , address=104.36.82.53:4126 , piece_id=A6LKRBLYVWJ7QN4UJ7JPDXKLCI5ZFAJ6I6ICCXDE7V7G2F5R65UA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.417277101 (13h33m44.006622088s) [noise=true , address=104.36.82.59:4332 , piece_id=K5TGW4E726UEFXZQSKWGYU2PHFX2IYE2BYKHXUG2WQXTZV5GV5ZA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.417316085 (13h33m44.006661072s) [noise=true , address=104.36.82.8:4977 , piece_id=PDXPXV3IJ7Y4P25A6BS3HTKCYLLNMRYIHAMLJCVV3QDMHGN4PXKQ, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.417537459 (13h33m44.006882436s) [noise=true , address=104.36.82.2:4757 , piece_id=LJ3JPH3HYJ7Z5QIWGSKWC3KGH7QJXYT3IDTFEN326A6QOLGAWVZA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.417738975 (13h33m44.007083952s) [noise=true , address=64.49.13.26:4946 , piece_id=VYQT7NJAPN6XN6Z5PKEBJCPSYWRZHR5ZCGG6257JA3Q2BTK6S4FA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.417952763 (13h33m44.00729774s) [noise=true , address=104.36.82.9:4037 , piece_id=RISOPBKZ6Y3UYM3HSBMJT6EI3ZXGQKGANDYNUSDK3IGPCNLC4QTQ, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:45.514309387 (13h33m45.103654374s) [upload=14069]: Next returned nil segment Aug 7 23:13:45.514327020 (13h33m45.103672007s) [upload=14069]: Waiting for error group managing segments... Aug 7 23:13:45.795258156 (13h33m45.384603143s) [noise=true , address=64.49.13.137:4499 , piece_id=7D5JZSVLNEFT26CX3TBD4L27WIZU2IWH2JK3O7IHPAVDCUDBZAJQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:45.802263469 (13h33m45.391608456s) [noise=true , address=79.127.205.233:4621 , piece_id=DSV45XV43BCLRGH7BJOICWBBDBCYFZXDC4CVACYRNT4LI745DK7Q, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:45.887877424 (13h33m45.477222401s) [noise=true , address=64.49.13.26:4946 , piece_id=VYQT7NJAPN6XN6Z5PKEBJCPSYWRZHR5ZCGG6257JA3Q2BTK6S4FA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.350209185 (13h33m45.939554172s) [noise=true , address=104.36.82.60:4362 , piece_id=LJMWWXQ4NXJY54S7WPOSCO37ALDF5PT2GEC5LKOAKB6YEW2PLEAA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.389672887 (13h33m45.979017874s) [noise=true , address=104.36.82.52:4086 , piece_id=DRDBLVU5QJKKQC5WCUJROGJYG7OQKTIRVAXDLFHQW3FIC2BV2WIA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.399409194 (13h33m45.988754171s) [noise=true , address=104.36.82.58:4291 , piece_id=DP64RXRSK2NLFM3N7TSZCZL4PEGZVHEQJHPJFXAIQXXLOEPQWY5Q, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.424971119 (13h33m46.014316106s) [noise=true , address=104.36.82.36:4642 , piece_id=3X7E6QOZKZ5KGPLWHIWL6WZRGOFHKZLFIZRKVZM3CGJM6VLA6OAA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.450918913 (13h33m46.0402639s) [noise=true , address=45.91.116.13:4158 , piece_id=62VB7AEIWCPAVJSLX3P2TOQRSALK5A6ZAD3ZFVQINR2IVT3AOXXQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.492609174 (13h33m46.081954161s) [noise=true , address=104.36.82.53:4126 , piece_id=A6LKRBLYVWJ7QN4UJ7JPDXKLCI5ZFAJ6I6ICCXDE7V7G2F5R65UA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.516716376 (13h33m46.106061353s) [noise=true , address=104.36.82.9:4037 , piece_id=RISOPBKZ6Y3UYM3HSBMJT6EI3ZXGQKGANDYNUSDK3IGPCNLC4QTQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.592315332 (13h33m46.181660319s) [noise=true , address=104.36.82.43:4868 , piece_id=TOYHNT5XLP5YVN6ZFSWOS4UJOSIIOKUKWXP4CNSG56CHGEUZFQDQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.746385819 (13h33m46.335730806s) [noise=true , address=45.91.116.11:4150 , piece_id=2YORTWP2L67BV5ENJQ5CKPY3NK337SK6WM5XL77RUZ6LMQWXVORA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.786539771 (13h33m46.375884758s) [noise=true , address=104.36.82.59:4332 , piece_id=K5TGW4E726UEFXZQSKWGYU2PHFX2IYE2BYKHXUG2WQXTZV5GV5ZA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.838001699 (13h33m46.427346676s) [noise=true , address=45.91.116.14:4194 , piece_id=2LO5TLHD4ZILDKAMW6EXNTMDHG4PCRFYS6F5EMLUN4EBP4RWEUPA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.838017648 (13h33m46.427362636s) [seg_pos={0 0}, upload=14069]: Segment reached optimal threshold of 14 pieces. Aug 7 23:13:46.838217681 (13h33m46.427562658s) [noise=true , address=104.36.82.14:4226 , piece_id=CLREJCNEC4WX6IGMUEJIV2IEZXFAVLSXLZVYMWJJBHHDHCQ6DVQA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126xWe3ELGmYQfSUk3tYtGTSa6LnGfDNWcGT164pR6tHEtFPAFt): protocol: expected piece hash; context canceled; EOF Aug 7 23:13:46.838422624 (13h33m46.427767601s) [noise=true , address=104.36.82.2:4757 , piece_id=LJ3JPH3HYJ7Z5QIWGSKWC3KGH7QJXYT3IDTFEN326A6QOLGAWVZA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q): protocol: expected piece hash; context canceled; EOF Aug 7 23:13:46.838575365 (13h33m46.427920352s) [noise=true , address=104.36.82.16:4296 , piece_id=HCYT2XGA5RGQQGMWKTDKJIF6OQWK6NR4Z4SUPSG5IOA6ECK3EYPQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cKhDbA4ptwux2LYNPNAJ7Qh8jECsGWd1KWaK6T6pD38bJWdt1): protocol: rpc: dial tcp 104.36.82.16:4296: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:13:46.838627195 (13h33m46.427972172s) [noise=true , address=104.36.82.8:4977 , piece_id=PDXPXV3IJ7Y4P25A6BS3HTKCYLLNMRYIHAMLJCVV3QDMHGN4PXKQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WCAm12tXwimFSPqz7Seh6A4oxWqQVxqsF459xux4a5CqjpbAT): protocol: write tcp 188.166.225.170:48060->104.36.82.8:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:13:46.838635551 (13h33m46.427980538s) [seg_pos={0 0}, upload=14069]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:13:46.838702128 (13h33m46.428047115s) [upload=14069]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:13:46.838705895 (13h33m46.428050882s) [upload=14069]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:47.127771955 (13h33m46.717116932s) [upload=14069]: Done uploading segments... Aug 7 23:13:47.127791082 (13h33m46.717136069s) [upload=14069]: Upload finished. err: <nil> Duration: 7.640491686s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 4 out of 60 is failed (threshold=3) [................................................XXXX........] duration: 1.899347083s FAIL (2025-08-07 23:23:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .175884539 (13h29m53.765229526s) [upload=14007]: Starting upload Aug 7 23:09:54.175937640 (13h29m53.765282618s) [upload=14007]: Done starting upload Aug 7 23:09:54.175979842 (13h29m53.765324829s) [upload=14007]: Uploading segments... Aug 7 23:09:54.176041389 (13h29m53.765386376s) [upload=14007]: Got next segment. Inline: true Aug 7 23:09:54.176045487 (13h29m53.765390474s) [upload=14007]: Waiting for error group managing segments... Aug 7 23:09:54.176077538 (13h29m53.765422525s) [upload=14007]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:54.176082318 (13h29m53.765427305s) [upload=14007]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:54.176084522 (13h29m53.765429509s) [upload=14007]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:54.453731274 (13h29m54.043076261s) [upload=14007]: Done uploading segments... Aug 7 23:09:54.453750932 (13h29m54.043095909s) [upload=14007]: Upload finished. err: <nil> Aug 7 23:10:39.330325211 (13h30m38.919670198s) [upload=14021]: Starting upload Aug 7 23:10:39.330417358 (13h30m38.919762345s) [upload=14021]: Done starting upload Aug 7 23:10:39.330504384 (13h30m38.919849371s) [upload=14021]: Uploading segments... Aug 7 23:10:39.330543368 (13h30m38.919888355s) [upload=14021]: Got next segment. Inline: true Aug 7 23:10:39.330546675 (13h30m38.919891662s) [upload=14021]: Waiting for error group managing segments... Aug 7 23:10:39.330581371 (13h30m38.919926349s) [upload=14021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:39.330585309 (13h30m38.919930286s) [upload=14021]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:39.330587663 (13h30m38.91993265s) [upload=14021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:39.610133078 (13h30m39.199478065s) [upload=14021]: Done uploading segments... Aug 7 23:10:39.610150061 (13h30m39.199495048s) [upload=14021]: Upload finished. err: <nil> Aug 7 23:11:32.695175239 (13h31m32.284520226s) [upload=14034]: Starting upload Aug 7 23:11:32.695285580 (13h31m32.284630567s) [upload=14034]: Done starting upload Aug 7 23:11:32.695336297 (13h31m32.284681284s) [upload=14034]: Uploading segments... Aug 7 23:11:32.695365753 (13h31m32.28471074s) [upload=14034]: Got next segment. Inline: true Aug 7 23:11:32.695369610 (13h31m32.284714598s) [upload=14034]: Waiting for error group managing segments... Aug 7 23:11:32.695401581 (13h31m32.284746569s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:32.695405168 (13h31m32.284750155s) [upload=14034]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:32.695407543 (13h31m32.28475253s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:32.954750627 (13h31m32.544095614s) [upload=14034]: Done uploading segments... Aug 7 23:11:32.954767650 (13h31m32.544112627s) [upload=14034]: Upload finished. err: <nil> Aug 7 23:12:37.502051138 (13h32m37.091396125s) [upload=14054]: Starting upload Aug 7 23:12:37.502104370 (13h32m37.091449357s) [upload=14054]: Done starting upload Aug 7 23:12:37.502150869 (13h32m37.091495856s) [upload=14054]: Uploading segments... Aug 7 23:12:37.502182519 (13h32m37.091527506s) [upload=14054]: Got next segment. Inline: true Aug 7 23:12:37.502185986 (13h32m37.091530963s) [upload=14054]: Waiting for error group managing segments... Aug 7 23:12:37.502216213 (13h32m37.0915612s) [upload=14054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.502219780 (13h32m37.091564767s) [upload=14054]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.502222425 (13h32m37.091567412s) [upload=14054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.772497714 (13h32m37.361842701s) [upload=14054]: Done uploading segments... Aug 7 23:12:37.772511151 (13h32m37.361856138s) [upload=14054]: Upload finished. err: <nil> Aug 7 23:13:44.834965069 (13h33m44.424310046s) [upload=14070]: Starting upload Aug 7 23:13:44.835030293 (13h33m44.42437528s) [upload=14070]: Done starting upload Aug 7 23:13:44.835093484 (13h33m44.424438471s) [upload=14070]: Uploading segments... Aug 7 23:13:44.835133321 (13h33m44.424478308s) [upload=14070]: Got next segment. Inline: true Aug 7 23:13:44.835136998 (13h33m44.424481985s) [upload=14070]: Waiting for error group managing segments... Aug 7 23:13:44.835214376 (13h33m44.424559363s) [upload=14070]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:44.835220728 (13h33m44.424565715s) [upload=14070]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:44.835222972 (13h33m44.424567959s) [upload=14070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:45.220889346 (13h33m44.810234343s) [upload=14070]: Done uploading segments... Aug 7 23:13:45.220937208 (13h33m44.810282205s) [upload=14070]: Upload finished. err: <nil> Aug 7 23:14:39.930436267 (13h34m39.519781254s) [upload=14085]: Starting upload Aug 7 23:14:39.930523634 (13h34m39.519868621s) [upload=14085]: Done starting upload Aug 7 23:14:39.930574131 (13h34m39.519919118s) [upload=14085]: Uploading segments... Aug 7 23:14:39.930614779 (13h34m39.519959766s) [upload=14085]: Got next segment. Inline: true Aug 7 23:14:39.930618847 (13h34m39.519963824s) [upload=14085]: Waiting for error group managing segments... Aug 7 23:14:39.930656519 (13h34m39.520001506s) [upload=14085]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:39.930660767 (13h34m39.520005754s) [upload=14085]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:39.930663112 (13h34m39.520008099s) [upload=14085]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:40.207143612 (13h34m39.796488599s) [upload=14085]: Done uploading segments... Aug 7 23:14:40.207161857 (13h34m39.796506844s) [upload=14085]: Upload finished. err: <nil> Aug 7 23:14:59.171620401 (13h34m58.760965388s) [upload=14091]: Starting upload Aug 7 23:14:59.171676218 (13h34m58.761021205s) [upload=14091]: Done starting upload Aug 7 23:14:59.171718468 (13h34m58.761063455s) [upload=14091]: Uploading segments... Aug 7 23:14:59.171763204 (13h34m58.761108191s) [upload=14091]: Got next segment. Inline: true Aug 7 23:14:59.171776990 (13h34m58.761121977s) [upload=14091]: Waiting for error group managing segments... Aug 7 23:14:59.171816787 (13h34m58.761161774s) [upload=14091]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:59.171820764 (13h34m58.761165741s) [upload=14091]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:59.171823429 (13h34m58.761168416s) [upload=14091]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:28.519847650 (13h35m28.109192637s) [upload=14091]: Done uploading segments... Aug 7 23:15:28.519870794 (13h35m28.109215781s) [upload=14091]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:34.319815009 (13h35m33.909159996s) [upload=14100]: Starting upload Aug 7 23:15:34.319867980 (13h35m33.909212967s) [upload=14100]: Done starting upload Aug 7 23:15:34.319916702 (13h35m33.909261679s) [upload=14100]: Uploading segments... Aug 7 23:15:34.319945739 (13h35m33.909290726s) [upload=14100]: Got next segment. Inline: true Aug 7 23:15:34.319949416 (13h35m33.909294403s) [upload=14100]: Waiting for error group managing segments... Aug 7 23:15:34.319988731 (13h35m33.909333718s) [upload=14100]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:34.319992698 (13h35m33.909337685s) [upload=14100]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:34.319995163 (13h35m33.90934015s) [upload=14100]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:03.672763058 (13h36m3.262108045s) [upload=14100]: Done uploading segments... Aug 7 23:16:03.672786503 (13h36m3.26213149s) [upload=14100]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:43.252280188 (13h36m42.841625175s) [upload=14114]: Starting upload Aug 7 23:16:43.252335193 (13h36m42.84168018s) [upload=14114]: Done starting upload Aug 7 23:16:43.252381041 (13h36m42.841726018s) [upload=14114]: Uploading segments... Aug 7 23:16:43.252412271 (13h36m42.841757258s) [upload=14114]: Got next segment. Inline: true Aug 7 23:16:43.252416208 (13h36m42.841761195s) [upload=14114]: Waiting for error group managing segments... Aug 7 23:16:43.252453641 (13h36m42.841798628s) [upload=14114]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:43.252457388 (13h36m42.841802375s) [upload=14114]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:43.252459622 (13h36m42.841804609s) [upload=14114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:12.607365819 (13h37m12.196710806s) [upload=14114]: Done uploading segments... Aug 7 23:17:12.607385377 (13h37m12.196730364s) [upload=14114]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:56.238720000 (13h37m55.828064987s) [upload=14132]: Starting upload Aug 7 23:17:56.238785927 (13h37m55.828130914s) [upload=14132]: Done starting upload Aug 7 23:17:56.238851903 (13h37m55.82819689s) [upload=14132]: Uploading segments... Aug 7 23:17:56.238885677 (13h37m55.828230664s) [upload=14132]: Got next segment. Inline: true Aug 7 23:17:56.238888933 (13h37m55.82823392s) [upload=14132]: Waiting for error group managing segments... Aug 7 23:17:56.238932207 (13h37m55.828277194s) [upload=14132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:56.238935984 (13h37m55.828280971s) [upload=14132]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:56.238938278 (13h37m55.828283265s) [upload=14132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:25.593341271 (13h38m25.182686258s) [upload=14132]: Done uploading segments... Aug 7 23:18:25.593355268 (13h38m25.182700245s) [upload=14132]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.899374425s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 659.305566ms target: https://us1.storj.io/login FAIL (2025-08-07 23:23:36 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 659.331895ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 4 out of 60 is failed (threshold=3) [............................................X...X..X.X......] duration: 1.860094594s FAIL (2025-08-07 23:23:35 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.860119931s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...........................................X..X.XXXX.X......] duration: 7.837105764s FAIL (2025-08-07 23:23:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 8]: Upload finished. err: context deadline exceeded Aug 7 23:18:32.336054087 (13h38m31.925399094s) [upload=14140]: Starting upload Aug 7 23:18:32.336180319 (13h38m31.925525306s) [upload=14140]: Done starting upload Aug 7 23:18:32.336261133 (13h38m31.92560616s) [upload=14140]: Uploading segments... Aug 7 23:18:32.336590001 (13h38m31.925935029s) [upload=14140]: Got next segment. Inline: false Aug 7 23:18:32.336650628 (13h38m31.925995655s) [upload=14140]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:32.336658754 (13h38m31.926003761s) [upload=14140]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:32.571732430 (13h38m32.161077417s) [seg_pos={0 0}, upload=14140]: Begin upload segment... Aug 7 23:18:32.571774531 (13h38m32.161119528s) [seg_pos={0 0}, upload=14140]: Done begin upload segment. Aug 7 23:18:32.571805400 (13h38m32.161150417s) [noise=true , address=77.90.31.12:4589 , piece_id=ZMKTDC4M5JHWVARXQDQAKRFSXM3OQQZ2IGQASJCK2OKYX3IQXHVA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.571844173 (13h38m32.161189161s) [noise=true , address=104.36.82.9:4007 , piece_id=VZLD7RU4SK5MAQE22D7OBVT6XOQAN75UNDMT7JNPCRZNRKFX36XQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572218380 (13h38m32.161563367s) [noise=true , address=104.36.82.10:4077 , piece_id=WYGEC4FYUPAR45MZIWLZQ2ASF5SUPECS4YI3UTRRHEVAKUEYCGYA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572289677 (13h38m32.161634654s) [noise=true , address=77.90.31.17:4770 , piece_id=ISA43NF4AKPK2SGYDPKT6ZIAD5WJ27A6GT6JZA2GHQUCAQBRRAFA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572494809 (13h38m32.161839796s) [noise=true , address=104.36.82.7:4911 , piece_id=FAMSQZJMALQIEAG5BVR5OV5WHEHN432Y4DZD3DNPC5CKP7X7GMQA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572588257 (13h38m32.161933244s) [noise=true , address=77.90.31.15:4697 , piece_id=6RYZJ22TCTJFRBMQV7M545VHB475CHDV2DBKCGO3MJP65VEMJ5FQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572772450 (13h38m32.162117427s) [noise=true , address=45.91.116.12:4225 , piece_id=22GAFTM5SITGTVJEELA3BJP7W2QZW7BVYJGJMKJOLE2RF2GZ5CGQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572865959 (13h38m32.162210946s) [noise=true , address=45.91.116.12:4227 , piece_id=T3JVU7VMSYBTYXBVGRD7LSNIG3PWF3QDVDNDQ24JQOSV7ZFIODNA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573029050 (13h38m32.162374037s) [noise=true , address=77.90.31.19:4842 , piece_id=3ZRHNMBPQUJH5SZZ75UJWLYIYZ5YI7CC5OMIJJFM7WVKZWJX6R3A, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573227882 (13h38m32.162572869s) [noise=true , address=104.36.82.4:4804 , piece_id=LTSBCPH74YSWH7IHN6MH4NWTBJ5FQ7534VERPCY5NIDV24KYPXMQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573293797 (13h38m32.162638774s) [noise=true , address=104.36.82.15:4259 , piece_id=GFCTNTXSY4WVSTPAD3AZIYCHE4O5EN24WN25VUNCRNWCSUWCYBMA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573504962 (13h38m32.162849949s) [noise=true , address=77.90.31.16:4734 , piece_id=OJPJZ5RF7QY4KNFBZ3MAVTA63HMC4JRG3HQLBN7SVROVKFQZY7BQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573527936 (13h38m32.162872923s) [noise=true , address=45.91.116.7:4009 , piece_id=VKYMGNJEGFLB5M3L3WLD37C5ZILQGLRJROSQ4ZEIBN46CQSY5JZQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.574078970 (13h38m32.163423947s) [noise=true , address=45.91.116.11:4119 , piece_id=UXO65SB4OZ57BCKOWGE2VDV7KCSKCGE7I7K4RTGRYJ7VU5GKFPHQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.574216453 (13h38m32.16356143s) [noise=true , address=104.36.82.12:4149 , piece_id=2K5BCSGU4UN43HF7VHG3MBNMQEGSIZPPHTDHDFI3XPY5DDG6NKOA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.576753347 (13h38m32.166098334s) [noise=true , address=104.36.82.8:4971 , piece_id=DNQVVV7QT2A2NFMIRURJ5IZRFPYZVX2NADRTX2P2YORA2CKMWY6Q, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:34.327895253 (13h38m33.91724024s) [upload=14140]: Next returned nil segment Aug 7 23:18:34.327912295 (13h38m33.917257282s) [upload=14140]: Waiting for error group managing segments... Aug 7 23:18:34.716828436 (13h38m34.306173413s) [noise=true , address=104.36.82.7:4911 , piece_id=FAMSQZJMALQIEAG5BVR5OV5WHEHN432Y4DZD3DNPC5CKP7X7GMQA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.729509878 (13h38m34.318854865s) [noise=true , address=104.36.82.12:4149 , piece_id=2K5BCSGU4UN43HF7VHG3MBNMQEGSIZPPHTDHDFI3XPY5DDG6NKOA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.738858565 (13h38m34.328203592s) [noise=true , address=104.36.82.4:4804 , piece_id=LTSBCPH74YSWH7IHN6MH4NWTBJ5FQ7534VERPCY5NIDV24KYPXMQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.783948702 (13h38m34.373293679s) [noise=true , address=77.90.31.19:4842 , piece_id=3ZRHNMBPQUJH5SZZ75UJWLYIYZ5YI7CC5OMIJJFM7WVKZWJX6R3A, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.824404112 (13h38m34.413749099s) [noise=true , address=104.36.82.15:4259 , piece_id=GFCTNTXSY4WVSTPAD3AZIYCHE4O5EN24WN25VUNCRNWCSUWCYBMA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.863114634 (13h38m34.452459611s) [noise=true , address=104.36.82.9:4007 , piece_id=VZLD7RU4SK5MAQE22D7OBVT6XOQAN75UNDMT7JNPCRZNRKFX36XQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.889984711 (13h38m34.479329698s) [noise=true , address=77.90.31.16:4734 , piece_id=OJPJZ5RF7QY4KNFBZ3MAVTA63HMC4JRG3HQLBN7SVROVKFQZY7BQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.982222673 (13h38m34.57156766s) [noise=true , address=104.36.82.10:4077 , piece_id=WYGEC4FYUPAR45MZIWLZQ2ASF5SUPECS4YI3UTRRHEVAKUEYCGYA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.999676977 (13h38m34.589021964s) [noise=true , address=77.90.31.17:4770 , piece_id=ISA43NF4AKPK2SGYDPKT6ZIAD5WJ27A6GT6JZA2GHQUCAQBRRAFA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.002083712 (13h38m34.591428699s) [noise=true , address=45.91.116.12:4225 , piece_id=22GAFTM5SITGTVJEELA3BJP7W2QZW7BVYJGJMKJOLE2RF2GZ5CGQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.055266482 (13h38m34.644611469s) [noise=true , address=45.91.116.12:4227 , piece_id=T3JVU7VMSYBTYXBVGRD7LSNIG3PWF3QDVDNDQ24JQOSV7ZFIODNA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.081593190 (13h38m34.670938177s) [noise=true , address=45.91.116.11:4119 , piece_id=UXO65SB4OZ57BCKOWGE2VDV7KCSKCGE7I7K4RTGRYJ7VU5GKFPHQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.123608726 (13h38m34.712953703s) [noise=true , address=77.90.31.12:4589 , piece_id=ZMKTDC4M5JHWVARXQDQAKRFSXM3OQQZ2IGQASJCK2OKYX3IQXHVA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.123631779 (13h38m34.712976766s) [seg_pos={0 0}, upload=14140]: Segment reached optimal threshold of 13 pieces. Aug 7 23:18:35.123785944 (13h38m34.713130931s) [noise=true , address=45.91.116.7:4009 , piece_id=VKYMGNJEGFLB5M3L3WLD37C5ZILQGLRJROSQ4ZEIBN46CQSY5JZQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:35.123945530 (13h38m34.713290517s) [noise=true , address=77.90.31.15:4697 , piece_id=6RYZJ22TCTJFRBMQV7M545VHB475CHDV2DBKCGO3MJP65VEMJ5FQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:35.123972431 (13h38m34.713317418s) [noise=true , address=104.36.82.8:4971 , piece_id=DNQVVV7QT2A2NFMIRURJ5IZRFPYZVX2NADRTX2P2YORA2CKMWY6Q, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: read tcp 188.166.225.170:44698->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:18:35.123986628 (13h38m34.713331615s) [seg_pos={0 0}, upload=14140]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:18:35.124083804 (13h38m34.713428791s) [upload=14140]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:18:35.124087611 (13h38m34.713432599s) [upload=14140]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:35.414558916 (13h38m35.003903903s) [upload=14140]: Done uploading segments... Aug 7 23:18:35.414577682 (13h38m35.00392267s) [upload=14140]: Upload finished. err: <nil> Aug 7 23:18:58.905554387 (13h38m58.494899394s) [upload=14146]: Starting upload Aug 7 23:18:58.905620613 (13h38m58.4949656s) [upload=14146]: Done starting upload Aug 7 23:18:58.905683373 (13h38m58.49502836s) [upload=14146]: Uploading segments... Aug 7 23:18:58.905783355 (13h38m58.495128342s) [upload=14146]: Got next segment. Inline: false Aug 7 23:18:58.905832459 (13h38m58.495177446s) [upload=14146]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:58.905836627 (13h38m58.495181614s) [upload=14146]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:19:28.246792926 (13h39m27.836137913s) [upload=14146]: Next segment err: context deadline exceeded Aug 7 23:19:28.246867379 (13h39m27.836212366s) [upload=14146]: Done uploading segments... Aug 7 23:19:28.246884292 (13h39m27.836229269s) [upload=14146]: Upload finished. err: context deadline exceeded Duration: 7.8371305s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................................XX.X......] duration: 7.702367619s FAIL (2025-08-07 23:23:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== PC5WABH5C4FB5BLC55F4FZQAU4IAHMEA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.821935980 (13h37m27.411280967s) [noise=true , address=104.36.82.8:4983 , piece_id=AVNQRSGZB5XHG3VFYFHRLBLKQ7B4YWXGIPWDK32DYJSMFXUFTSNA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.822230193 (13h37m27.41157518s) [noise=true , address=104.36.82.16:4310 , piece_id=OAD3ECB3B5RXQB362MCLIWSQC4PKS3QUHWVMIMK4XPGH63JKO3LQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.822283705 (13h37m27.411628692s) [noise=true , address=104.36.82.59:4323 , piece_id=WNEN4TRJFQ64H74WZOQGGP3V5F7Y3A7OIMORF7M7WEYOR252JSRA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.822684663 (13h37m27.41202965s) [noise=true , address=104.36.82.39:4747 , piece_id=7OQQP63SGNWWQEYR75RRJRHZSL52NRYH3BCS65XI4SWA24BKERUQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.823034542 (13h37m27.412379529s) [noise=true , address=104.36.82.59:4322 , piece_id=D4THBNGRCWHLJC6HETJRODVC7ZRMOWMIAHGM6GYVU7LFIJNBTOZQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.823366667 (13h37m27.412711654s) [noise=true , address=104.36.82.60:4343 , piece_id=UHOQFZPYXHO56TUWMTSQE5XVWFGO66ATAA3EU7TQ5KX2MIN2ZJQQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.823697740 (13h37m27.413042727s) [noise=true , address=79.127.201.215:4355 , piece_id=KUJDHE66HM6IAFRND7KRYDII2FB7AA3S5K5S244JPQZUD4BYXFWA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.824090602 (13h37m27.413435589s) [noise=true , address=104.36.82.10:4093 , piece_id=VQUPLYS4X67PIYNTDH76KJ3DTBIWIH6UM3TQJ6VEREZIUJIH7A2Q, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.824398111 (13h37m27.413743098s) [noise=true , address=64.49.13.170:4718 , piece_id=VXSII2V5QDMEJ5G5PJWNHXSJ6HF5NLLTGEDN5CHKKRWYHIAKHMVA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.824731088 (13h37m27.414076065s) [noise=true , address=104.36.82.43:4847 , piece_id=5Y56Z5NDMWSATFWNBYDWKBZB7SUSJ3ABAFMOY6PJ3B24SPPGJH3A, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.828362365 (13h37m27.417707362s) [noise=true , address=104.36.82.61:4404 , piece_id=KGEALBUHVOZEFP7V7LPYXQG5APDPXBEWG6UG34QGOCYOHPFHEZNQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.831994514 (13h37m27.421339501s) [noise=true , address=104.36.82.18:4370 , piece_id=HHYIBQ2JWTCPEOVRZQLKEWT7BFUZDXXZP6FYED3MPEWTVODBQUQA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.835602707 (13h37m27.424947694s) [noise=true , address=45.91.116.6:4988 , piece_id=QF7HXJOV5JYZ73FJMUUGT2JJUR3W2PY3C7CE6EQQCYGM72VDSTXQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.837475180 (13h37m27.426820177s) [noise=true , address=104.36.82.34:4547 , piece_id=2XYJPP25OROUVGLPDMGFVMDOD4AYOFE6BRQ4OFMVEO2D73XVPNAQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.841864818 (13h37m27.431209825s) [noise=true , address=104.36.82.62:4439 , piece_id=F4FXK4T3IKUXUGL37K3DEDNWWAUZZFEJVT7WBYDVVEZSXO6QG4OQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:29.010273953 (13h37m28.59961894s) [upload=14126]: Next returned nil segment Aug 7 23:17:29.010293008 (13h37m28.599637996s) [upload=14126]: Waiting for error group managing segments... Aug 7 23:17:29.309514867 (13h37m28.898859844s) [noise=true , address=64.49.13.170:4718 , piece_id=VXSII2V5QDMEJ5G5PJWNHXSJ6HF5NLLTGEDN5CHKKRWYHIAKHMVA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.779221569 (13h37m29.368566556s) [noise=true , address=79.127.201.215:4355 , piece_id=KUJDHE66HM6IAFRND7KRYDII2FB7AA3S5K5S244JPQZUD4BYXFWA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.827123595 (13h37m29.416468582s) [noise=true , address=104.36.82.34:4547 , piece_id=2XYJPP25OROUVGLPDMGFVMDOD4AYOFE6BRQ4OFMVEO2D73XVPNAQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.905373281 (13h37m29.494718258s) [noise=true , address=104.36.82.39:4747 , piece_id=7OQQP63SGNWWQEYR75RRJRHZSL52NRYH3BCS65XI4SWA24BKERUQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.927717560 (13h37m29.517062537s) [noise=true , address=104.36.82.43:4847 , piece_id=5Y56Z5NDMWSATFWNBYDWKBZB7SUSJ3ABAFMOY6PJ3B24SPPGJH3A, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.139039260 (13h37m29.728384247s) [noise=true , address=104.36.82.46:4511 , piece_id=4A43V3IOJTGJNWLKNYB7PC5WABH5C4FB5BLC55F4FZQAU4IAHMEA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.142609049 (13h37m29.731954036s) [noise=true , address=104.36.82.10:4093 , piece_id=VQUPLYS4X67PIYNTDH76KJ3DTBIWIH6UM3TQJ6VEREZIUJIH7A2Q, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.265344466 (13h37m29.854689463s) [noise=true , address=104.36.82.8:4983 , piece_id=AVNQRSGZB5XHG3VFYFHRLBLKQ7B4YWXGIPWDK32DYJSMFXUFTSNA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.341283132 (13h37m29.930628119s) [noise=true , address=45.91.116.6:4988 , piece_id=QF7HXJOV5JYZ73FJMUUGT2JJUR3W2PY3C7CE6EQQCYGM72VDSTXQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.343588482 (13h37m29.932933459s) [noise=true , address=104.36.82.61:4404 , piece_id=KGEALBUHVOZEFP7V7LPYXQG5APDPXBEWG6UG34QGOCYOHPFHEZNQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.538382992 (13h37m30.127727969s) [noise=true , address=104.36.82.37:4666 , piece_id=WXBTFS2SS7DMVYBL5BHW7TBXH7FJODGB5JJDT6WP4LOO3LU4GNDA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.631974070 (13h37m30.221319047s) [noise=true , address=104.36.82.9:4028 , piece_id=HSPPTMR3OOL5DWG25IF3LXYTSOUASUM3BFX7RKHZHCUTBP4B3GQA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:31.962954475 (13h37m31.552299462s) [noise=true , address=104.36.82.59:4322 , piece_id=D4THBNGRCWHLJC6HETJRODVC7ZRMOWMIAHGM6GYVU7LFIJNBTOZQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:32.006950608 (13h37m31.596295635s) [noise=true , address=104.36.82.59:4323 , piece_id=WNEN4TRJFQ64H74WZOQGGP3V5F7Y3A7OIMORF7M7WEYOR252JSRA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:32.006984062 (13h37m31.596329049s) [seg_pos={0 0}, upload=14126]: Segment reached optimal threshold of 14 pieces. Aug 7 23:17:32.007159788 (13h37m31.596504775s) [noise=true , address=104.36.82.18:4370 , piece_id=HHYIBQ2JWTCPEOVRZQLKEWT7BFUZDXXZP6FYED3MPEWTVODBQUQA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SXZN7WJvcHys9BRd4Lq8NAZ3MSgHf9ZVv7SgaiJNRosdmFVpm): protocol: expected piece hash; context canceled; EOF Aug 7 23:17:32.007279266 (13h37m31.596624253s) [noise=true , address=104.36.82.60:4343 , piece_id=UHOQFZPYXHO56TUWMTSQE5XVWFGO66ATAA3EU7TQ5KX2MIN2ZJQQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16EsEMRW53sjKVag2WpSDGLhELyE2bSXyW4aAurCvDFKRNkXuf): protocol: write tcp 188.166.225.170:53332->104.36.82.60:4343: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:17:32.007291060 (13h37m31.596636047s) [noise=true , address=104.36.82.16:4310 , piece_id=OAD3ECB3B5RXQB362MCLIWSQC4PKS3QUHWVMIMK4XPGH63JKO3LQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R): protocol: write tcp 188.166.225.170:46238->104.36.82.16:4310: use of closed network connection; write tcp 188.166.225.170:46238->104.36.82.16:4310: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46238->104.36.82.16:4310: use of closed network connection Aug 7 23:17:32.007409626 (13h37m31.596754613s) [noise=true , address=104.36.82.62:4439 , piece_id=F4FXK4T3IKUXUGL37K3DEDNWWAUZZFEJVT7WBYDVVEZSXO6QG4OQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HRW3L15ka62qA7mWmEJaJ2ydTA5f8sVY1iL3Fn1R7ubisPF3W): protocol: read tcp 188.166.225.170:33134->104.36.82.62:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:17:32.007441958 (13h37m31.596786945s) [seg_pos={0 0}, upload=14126]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:17:32.007533564 (13h37m31.596878551s) [upload=14126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:17:32.007541158 (13h37m31.596886145s) [upload=14126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:39.583862369 (13h37m39.173207346s) [upload=14126]: Done uploading segments... Aug 7 23:17:39.583880514 (13h37m39.173225501s) [upload=14126]: Upload finished. err: <nil> Aug 7 23:17:57.117137796 (13h37m56.706482783s) [upload=14133]: Starting upload Aug 7 23:17:57.117195166 (13h37m56.706540153s) [upload=14133]: Done starting upload Aug 7 23:17:57.117244931 (13h37m56.706589918s) [upload=14133]: Uploading segments... Aug 7 23:17:57.117378968 (13h37m56.706723955s) [upload=14133]: Got next segment. Inline: false Aug 7 23:17:57.117404277 (13h37m56.706749264s) [upload=14133]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:57.117408454 (13h37m56.706753441s) [upload=14133]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:56.470635719 (13h38m56.059980716s) [upload=14133]: Next segment err: context deadline exceeded Aug 7 23:18:56.470717675 (13h38m56.060062703s) [upload=14133]: Done uploading segments... Aug 7 23:18:56.470739287 (13h38m56.060084304s) [upload=14133]: Upload finished. err: context deadline exceeded Duration: 7.702395664s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [................................................XXXXXX.X.X..] duration: 18.62191676s FAIL (2025-08-07 23:23:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.621939233s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..............................................XXXX.X..XX....] duration: 6.709591526s FAIL (2025-08-07 23:23:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.709617435s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X......X.........] duration: 2.884116704s FAIL (2025-08-07 23:23:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== 30.892541890 (13h31m30.481886877s) [upload=14032]: Upload finished. err: <nil> Aug 7 23:11:48.390915893 (13h31m47.98026088s) [upload=14038]: Starting upload Aug 7 23:11:48.390965198 (13h31m47.980310185s) [upload=14038]: Done starting upload Aug 7 23:11:48.391014242 (13h31m47.980359229s) [upload=14038]: Uploading segments... Aug 7 23:11:48.391054007 (13h31m47.980399004s) [upload=14038]: Got next segment. Inline: true Aug 7 23:11:48.391057494 (13h31m47.980402471s) [upload=14038]: Waiting for error group managing segments... Aug 7 23:11:48.391089254 (13h31m47.980434231s) [upload=14038]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:48.391093213 (13h31m47.98043819s) [upload=14038]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:48.391095457 (13h31m47.980440444s) [upload=14038]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:48.701950724 (13h31m48.291295711s) [upload=14038]: Done uploading segments... Aug 7 23:11:48.701969449 (13h31m48.291314436s) [upload=14038]: Upload finished. err: <nil> Aug 7 23:12:06.430245154 (13h32m6.019590141s) [upload=14048]: Starting upload Aug 7 23:12:06.430293076 (13h32m6.019638063s) [upload=14048]: Done starting upload Aug 7 23:12:06.430361757 (13h32m6.019706745s) [upload=14048]: Uploading segments... Aug 7 23:12:06.430459223 (13h32m6.01980421s) [upload=14048]: Got next segment. Inline: true Aug 7 23:12:06.430464543 (13h32m6.01980953s) [upload=14048]: Waiting for error group managing segments... Aug 7 23:12:06.430527555 (13h32m6.019872542s) [upload=14048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:06.430533796 (13h32m6.019878783s) [upload=14048]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:06.430536692 (13h32m6.019881679s) [upload=14048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:06.729952062 (13h32m6.319297049s) [upload=14048]: Done uploading segments... Aug 7 23:12:06.729968844 (13h32m6.319313832s) [upload=14048]: Upload finished. err: <nil> Aug 7 23:12:37.551701338 (13h32m37.141046325s) [upload=14055]: Starting upload Aug 7 23:12:37.551746985 (13h32m37.141091972s) [upload=14055]: Done starting upload Aug 7 23:12:37.551807301 (13h32m37.141152288s) [upload=14055]: Uploading segments... Aug 7 23:12:37.551914826 (13h32m37.141259833s) [upload=14055]: Got next segment. Inline: true Aug 7 23:12:37.551921779 (13h32m37.141266766s) [upload=14055]: Waiting for error group managing segments... Aug 7 23:12:37.551977145 (13h32m37.141322122s) [upload=14055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.551984059 (13h32m37.141329086s) [upload=14055]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.551988918 (13h32m37.141333936s) [upload=14055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.767068918 (13h32m37.356413905s) [upload=14055]: Done uploading segments... Aug 7 23:12:37.767104365 (13h32m37.356449342s) [upload=14055]: Upload finished. err: metaclient: Too Many Requests Aug 7 23:13:47.522489627 (13h33m47.111834614s) [upload=14072]: Starting upload Aug 7 23:13:47.522559681 (13h33m47.111904668s) [upload=14072]: Done starting upload Aug 7 23:13:47.522629674 (13h33m47.111974661s) [upload=14072]: Uploading segments... Aug 7 23:13:47.522726771 (13h33m47.112071778s) [upload=14072]: Got next segment. Inline: true Aug 7 23:13:47.522735617 (13h33m47.112080614s) [upload=14072]: Waiting for error group managing segments... Aug 7 23:13:47.522790582 (13h33m47.112135569s) [upload=14072]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:47.522796493 (13h33m47.112141481s) [upload=14072]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:47.522799320 (13h33m47.112144307s) [upload=14072]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:47.796144342 (13h33m47.385489329s) [upload=14072]: Done uploading segments... Aug 7 23:13:47.796162656 (13h33m47.385507643s) [upload=14072]: Upload finished. err: <nil> Aug 7 23:14:22.239921861 (13h34m21.829266848s) [upload=14081]: Starting upload Aug 7 23:14:22.239973791 (13h34m21.829318778s) [upload=14081]: Done starting upload Aug 7 23:14:22.240087399 (13h34m21.829432396s) [upload=14081]: Uploading segments... Aug 7 23:14:22.240187159 (13h34m21.829532156s) [upload=14081]: Got next segment. Inline: true Aug 7 23:14:22.240194182 (13h34m21.829539179s) [upload=14081]: Waiting for error group managing segments... Aug 7 23:14:22.240255971 (13h34m21.829600958s) [upload=14081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:22.240261311 (13h34m21.829606288s) [upload=14081]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:22.240263816 (13h34m21.829608803s) [upload=14081]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:22.537482123 (13h34m22.12682711s) [upload=14081]: Done uploading segments... Aug 7 23:14:22.537500677 (13h34m22.126845664s) [upload=14081]: Upload finished. err: <nil> Aug 7 23:14:56.968922831 (13h34m56.558267818s) [upload=14090]: Starting upload Aug 7 23:14:56.968974069 (13h34m56.558319056s) [upload=14090]: Done starting upload Aug 7 23:14:56.969028383 (13h34m56.55837337s) [upload=14090]: Uploading segments... Aug 7 23:14:56.969059602 (13h34m56.55840459s) [upload=14090]: Got next segment. Inline: true Aug 7 23:14:56.969063500 (13h34m56.558408487s) [upload=14090]: Waiting for error group managing segments... Aug 7 23:14:56.969102906 (13h34m56.558447893s) [upload=14090]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:56.969107034 (13h34m56.558452021s) [upload=14090]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:56.969109899 (13h34m56.558454886s) [upload=14090]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:57.268863065 (13h34m56.858208052s) [upload=14090]: Done uploading segments... Aug 7 23:14:57.268882201 (13h34m56.858227188s) [upload=14090]: Upload finished. err: <nil> Aug 7 23:16:09.138990771 (13h36m8.728335758s) [upload=14107]: Starting upload Aug 7 23:16:09.139048231 (13h36m8.728393208s) [upload=14107]: Done starting upload Aug 7 23:16:09.139094229 (13h36m8.728439226s) [upload=14107]: Uploading segments... Aug 7 23:16:09.139131069 (13h36m8.728476056s) [upload=14107]: Got next segment. Inline: true Aug 7 23:16:09.139135007 (13h36m8.728479994s) [upload=14107]: Waiting for error group managing segments... Aug 7 23:16:09.139167859 (13h36m8.728512846s) [upload=14107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:09.139171937 (13h36m8.728516924s) [upload=14107]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:09.139199730 (13h36m8.728544707s) [upload=14107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:09.409287150 (13h36m8.998632137s) [upload=14107]: Done uploading segments... Aug 7 23:16:09.409303952 (13h36m8.998648939s) [upload=14107]: Upload finished. err: <nil> Aug 7 23:16:30.665596455 (13h36m30.254941442s) [upload=14111]: Starting upload Aug 7 23:16:30.665642382 (13h36m30.254987369s) [upload=14111]: Done starting upload Aug 7 23:16:30.665687749 (13h36m30.255032726s) [upload=14111]: Uploading segments... Aug 7 23:16:30.665725801 (13h36m30.255070788s) [upload=14111]: Got next segment. Inline: true Aug 7 23:16:30.665729409 (13h36m30.255074386s) [upload=14111]: Waiting for error group managing segments... Aug 7 23:16:30.665763554 (13h36m30.255108541s) [upload=14111]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:30.665768984 (13h36m30.255113971s) [upload=14111]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:30.665771289 (13h36m30.255116276s) [upload=14111]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:30.945395745 (13h36m30.534740732s) [upload=14111]: Done uploading segments... Aug 7 23:16:30.945418538 (13h36m30.534763526s) [upload=14111]: Upload finished. err: <nil> Aug 7 23:17:25.677744155 (13h37m25.267089143s) [upload=14125]: Starting upload Aug 7 23:17:25.677793930 (13h37m25.267138917s) [upload=14125]: Done starting upload Aug 7 23:17:25.677862562 (13h37m25.267207539s) [upload=14125]: Uploading segments... Aug 7 23:17:25.677896967 (13h37m25.267241954s) [upload=14125]: Got next segment. Inline: true Aug 7 23:17:25.677900704 (13h37m25.267245681s) [upload=14125]: Waiting for error group managing segments... Aug 7 23:17:25.677933046 (13h37m25.267278033s) [upload=14125]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:25.677936884 (13h37m25.267281871s) [upload=14125]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:25.677939309 (13h37m25.267284296s) [upload=14125]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:26.502642065 (13h37m26.091987052s) [upload=14125]: Done uploading segments... Aug 7 23:17:26.502660942 (13h37m26.092005919s) [upload=14125]: Upload finished. err: <nil> Aug 7 23:17:53.025114325 (13h37m52.614459313s) [upload=14131]: Starting upload Aug 7 23:17:53.025169641 (13h37m52.614514618s) [upload=14131]: Done starting upload Aug 7 23:17:53.025219857 (13h37m52.614564844s) [upload=14131]: Uploading segments... Aug 7 23:17:53.025267939 (13h37m52.614612936s) [upload=14131]: Got next segment. Inline: true Aug 7 23:17:53.025271916 (13h37m52.614616903s) [upload=14131]: Waiting for error group managing segments... Aug 7 23:17:53.025302495 (13h37m52.614647482s) [upload=14131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:53.025306382 (13h37m52.614651369s) [upload=14131]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:53.025308497 (13h37m52.614653484s) [upload=14131]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:53.302835519 (13h37m52.892180506s) [upload=14131]: Done uploading segments... Aug 7 23:17:53.302854656 (13h37m52.892199633s) [upload=14131]: Upload finished. err: metaclient: Too Many Requests Duration: 2.884141552s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [............................................X......X........] duration: 1.978403966s FAIL (2025-08-07 23:23:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== 30.892541890 (13h31m30.481886877s) [upload=14032]: Upload finished. err: <nil> Aug 7 23:11:48.390915893 (13h31m47.98026088s) [upload=14038]: Starting upload Aug 7 23:11:48.390965198 (13h31m47.980310185s) [upload=14038]: Done starting upload Aug 7 23:11:48.391014242 (13h31m47.980359229s) [upload=14038]: Uploading segments... Aug 7 23:11:48.391054007 (13h31m47.980399004s) [upload=14038]: Got next segment. Inline: true Aug 7 23:11:48.391057494 (13h31m47.980402471s) [upload=14038]: Waiting for error group managing segments... Aug 7 23:11:48.391089254 (13h31m47.980434231s) [upload=14038]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:48.391093213 (13h31m47.98043819s) [upload=14038]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:48.391095457 (13h31m47.980440444s) [upload=14038]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:48.701950724 (13h31m48.291295711s) [upload=14038]: Done uploading segments... Aug 7 23:11:48.701969449 (13h31m48.291314436s) [upload=14038]: Upload finished. err: <nil> Aug 7 23:12:06.430245154 (13h32m6.019590141s) [upload=14048]: Starting upload Aug 7 23:12:06.430293076 (13h32m6.019638063s) [upload=14048]: Done starting upload Aug 7 23:12:06.430361757 (13h32m6.019706745s) [upload=14048]: Uploading segments... Aug 7 23:12:06.430459223 (13h32m6.01980421s) [upload=14048]: Got next segment. Inline: true Aug 7 23:12:06.430464543 (13h32m6.01980953s) [upload=14048]: Waiting for error group managing segments... Aug 7 23:12:06.430527555 (13h32m6.019872542s) [upload=14048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:06.430533796 (13h32m6.019878783s) [upload=14048]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:06.430536692 (13h32m6.019881679s) [upload=14048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:06.729952062 (13h32m6.319297049s) [upload=14048]: Done uploading segments... Aug 7 23:12:06.729968844 (13h32m6.319313832s) [upload=14048]: Upload finished. err: <nil> Aug 7 23:12:37.551701338 (13h32m37.141046325s) [upload=14055]: Starting upload Aug 7 23:12:37.551746985 (13h32m37.141091972s) [upload=14055]: Done starting upload Aug 7 23:12:37.551807301 (13h32m37.141152288s) [upload=14055]: Uploading segments... Aug 7 23:12:37.551914826 (13h32m37.141259833s) [upload=14055]: Got next segment. Inline: true Aug 7 23:12:37.551921779 (13h32m37.141266766s) [upload=14055]: Waiting for error group managing segments... Aug 7 23:12:37.551977145 (13h32m37.141322122s) [upload=14055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.551984059 (13h32m37.141329086s) [upload=14055]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.551988918 (13h32m37.141333936s) [upload=14055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.767068918 (13h32m37.356413905s) [upload=14055]: Done uploading segments... Aug 7 23:12:37.767104365 (13h32m37.356449342s) [upload=14055]: Upload finished. err: metaclient: Too Many Requests Aug 7 23:13:47.522489627 (13h33m47.111834614s) [upload=14072]: Starting upload Aug 7 23:13:47.522559681 (13h33m47.111904668s) [upload=14072]: Done starting upload Aug 7 23:13:47.522629674 (13h33m47.111974661s) [upload=14072]: Uploading segments... Aug 7 23:13:47.522726771 (13h33m47.112071778s) [upload=14072]: Got next segment. Inline: true Aug 7 23:13:47.522735617 (13h33m47.112080614s) [upload=14072]: Waiting for error group managing segments... Aug 7 23:13:47.522790582 (13h33m47.112135569s) [upload=14072]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:47.522796493 (13h33m47.112141481s) [upload=14072]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:47.522799320 (13h33m47.112144307s) [upload=14072]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:47.796144342 (13h33m47.385489329s) [upload=14072]: Done uploading segments... Aug 7 23:13:47.796162656 (13h33m47.385507643s) [upload=14072]: Upload finished. err: <nil> Aug 7 23:14:22.239921861 (13h34m21.829266848s) [upload=14081]: Starting upload Aug 7 23:14:22.239973791 (13h34m21.829318778s) [upload=14081]: Done starting upload Aug 7 23:14:22.240087399 (13h34m21.829432396s) [upload=14081]: Uploading segments... Aug 7 23:14:22.240187159 (13h34m21.829532156s) [upload=14081]: Got next segment. Inline: true Aug 7 23:14:22.240194182 (13h34m21.829539179s) [upload=14081]: Waiting for error group managing segments... Aug 7 23:14:22.240255971 (13h34m21.829600958s) [upload=14081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:22.240261311 (13h34m21.829606288s) [upload=14081]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:22.240263816 (13h34m21.829608803s) [upload=14081]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:22.537482123 (13h34m22.12682711s) [upload=14081]: Done uploading segments... Aug 7 23:14:22.537500677 (13h34m22.126845664s) [upload=14081]: Upload finished. err: <nil> Aug 7 23:14:56.968922831 (13h34m56.558267818s) [upload=14090]: Starting upload Aug 7 23:14:56.968974069 (13h34m56.558319056s) [upload=14090]: Done starting upload Aug 7 23:14:56.969028383 (13h34m56.55837337s) [upload=14090]: Uploading segments... Aug 7 23:14:56.969059602 (13h34m56.55840459s) [upload=14090]: Got next segment. Inline: true Aug 7 23:14:56.969063500 (13h34m56.558408487s) [upload=14090]: Waiting for error group managing segments... Aug 7 23:14:56.969102906 (13h34m56.558447893s) [upload=14090]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:56.969107034 (13h34m56.558452021s) [upload=14090]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:56.969109899 (13h34m56.558454886s) [upload=14090]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:57.268863065 (13h34m56.858208052s) [upload=14090]: Done uploading segments... Aug 7 23:14:57.268882201 (13h34m56.858227188s) [upload=14090]: Upload finished. err: <nil> Aug 7 23:16:09.138990771 (13h36m8.728335758s) [upload=14107]: Starting upload Aug 7 23:16:09.139048231 (13h36m8.728393208s) [upload=14107]: Done starting upload Aug 7 23:16:09.139094229 (13h36m8.728439226s) [upload=14107]: Uploading segments... Aug 7 23:16:09.139131069 (13h36m8.728476056s) [upload=14107]: Got next segment. Inline: true Aug 7 23:16:09.139135007 (13h36m8.728479994s) [upload=14107]: Waiting for error group managing segments... Aug 7 23:16:09.139167859 (13h36m8.728512846s) [upload=14107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:09.139171937 (13h36m8.728516924s) [upload=14107]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:09.139199730 (13h36m8.728544707s) [upload=14107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:09.409287150 (13h36m8.998632137s) [upload=14107]: Done uploading segments... Aug 7 23:16:09.409303952 (13h36m8.998648939s) [upload=14107]: Upload finished. err: <nil> Aug 7 23:16:30.665596455 (13h36m30.254941442s) [upload=14111]: Starting upload Aug 7 23:16:30.665642382 (13h36m30.254987369s) [upload=14111]: Done starting upload Aug 7 23:16:30.665687749 (13h36m30.255032726s) [upload=14111]: Uploading segments... Aug 7 23:16:30.665725801 (13h36m30.255070788s) [upload=14111]: Got next segment. Inline: true Aug 7 23:16:30.665729409 (13h36m30.255074386s) [upload=14111]: Waiting for error group managing segments... Aug 7 23:16:30.665763554 (13h36m30.255108541s) [upload=14111]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:30.665768984 (13h36m30.255113971s) [upload=14111]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:30.665771289 (13h36m30.255116276s) [upload=14111]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:30.945395745 (13h36m30.534740732s) [upload=14111]: Done uploading segments... Aug 7 23:16:30.945418538 (13h36m30.534763526s) [upload=14111]: Upload finished. err: <nil> Aug 7 23:17:25.677744155 (13h37m25.267089143s) [upload=14125]: Starting upload Aug 7 23:17:25.677793930 (13h37m25.267138917s) [upload=14125]: Done starting upload Aug 7 23:17:25.677862562 (13h37m25.267207539s) [upload=14125]: Uploading segments... Aug 7 23:17:25.677896967 (13h37m25.267241954s) [upload=14125]: Got next segment. Inline: true Aug 7 23:17:25.677900704 (13h37m25.267245681s) [upload=14125]: Waiting for error group managing segments... Aug 7 23:17:25.677933046 (13h37m25.267278033s) [upload=14125]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:25.677936884 (13h37m25.267281871s) [upload=14125]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:25.677939309 (13h37m25.267284296s) [upload=14125]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:26.502642065 (13h37m26.091987052s) [upload=14125]: Done uploading segments... Aug 7 23:17:26.502660942 (13h37m26.092005919s) [upload=14125]: Upload finished. err: <nil> Aug 7 23:17:53.025114325 (13h37m52.614459313s) [upload=14131]: Starting upload Aug 7 23:17:53.025169641 (13h37m52.614514618s) [upload=14131]: Done starting upload Aug 7 23:17:53.025219857 (13h37m52.614564844s) [upload=14131]: Uploading segments... Aug 7 23:17:53.025267939 (13h37m52.614612936s) [upload=14131]: Got next segment. Inline: true Aug 7 23:17:53.025271916 (13h37m52.614616903s) [upload=14131]: Waiting for error group managing segments... Aug 7 23:17:53.025302495 (13h37m52.614647482s) [upload=14131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:53.025306382 (13h37m52.614651369s) [upload=14131]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:53.025308497 (13h37m52.614653484s) [upload=14131]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:53.302835519 (13h37m52.892180506s) [upload=14131]: Done uploading segments... Aug 7 23:17:53.302854656 (13h37m52.892199633s) [upload=14131]: Upload finished. err: metaclient: Too Many Requests Duration: 1.978424615s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 4 out of 60 is failed (threshold=3) [.................................................XXXX.......] duration: 3.133546635s FAIL (2025-08-07 23:22:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .175884539 (13h29m53.765229526s) [upload=14007]: Starting upload Aug 7 23:09:54.175937640 (13h29m53.765282618s) [upload=14007]: Done starting upload Aug 7 23:09:54.175979842 (13h29m53.765324829s) [upload=14007]: Uploading segments... Aug 7 23:09:54.176041389 (13h29m53.765386376s) [upload=14007]: Got next segment. Inline: true Aug 7 23:09:54.176045487 (13h29m53.765390474s) [upload=14007]: Waiting for error group managing segments... Aug 7 23:09:54.176077538 (13h29m53.765422525s) [upload=14007]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:54.176082318 (13h29m53.765427305s) [upload=14007]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:54.176084522 (13h29m53.765429509s) [upload=14007]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:54.453731274 (13h29m54.043076261s) [upload=14007]: Done uploading segments... Aug 7 23:09:54.453750932 (13h29m54.043095909s) [upload=14007]: Upload finished. err: <nil> Aug 7 23:10:39.330325211 (13h30m38.919670198s) [upload=14021]: Starting upload Aug 7 23:10:39.330417358 (13h30m38.919762345s) [upload=14021]: Done starting upload Aug 7 23:10:39.330504384 (13h30m38.919849371s) [upload=14021]: Uploading segments... Aug 7 23:10:39.330543368 (13h30m38.919888355s) [upload=14021]: Got next segment. Inline: true Aug 7 23:10:39.330546675 (13h30m38.919891662s) [upload=14021]: Waiting for error group managing segments... Aug 7 23:10:39.330581371 (13h30m38.919926349s) [upload=14021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:39.330585309 (13h30m38.919930286s) [upload=14021]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:39.330587663 (13h30m38.91993265s) [upload=14021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:39.610133078 (13h30m39.199478065s) [upload=14021]: Done uploading segments... Aug 7 23:10:39.610150061 (13h30m39.199495048s) [upload=14021]: Upload finished. err: <nil> Aug 7 23:11:32.695175239 (13h31m32.284520226s) [upload=14034]: Starting upload Aug 7 23:11:32.695285580 (13h31m32.284630567s) [upload=14034]: Done starting upload Aug 7 23:11:32.695336297 (13h31m32.284681284s) [upload=14034]: Uploading segments... Aug 7 23:11:32.695365753 (13h31m32.28471074s) [upload=14034]: Got next segment. Inline: true Aug 7 23:11:32.695369610 (13h31m32.284714598s) [upload=14034]: Waiting for error group managing segments... Aug 7 23:11:32.695401581 (13h31m32.284746569s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:32.695405168 (13h31m32.284750155s) [upload=14034]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:32.695407543 (13h31m32.28475253s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:32.954750627 (13h31m32.544095614s) [upload=14034]: Done uploading segments... Aug 7 23:11:32.954767650 (13h31m32.544112627s) [upload=14034]: Upload finished. err: <nil> Aug 7 23:12:37.502051138 (13h32m37.091396125s) [upload=14054]: Starting upload Aug 7 23:12:37.502104370 (13h32m37.091449357s) [upload=14054]: Done starting upload Aug 7 23:12:37.502150869 (13h32m37.091495856s) [upload=14054]: Uploading segments... Aug 7 23:12:37.502182519 (13h32m37.091527506s) [upload=14054]: Got next segment. Inline: true Aug 7 23:12:37.502185986 (13h32m37.091530963s) [upload=14054]: Waiting for error group managing segments... Aug 7 23:12:37.502216213 (13h32m37.0915612s) [upload=14054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.502219780 (13h32m37.091564767s) [upload=14054]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.502222425 (13h32m37.091567412s) [upload=14054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.772497714 (13h32m37.361842701s) [upload=14054]: Done uploading segments... Aug 7 23:12:37.772511151 (13h32m37.361856138s) [upload=14054]: Upload finished. err: <nil> Aug 7 23:13:44.834965069 (13h33m44.424310046s) [upload=14070]: Starting upload Aug 7 23:13:44.835030293 (13h33m44.42437528s) [upload=14070]: Done starting upload Aug 7 23:13:44.835093484 (13h33m44.424438471s) [upload=14070]: Uploading segments... Aug 7 23:13:44.835133321 (13h33m44.424478308s) [upload=14070]: Got next segment. Inline: true Aug 7 23:13:44.835136998 (13h33m44.424481985s) [upload=14070]: Waiting for error group managing segments... Aug 7 23:13:44.835214376 (13h33m44.424559363s) [upload=14070]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:44.835220728 (13h33m44.424565715s) [upload=14070]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:44.835222972 (13h33m44.424567959s) [upload=14070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:45.220889346 (13h33m44.810234343s) [upload=14070]: Done uploading segments... Aug 7 23:13:45.220937208 (13h33m44.810282205s) [upload=14070]: Upload finished. err: <nil> Aug 7 23:14:39.930436267 (13h34m39.519781254s) [upload=14085]: Starting upload Aug 7 23:14:39.930523634 (13h34m39.519868621s) [upload=14085]: Done starting upload Aug 7 23:14:39.930574131 (13h34m39.519919118s) [upload=14085]: Uploading segments... Aug 7 23:14:39.930614779 (13h34m39.519959766s) [upload=14085]: Got next segment. Inline: true Aug 7 23:14:39.930618847 (13h34m39.519963824s) [upload=14085]: Waiting for error group managing segments... Aug 7 23:14:39.930656519 (13h34m39.520001506s) [upload=14085]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:39.930660767 (13h34m39.520005754s) [upload=14085]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:39.930663112 (13h34m39.520008099s) [upload=14085]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:40.207143612 (13h34m39.796488599s) [upload=14085]: Done uploading segments... Aug 7 23:14:40.207161857 (13h34m39.796506844s) [upload=14085]: Upload finished. err: <nil> Aug 7 23:14:59.171620401 (13h34m58.760965388s) [upload=14091]: Starting upload Aug 7 23:14:59.171676218 (13h34m58.761021205s) [upload=14091]: Done starting upload Aug 7 23:14:59.171718468 (13h34m58.761063455s) [upload=14091]: Uploading segments... Aug 7 23:14:59.171763204 (13h34m58.761108191s) [upload=14091]: Got next segment. Inline: true Aug 7 23:14:59.171776990 (13h34m58.761121977s) [upload=14091]: Waiting for error group managing segments... Aug 7 23:14:59.171816787 (13h34m58.761161774s) [upload=14091]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:59.171820764 (13h34m58.761165741s) [upload=14091]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:59.171823429 (13h34m58.761168416s) [upload=14091]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:28.519847650 (13h35m28.109192637s) [upload=14091]: Done uploading segments... Aug 7 23:15:28.519870794 (13h35m28.109215781s) [upload=14091]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:34.319815009 (13h35m33.909159996s) [upload=14100]: Starting upload Aug 7 23:15:34.319867980 (13h35m33.909212967s) [upload=14100]: Done starting upload Aug 7 23:15:34.319916702 (13h35m33.909261679s) [upload=14100]: Uploading segments... Aug 7 23:15:34.319945739 (13h35m33.909290726s) [upload=14100]: Got next segment. Inline: true Aug 7 23:15:34.319949416 (13h35m33.909294403s) [upload=14100]: Waiting for error group managing segments... Aug 7 23:15:34.319988731 (13h35m33.909333718s) [upload=14100]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:34.319992698 (13h35m33.909337685s) [upload=14100]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:34.319995163 (13h35m33.90934015s) [upload=14100]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:03.672763058 (13h36m3.262108045s) [upload=14100]: Done uploading segments... Aug 7 23:16:03.672786503 (13h36m3.26213149s) [upload=14100]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:43.252280188 (13h36m42.841625175s) [upload=14114]: Starting upload Aug 7 23:16:43.252335193 (13h36m42.84168018s) [upload=14114]: Done starting upload Aug 7 23:16:43.252381041 (13h36m42.841726018s) [upload=14114]: Uploading segments... Aug 7 23:16:43.252412271 (13h36m42.841757258s) [upload=14114]: Got next segment. Inline: true Aug 7 23:16:43.252416208 (13h36m42.841761195s) [upload=14114]: Waiting for error group managing segments... Aug 7 23:16:43.252453641 (13h36m42.841798628s) [upload=14114]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:43.252457388 (13h36m42.841802375s) [upload=14114]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:43.252459622 (13h36m42.841804609s) [upload=14114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:12.607365819 (13h37m12.196710806s) [upload=14114]: Done uploading segments... Aug 7 23:17:12.607385377 (13h37m12.196730364s) [upload=14114]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:56.238720000 (13h37m55.828064987s) [upload=14132]: Starting upload Aug 7 23:17:56.238785927 (13h37m55.828130914s) [upload=14132]: Done starting upload Aug 7 23:17:56.238851903 (13h37m55.82819689s) [upload=14132]: Uploading segments... Aug 7 23:17:56.238885677 (13h37m55.828230664s) [upload=14132]: Got next segment. Inline: true Aug 7 23:17:56.238888933 (13h37m55.82823392s) [upload=14132]: Waiting for error group managing segments... Aug 7 23:17:56.238932207 (13h37m55.828277194s) [upload=14132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:56.238935984 (13h37m55.828280971s) [upload=14132]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:56.238938278 (13h37m55.828283265s) [upload=14132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:25.593341271 (13h38m25.182686258s) [upload=14132]: Done uploading segments... Aug 7 23:18:25.593355268 (13h38m25.182700245s) [upload=14132]: Upload finished. err: metaclient: context deadline exceeded Duration: 3.133583044s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.752041684s FAIL (2025-08-07 23:22:59 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.752068826s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [............................................X..X.XXXX.X.....] duration: 17.97584013s FAIL (2025-08-07 23:22:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 8]: Upload finished. err: context deadline exceeded Aug 7 23:18:32.336054087 (13h38m31.925399094s) [upload=14140]: Starting upload Aug 7 23:18:32.336180319 (13h38m31.925525306s) [upload=14140]: Done starting upload Aug 7 23:18:32.336261133 (13h38m31.92560616s) [upload=14140]: Uploading segments... Aug 7 23:18:32.336590001 (13h38m31.925935029s) [upload=14140]: Got next segment. Inline: false Aug 7 23:18:32.336650628 (13h38m31.925995655s) [upload=14140]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:32.336658754 (13h38m31.926003761s) [upload=14140]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:32.571732430 (13h38m32.161077417s) [seg_pos={0 0}, upload=14140]: Begin upload segment... Aug 7 23:18:32.571774531 (13h38m32.161119528s) [seg_pos={0 0}, upload=14140]: Done begin upload segment. Aug 7 23:18:32.571805400 (13h38m32.161150417s) [noise=true , address=77.90.31.12:4589 , piece_id=ZMKTDC4M5JHWVARXQDQAKRFSXM3OQQZ2IGQASJCK2OKYX3IQXHVA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.571844173 (13h38m32.161189161s) [noise=true , address=104.36.82.9:4007 , piece_id=VZLD7RU4SK5MAQE22D7OBVT6XOQAN75UNDMT7JNPCRZNRKFX36XQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572218380 (13h38m32.161563367s) [noise=true , address=104.36.82.10:4077 , piece_id=WYGEC4FYUPAR45MZIWLZQ2ASF5SUPECS4YI3UTRRHEVAKUEYCGYA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572289677 (13h38m32.161634654s) [noise=true , address=77.90.31.17:4770 , piece_id=ISA43NF4AKPK2SGYDPKT6ZIAD5WJ27A6GT6JZA2GHQUCAQBRRAFA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572494809 (13h38m32.161839796s) [noise=true , address=104.36.82.7:4911 , piece_id=FAMSQZJMALQIEAG5BVR5OV5WHEHN432Y4DZD3DNPC5CKP7X7GMQA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572588257 (13h38m32.161933244s) [noise=true , address=77.90.31.15:4697 , piece_id=6RYZJ22TCTJFRBMQV7M545VHB475CHDV2DBKCGO3MJP65VEMJ5FQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572772450 (13h38m32.162117427s) [noise=true , address=45.91.116.12:4225 , piece_id=22GAFTM5SITGTVJEELA3BJP7W2QZW7BVYJGJMKJOLE2RF2GZ5CGQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572865959 (13h38m32.162210946s) [noise=true , address=45.91.116.12:4227 , piece_id=T3JVU7VMSYBTYXBVGRD7LSNIG3PWF3QDVDNDQ24JQOSV7ZFIODNA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573029050 (13h38m32.162374037s) [noise=true , address=77.90.31.19:4842 , piece_id=3ZRHNMBPQUJH5SZZ75UJWLYIYZ5YI7CC5OMIJJFM7WVKZWJX6R3A, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573227882 (13h38m32.162572869s) [noise=true , address=104.36.82.4:4804 , piece_id=LTSBCPH74YSWH7IHN6MH4NWTBJ5FQ7534VERPCY5NIDV24KYPXMQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573293797 (13h38m32.162638774s) [noise=true , address=104.36.82.15:4259 , piece_id=GFCTNTXSY4WVSTPAD3AZIYCHE4O5EN24WN25VUNCRNWCSUWCYBMA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573504962 (13h38m32.162849949s) [noise=true , address=77.90.31.16:4734 , piece_id=OJPJZ5RF7QY4KNFBZ3MAVTA63HMC4JRG3HQLBN7SVROVKFQZY7BQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573527936 (13h38m32.162872923s) [noise=true , address=45.91.116.7:4009 , piece_id=VKYMGNJEGFLB5M3L3WLD37C5ZILQGLRJROSQ4ZEIBN46CQSY5JZQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.574078970 (13h38m32.163423947s) [noise=true , address=45.91.116.11:4119 , piece_id=UXO65SB4OZ57BCKOWGE2VDV7KCSKCGE7I7K4RTGRYJ7VU5GKFPHQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.574216453 (13h38m32.16356143s) [noise=true , address=104.36.82.12:4149 , piece_id=2K5BCSGU4UN43HF7VHG3MBNMQEGSIZPPHTDHDFI3XPY5DDG6NKOA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.576753347 (13h38m32.166098334s) [noise=true , address=104.36.82.8:4971 , piece_id=DNQVVV7QT2A2NFMIRURJ5IZRFPYZVX2NADRTX2P2YORA2CKMWY6Q, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:34.327895253 (13h38m33.91724024s) [upload=14140]: Next returned nil segment Aug 7 23:18:34.327912295 (13h38m33.917257282s) [upload=14140]: Waiting for error group managing segments... Aug 7 23:18:34.716828436 (13h38m34.306173413s) [noise=true , address=104.36.82.7:4911 , piece_id=FAMSQZJMALQIEAG5BVR5OV5WHEHN432Y4DZD3DNPC5CKP7X7GMQA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.729509878 (13h38m34.318854865s) [noise=true , address=104.36.82.12:4149 , piece_id=2K5BCSGU4UN43HF7VHG3MBNMQEGSIZPPHTDHDFI3XPY5DDG6NKOA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.738858565 (13h38m34.328203592s) [noise=true , address=104.36.82.4:4804 , piece_id=LTSBCPH74YSWH7IHN6MH4NWTBJ5FQ7534VERPCY5NIDV24KYPXMQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.783948702 (13h38m34.373293679s) [noise=true , address=77.90.31.19:4842 , piece_id=3ZRHNMBPQUJH5SZZ75UJWLYIYZ5YI7CC5OMIJJFM7WVKZWJX6R3A, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.824404112 (13h38m34.413749099s) [noise=true , address=104.36.82.15:4259 , piece_id=GFCTNTXSY4WVSTPAD3AZIYCHE4O5EN24WN25VUNCRNWCSUWCYBMA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.863114634 (13h38m34.452459611s) [noise=true , address=104.36.82.9:4007 , piece_id=VZLD7RU4SK5MAQE22D7OBVT6XOQAN75UNDMT7JNPCRZNRKFX36XQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.889984711 (13h38m34.479329698s) [noise=true , address=77.90.31.16:4734 , piece_id=OJPJZ5RF7QY4KNFBZ3MAVTA63HMC4JRG3HQLBN7SVROVKFQZY7BQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.982222673 (13h38m34.57156766s) [noise=true , address=104.36.82.10:4077 , piece_id=WYGEC4FYUPAR45MZIWLZQ2ASF5SUPECS4YI3UTRRHEVAKUEYCGYA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.999676977 (13h38m34.589021964s) [noise=true , address=77.90.31.17:4770 , piece_id=ISA43NF4AKPK2SGYDPKT6ZIAD5WJ27A6GT6JZA2GHQUCAQBRRAFA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.002083712 (13h38m34.591428699s) [noise=true , address=45.91.116.12:4225 , piece_id=22GAFTM5SITGTVJEELA3BJP7W2QZW7BVYJGJMKJOLE2RF2GZ5CGQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.055266482 (13h38m34.644611469s) [noise=true , address=45.91.116.12:4227 , piece_id=T3JVU7VMSYBTYXBVGRD7LSNIG3PWF3QDVDNDQ24JQOSV7ZFIODNA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.081593190 (13h38m34.670938177s) [noise=true , address=45.91.116.11:4119 , piece_id=UXO65SB4OZ57BCKOWGE2VDV7KCSKCGE7I7K4RTGRYJ7VU5GKFPHQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.123608726 (13h38m34.712953703s) [noise=true , address=77.90.31.12:4589 , piece_id=ZMKTDC4M5JHWVARXQDQAKRFSXM3OQQZ2IGQASJCK2OKYX3IQXHVA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.123631779 (13h38m34.712976766s) [seg_pos={0 0}, upload=14140]: Segment reached optimal threshold of 13 pieces. Aug 7 23:18:35.123785944 (13h38m34.713130931s) [noise=true , address=45.91.116.7:4009 , piece_id=VKYMGNJEGFLB5M3L3WLD37C5ZILQGLRJROSQ4ZEIBN46CQSY5JZQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:35.123945530 (13h38m34.713290517s) [noise=true , address=77.90.31.15:4697 , piece_id=6RYZJ22TCTJFRBMQV7M545VHB475CHDV2DBKCGO3MJP65VEMJ5FQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:35.123972431 (13h38m34.713317418s) [noise=true , address=104.36.82.8:4971 , piece_id=DNQVVV7QT2A2NFMIRURJ5IZRFPYZVX2NADRTX2P2YORA2CKMWY6Q, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: read tcp 188.166.225.170:44698->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:18:35.123986628 (13h38m34.713331615s) [seg_pos={0 0}, upload=14140]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:18:35.124083804 (13h38m34.713428791s) [upload=14140]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:18:35.124087611 (13h38m34.713432599s) [upload=14140]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:35.414558916 (13h38m35.003903903s) [upload=14140]: Done uploading segments... Aug 7 23:18:35.414577682 (13h38m35.00392267s) [upload=14140]: Upload finished. err: <nil> Aug 7 23:18:58.905554387 (13h38m58.494899394s) [upload=14146]: Starting upload Aug 7 23:18:58.905620613 (13h38m58.4949656s) [upload=14146]: Done starting upload Aug 7 23:18:58.905683373 (13h38m58.49502836s) [upload=14146]: Uploading segments... Aug 7 23:18:58.905783355 (13h38m58.495128342s) [upload=14146]: Got next segment. Inline: false Aug 7 23:18:58.905832459 (13h38m58.495177446s) [upload=14146]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:58.905836627 (13h38m58.495181614s) [upload=14146]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:19:28.246792926 (13h39m27.836137913s) [upload=14146]: Next segment err: context deadline exceeded Aug 7 23:19:28.246867379 (13h39m27.836212366s) [upload=14146]: Done uploading segments... Aug 7 23:19:28.246884292 (13h39m27.836229269s) [upload=14146]: Upload finished. err: context deadline exceeded Duration: 17.975860599s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [..................................................X.....X.X.] duration: 4.348248665s FAIL (2025-08-07 23:22:56 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.348277148s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...............................................XXXX.X..XX...] duration: 9.717884966s FAIL (2025-08-07 23:22:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.717907619s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X...................X....................X...........] duration: 6.25074298s FAIL (2025-08-07 23:22:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after e85db788dbeec38ad8084daa355e1bf59b8b718af998fc40c7451ca1ba6627b2 != before 1cad814b0341363008cdb7d22dd66bb6d87f0105cd39850888ced4405a0b4b43 ======LOGS====== upload Aug 7 23:13:43.967401320 (13h33m43.556746307s) [upload=14069]: Uploading segments... Aug 7 23:13:43.967498206 (13h33m43.556843193s) [upload=14069]: Got next segment. Inline: false Aug 7 23:13:43.967560455 (13h33m43.556905442s) [upload=14069]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:43.967568550 (13h33m43.556913577s) [upload=14069]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:13:44.414536988 (13h33m44.003881975s) [seg_pos={0 0}, upload=14069]: Begin upload segment... Aug 7 23:13:44.414587104 (13h33m44.003932091s) [seg_pos={0 0}, upload=14069]: Done begin upload segment. Aug 7 23:13:44.414606933 (13h33m44.00395192s) [noise=true , address=104.36.82.60:4362 , piece_id=LJMWWXQ4NXJY54S7WPOSCO37ALDF5PT2GEC5LKOAKB6YEW2PLEAA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.414666236 (13h33m44.004011223s) [noise=true , address=104.36.82.43:4868 , piece_id=TOYHNT5XLP5YVN6ZFSWOS4UJOSIIOKUKWXP4CNSG56CHGEUZFQDQ, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.414966160 (13h33m44.004311147s) [noise=true , address=104.36.82.52:4086 , piece_id=DRDBLVU5QJKKQC5WCUJROGJYG7OQKTIRVAXDLFHQW3FIC2BV2WIA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.415118381 (13h33m44.004463369s) [noise=true , address=104.36.82.16:4296 , piece_id=HCYT2XGA5RGQQGMWKTDKJIF6OQWK6NR4Z4SUPSG5IOA6ECK3EYPQ, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.415228682 (13h33m44.00457367s) [noise=true , address=79.127.205.233:4621 , piece_id=DSV45XV43BCLRGH7BJOICWBBDBCYFZXDC4CVACYRNT4LI745DK7Q, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.415496494 (13h33m44.004841471s) [noise=true , address=104.36.82.58:4291 , piece_id=DP64RXRSK2NLFM3N7TSZCZL4PEGZVHEQJHPJFXAIQXXLOEPQWY5Q, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.415745370 (13h33m44.005090357s) [noise=true , address=45.91.116.13:4158 , piece_id=62VB7AEIWCPAVJSLX3P2TOQRSALK5A6ZAD3ZFVQINR2IVT3AOXXQ, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.416026788 (13h33m44.005371766s) [noise=true , address=104.36.82.36:4642 , piece_id=3X7E6QOZKZ5KGPLWHIWL6WZRGOFHKZLFIZRKVZM3CGJM6VLA6OAA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.416256569 (13h33m44.005601556s) [noise=true , address=104.36.82.14:4226 , piece_id=CLREJCNEC4WX6IGMUEJIV2IEZXFAVLSXLZVYMWJJBHHDHCQ6DVQA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.416456080 (13h33m44.005801067s) [noise=true , address=64.49.13.137:4499 , piece_id=7D5JZSVLNEFT26CX3TBD4L27WIZU2IWH2JK3O7IHPAVDCUDBZAJQ, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.416552746 (13h33m44.005897753s) [noise=true , address=45.91.116.14:4194 , piece_id=2LO5TLHD4ZILDKAMW6EXNTMDHG4PCRFYS6F5EMLUN4EBP4RWEUPA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.416803596 (13h33m44.006148583s) [noise=true , address=45.91.116.11:4150 , piece_id=2YORTWP2L67BV5ENJQ5CKPY3NK337SK6WM5XL77RUZ6LMQWXVORA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.417042221 (13h33m44.006387208s) [noise=true , address=104.36.82.53:4126 , piece_id=A6LKRBLYVWJ7QN4UJ7JPDXKLCI5ZFAJ6I6ICCXDE7V7G2F5R65UA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.417277101 (13h33m44.006622088s) [noise=true , address=104.36.82.59:4332 , piece_id=K5TGW4E726UEFXZQSKWGYU2PHFX2IYE2BYKHXUG2WQXTZV5GV5ZA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.417316085 (13h33m44.006661072s) [noise=true , address=104.36.82.8:4977 , piece_id=PDXPXV3IJ7Y4P25A6BS3HTKCYLLNMRYIHAMLJCVV3QDMHGN4PXKQ, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.417537459 (13h33m44.006882436s) [noise=true , address=104.36.82.2:4757 , piece_id=LJ3JPH3HYJ7Z5QIWGSKWC3KGH7QJXYT3IDTFEN326A6QOLGAWVZA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.417738975 (13h33m44.007083952s) [noise=true , address=64.49.13.26:4946 , piece_id=VYQT7NJAPN6XN6Z5PKEBJCPSYWRZHR5ZCGG6257JA3Q2BTK6S4FA, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:44.417952763 (13h33m44.00729774s) [noise=true , address=104.36.82.9:4037 , piece_id=RISOPBKZ6Y3UYM3HSBMJT6EI3ZXGQKGANDYNUSDK3IGPCNLC4QTQ, seg_pos={0 0}, upload=14069]: Uploading piece... Aug 7 23:13:45.514309387 (13h33m45.103654374s) [upload=14069]: Next returned nil segment Aug 7 23:13:45.514327020 (13h33m45.103672007s) [upload=14069]: Waiting for error group managing segments... Aug 7 23:13:45.795258156 (13h33m45.384603143s) [noise=true , address=64.49.13.137:4499 , piece_id=7D5JZSVLNEFT26CX3TBD4L27WIZU2IWH2JK3O7IHPAVDCUDBZAJQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:45.802263469 (13h33m45.391608456s) [noise=true , address=79.127.205.233:4621 , piece_id=DSV45XV43BCLRGH7BJOICWBBDBCYFZXDC4CVACYRNT4LI745DK7Q, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:45.887877424 (13h33m45.477222401s) [noise=true , address=64.49.13.26:4946 , piece_id=VYQT7NJAPN6XN6Z5PKEBJCPSYWRZHR5ZCGG6257JA3Q2BTK6S4FA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.350209185 (13h33m45.939554172s) [noise=true , address=104.36.82.60:4362 , piece_id=LJMWWXQ4NXJY54S7WPOSCO37ALDF5PT2GEC5LKOAKB6YEW2PLEAA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.389672887 (13h33m45.979017874s) [noise=true , address=104.36.82.52:4086 , piece_id=DRDBLVU5QJKKQC5WCUJROGJYG7OQKTIRVAXDLFHQW3FIC2BV2WIA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.399409194 (13h33m45.988754171s) [noise=true , address=104.36.82.58:4291 , piece_id=DP64RXRSK2NLFM3N7TSZCZL4PEGZVHEQJHPJFXAIQXXLOEPQWY5Q, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.424971119 (13h33m46.014316106s) [noise=true , address=104.36.82.36:4642 , piece_id=3X7E6QOZKZ5KGPLWHIWL6WZRGOFHKZLFIZRKVZM3CGJM6VLA6OAA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.450918913 (13h33m46.0402639s) [noise=true , address=45.91.116.13:4158 , piece_id=62VB7AEIWCPAVJSLX3P2TOQRSALK5A6ZAD3ZFVQINR2IVT3AOXXQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.492609174 (13h33m46.081954161s) [noise=true , address=104.36.82.53:4126 , piece_id=A6LKRBLYVWJ7QN4UJ7JPDXKLCI5ZFAJ6I6ICCXDE7V7G2F5R65UA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.516716376 (13h33m46.106061353s) [noise=true , address=104.36.82.9:4037 , piece_id=RISOPBKZ6Y3UYM3HSBMJT6EI3ZXGQKGANDYNUSDK3IGPCNLC4QTQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.592315332 (13h33m46.181660319s) [noise=true , address=104.36.82.43:4868 , piece_id=TOYHNT5XLP5YVN6ZFSWOS4UJOSIIOKUKWXP4CNSG56CHGEUZFQDQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.746385819 (13h33m46.335730806s) [noise=true , address=45.91.116.11:4150 , piece_id=2YORTWP2L67BV5ENJQ5CKPY3NK337SK6WM5XL77RUZ6LMQWXVORA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.786539771 (13h33m46.375884758s) [noise=true , address=104.36.82.59:4332 , piece_id=K5TGW4E726UEFXZQSKWGYU2PHFX2IYE2BYKHXUG2WQXTZV5GV5ZA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.838001699 (13h33m46.427346676s) [noise=true , address=45.91.116.14:4194 , piece_id=2LO5TLHD4ZILDKAMW6EXNTMDHG4PCRFYS6F5EMLUN4EBP4RWEUPA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil> Aug 7 23:13:46.838017648 (13h33m46.427362636s) [seg_pos={0 0}, upload=14069]: Segment reached optimal threshold of 14 pieces. Aug 7 23:13:46.838217681 (13h33m46.427562658s) [noise=true , address=104.36.82.14:4226 , piece_id=CLREJCNEC4WX6IGMUEJIV2IEZXFAVLSXLZVYMWJJBHHDHCQ6DVQA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126xWe3ELGmYQfSUk3tYtGTSa6LnGfDNWcGT164pR6tHEtFPAFt): protocol: expected piece hash; context canceled; EOF Aug 7 23:13:46.838422624 (13h33m46.427767601s) [noise=true , address=104.36.82.2:4757 , piece_id=LJ3JPH3HYJ7Z5QIWGSKWC3KGH7QJXYT3IDTFEN326A6QOLGAWVZA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q): protocol: expected piece hash; context canceled; EOF Aug 7 23:13:46.838575365 (13h33m46.427920352s) [noise=true , address=104.36.82.16:4296 , piece_id=HCYT2XGA5RGQQGMWKTDKJIF6OQWK6NR4Z4SUPSG5IOA6ECK3EYPQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cKhDbA4ptwux2LYNPNAJ7Qh8jECsGWd1KWaK6T6pD38bJWdt1): protocol: rpc: dial tcp 104.36.82.16:4296: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:13:46.838627195 (13h33m46.427972172s) [noise=true , address=104.36.82.8:4977 , piece_id=PDXPXV3IJ7Y4P25A6BS3HTKCYLLNMRYIHAMLJCVV3QDMHGN4PXKQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WCAm12tXwimFSPqz7Seh6A4oxWqQVxqsF459xux4a5CqjpbAT): protocol: write tcp 188.166.225.170:48060->104.36.82.8:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:13:46.838635551 (13h33m46.427980538s) [seg_pos={0 0}, upload=14069]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:13:46.838702128 (13h33m46.428047115s) [upload=14069]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:13:46.838705895 (13h33m46.428050882s) [upload=14069]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:47.127771955 (13h33m46.717116932s) [upload=14069]: Done uploading segments... Aug 7 23:13:47.127791082 (13h33m46.717136069s) [upload=14069]: Upload finished. err: <nil> Duration: 6.250767728s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 4 out of 60 is failed (threshold=3) [..................................................XXXX......] duration: 2.492920279s FAIL (2025-08-07 23:22:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .175884539 (13h29m53.765229526s) [upload=14007]: Starting upload Aug 7 23:09:54.175937640 (13h29m53.765282618s) [upload=14007]: Done starting upload Aug 7 23:09:54.175979842 (13h29m53.765324829s) [upload=14007]: Uploading segments... Aug 7 23:09:54.176041389 (13h29m53.765386376s) [upload=14007]: Got next segment. Inline: true Aug 7 23:09:54.176045487 (13h29m53.765390474s) [upload=14007]: Waiting for error group managing segments... Aug 7 23:09:54.176077538 (13h29m53.765422525s) [upload=14007]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:54.176082318 (13h29m53.765427305s) [upload=14007]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:54.176084522 (13h29m53.765429509s) [upload=14007]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:54.453731274 (13h29m54.043076261s) [upload=14007]: Done uploading segments... Aug 7 23:09:54.453750932 (13h29m54.043095909s) [upload=14007]: Upload finished. err: <nil> Aug 7 23:10:39.330325211 (13h30m38.919670198s) [upload=14021]: Starting upload Aug 7 23:10:39.330417358 (13h30m38.919762345s) [upload=14021]: Done starting upload Aug 7 23:10:39.330504384 (13h30m38.919849371s) [upload=14021]: Uploading segments... Aug 7 23:10:39.330543368 (13h30m38.919888355s) [upload=14021]: Got next segment. Inline: true Aug 7 23:10:39.330546675 (13h30m38.919891662s) [upload=14021]: Waiting for error group managing segments... Aug 7 23:10:39.330581371 (13h30m38.919926349s) [upload=14021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:39.330585309 (13h30m38.919930286s) [upload=14021]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:39.330587663 (13h30m38.91993265s) [upload=14021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:39.610133078 (13h30m39.199478065s) [upload=14021]: Done uploading segments... Aug 7 23:10:39.610150061 (13h30m39.199495048s) [upload=14021]: Upload finished. err: <nil> Aug 7 23:11:32.695175239 (13h31m32.284520226s) [upload=14034]: Starting upload Aug 7 23:11:32.695285580 (13h31m32.284630567s) [upload=14034]: Done starting upload Aug 7 23:11:32.695336297 (13h31m32.284681284s) [upload=14034]: Uploading segments... Aug 7 23:11:32.695365753 (13h31m32.28471074s) [upload=14034]: Got next segment. Inline: true Aug 7 23:11:32.695369610 (13h31m32.284714598s) [upload=14034]: Waiting for error group managing segments... Aug 7 23:11:32.695401581 (13h31m32.284746569s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:32.695405168 (13h31m32.284750155s) [upload=14034]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:32.695407543 (13h31m32.28475253s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:32.954750627 (13h31m32.544095614s) [upload=14034]: Done uploading segments... Aug 7 23:11:32.954767650 (13h31m32.544112627s) [upload=14034]: Upload finished. err: <nil> Aug 7 23:12:37.502051138 (13h32m37.091396125s) [upload=14054]: Starting upload Aug 7 23:12:37.502104370 (13h32m37.091449357s) [upload=14054]: Done starting upload Aug 7 23:12:37.502150869 (13h32m37.091495856s) [upload=14054]: Uploading segments... Aug 7 23:12:37.502182519 (13h32m37.091527506s) [upload=14054]: Got next segment. Inline: true Aug 7 23:12:37.502185986 (13h32m37.091530963s) [upload=14054]: Waiting for error group managing segments... Aug 7 23:12:37.502216213 (13h32m37.0915612s) [upload=14054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.502219780 (13h32m37.091564767s) [upload=14054]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.502222425 (13h32m37.091567412s) [upload=14054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.772497714 (13h32m37.361842701s) [upload=14054]: Done uploading segments... Aug 7 23:12:37.772511151 (13h32m37.361856138s) [upload=14054]: Upload finished. err: <nil> Aug 7 23:13:44.834965069 (13h33m44.424310046s) [upload=14070]: Starting upload Aug 7 23:13:44.835030293 (13h33m44.42437528s) [upload=14070]: Done starting upload Aug 7 23:13:44.835093484 (13h33m44.424438471s) [upload=14070]: Uploading segments... Aug 7 23:13:44.835133321 (13h33m44.424478308s) [upload=14070]: Got next segment. Inline: true Aug 7 23:13:44.835136998 (13h33m44.424481985s) [upload=14070]: Waiting for error group managing segments... Aug 7 23:13:44.835214376 (13h33m44.424559363s) [upload=14070]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:44.835220728 (13h33m44.424565715s) [upload=14070]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:44.835222972 (13h33m44.424567959s) [upload=14070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:45.220889346 (13h33m44.810234343s) [upload=14070]: Done uploading segments... Aug 7 23:13:45.220937208 (13h33m44.810282205s) [upload=14070]: Upload finished. err: <nil> Aug 7 23:14:39.930436267 (13h34m39.519781254s) [upload=14085]: Starting upload Aug 7 23:14:39.930523634 (13h34m39.519868621s) [upload=14085]: Done starting upload Aug 7 23:14:39.930574131 (13h34m39.519919118s) [upload=14085]: Uploading segments... Aug 7 23:14:39.930614779 (13h34m39.519959766s) [upload=14085]: Got next segment. Inline: true Aug 7 23:14:39.930618847 (13h34m39.519963824s) [upload=14085]: Waiting for error group managing segments... Aug 7 23:14:39.930656519 (13h34m39.520001506s) [upload=14085]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:39.930660767 (13h34m39.520005754s) [upload=14085]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:39.930663112 (13h34m39.520008099s) [upload=14085]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:40.207143612 (13h34m39.796488599s) [upload=14085]: Done uploading segments... Aug 7 23:14:40.207161857 (13h34m39.796506844s) [upload=14085]: Upload finished. err: <nil> Aug 7 23:14:59.171620401 (13h34m58.760965388s) [upload=14091]: Starting upload Aug 7 23:14:59.171676218 (13h34m58.761021205s) [upload=14091]: Done starting upload Aug 7 23:14:59.171718468 (13h34m58.761063455s) [upload=14091]: Uploading segments... Aug 7 23:14:59.171763204 (13h34m58.761108191s) [upload=14091]: Got next segment. Inline: true Aug 7 23:14:59.171776990 (13h34m58.761121977s) [upload=14091]: Waiting for error group managing segments... Aug 7 23:14:59.171816787 (13h34m58.761161774s) [upload=14091]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:59.171820764 (13h34m58.761165741s) [upload=14091]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:59.171823429 (13h34m58.761168416s) [upload=14091]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:28.519847650 (13h35m28.109192637s) [upload=14091]: Done uploading segments... Aug 7 23:15:28.519870794 (13h35m28.109215781s) [upload=14091]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:34.319815009 (13h35m33.909159996s) [upload=14100]: Starting upload Aug 7 23:15:34.319867980 (13h35m33.909212967s) [upload=14100]: Done starting upload Aug 7 23:15:34.319916702 (13h35m33.909261679s) [upload=14100]: Uploading segments... Aug 7 23:15:34.319945739 (13h35m33.909290726s) [upload=14100]: Got next segment. Inline: true Aug 7 23:15:34.319949416 (13h35m33.909294403s) [upload=14100]: Waiting for error group managing segments... Aug 7 23:15:34.319988731 (13h35m33.909333718s) [upload=14100]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:34.319992698 (13h35m33.909337685s) [upload=14100]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:34.319995163 (13h35m33.90934015s) [upload=14100]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:03.672763058 (13h36m3.262108045s) [upload=14100]: Done uploading segments... Aug 7 23:16:03.672786503 (13h36m3.26213149s) [upload=14100]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:43.252280188 (13h36m42.841625175s) [upload=14114]: Starting upload Aug 7 23:16:43.252335193 (13h36m42.84168018s) [upload=14114]: Done starting upload Aug 7 23:16:43.252381041 (13h36m42.841726018s) [upload=14114]: Uploading segments... Aug 7 23:16:43.252412271 (13h36m42.841757258s) [upload=14114]: Got next segment. Inline: true Aug 7 23:16:43.252416208 (13h36m42.841761195s) [upload=14114]: Waiting for error group managing segments... Aug 7 23:16:43.252453641 (13h36m42.841798628s) [upload=14114]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:43.252457388 (13h36m42.841802375s) [upload=14114]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:43.252459622 (13h36m42.841804609s) [upload=14114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:12.607365819 (13h37m12.196710806s) [upload=14114]: Done uploading segments... Aug 7 23:17:12.607385377 (13h37m12.196730364s) [upload=14114]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:56.238720000 (13h37m55.828064987s) [upload=14132]: Starting upload Aug 7 23:17:56.238785927 (13h37m55.828130914s) [upload=14132]: Done starting upload Aug 7 23:17:56.238851903 (13h37m55.82819689s) [upload=14132]: Uploading segments... Aug 7 23:17:56.238885677 (13h37m55.828230664s) [upload=14132]: Got next segment. Inline: true Aug 7 23:17:56.238888933 (13h37m55.82823392s) [upload=14132]: Waiting for error group managing segments... Aug 7 23:17:56.238932207 (13h37m55.828277194s) [upload=14132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:56.238935984 (13h37m55.828280971s) [upload=14132]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:56.238938278 (13h37m55.828283265s) [upload=14132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:25.593341271 (13h38m25.182686258s) [upload=14132]: Done uploading segments... Aug 7 23:18:25.593355268 (13h38m25.182700245s) [upload=14132]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.492944885s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 4 out of 60 is failed (threshold=3) [.............................................X...X..X.X.....] duration: 1.92203499s FAIL (2025-08-07 23:22:39 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.92206071s core uplink us1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)