Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [....................XXXXXXXXXXXXXXXXX.X.....................] duration: 5.635706646s FAIL (2025-08-07 22:45:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.635723879s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........X..............X.........X....X.................X.] duration: 7.054280033s FAIL (2025-08-07 22:45:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 81af0bd8b34b2e69389f7db10076d194a3d6fbdd58cd01527d1b66bf7a00fe45 != before 7800c7ceda312caa58fa18fda98736593dbd42db38936a6dbdeb420eade030ff ======LOGS====== pload=13556]: Done starting upload Aug 7 22:44:47.819148413 (13h4m47.4084934s) [upload=13556]: Uploading segments... Aug 7 22:44:47.819394214 (13h4m47.408739201s) [upload=13556]: Got next segment. Inline: false Aug 7 22:44:47.819437876 (13h4m47.408782864s) [upload=13556]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:44:47.819442115 (13h4m47.408787092s) [upload=13556]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:44:48.269086796 (13h4m47.858431783s) [seg_pos={0 0}, upload=13556]: Begin upload segment... Aug 7 22:44:48.269139916 (13h4m47.858484903s) [seg_pos={0 0}, upload=13556]: Done begin upload segment. Aug 7 22:44:48.269169674 (13h4m47.858514661s) [noise=true , address=104.36.82.45:4938 , piece_id=NF4CEKTH5KGR2VLO64EISWJ64CRKFHKXBP3R77ZAZ3HVAFAYJJ4Q, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.269176958 (13h4m47.858521965s) [noise=true , address=104.36.82.60:4343 , piece_id=7ZZ3C77QFADRAQU3ZQF3LRXC5S6NF3XBNFXZ6XICWDCI3HFZ6EUQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.269619725 (13h4m47.858964712s) [noise=true , address=104.36.82.41:4768 , piece_id=GSW4YJR4SSW5ZKGPYI7L2KRRSHL3FTEFUE2RSARPDARD7CWOSXBA, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.269688626 (13h4m47.859033623s) [noise=true , address=104.36.82.9:4040 , piece_id=7YWI2TUXLX4QSI6GRFGIGLZRU3RUEUMJNJRVGAJON3NVRIZJ27DA, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.269932232 (13h4m47.85927722s) [noise=true , address=193.149.164.143:4696 , piece_id=D74PAAUDHRBQJSBQ7T5R6QDKUDUS4R7425SO7H53NHS4QZPTGBMQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.270291710 (13h4m47.859636717s) [noise=true , address=104.36.82.2:4742 , piece_id=TREABDVVCEXW3MN7U6TE5CWVGQXWW2Z5REAV7PEZWYRYNBK27ECQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.271203444 (13h4m47.860548432s) [noise=true , address=104.36.82.44:4893 , piece_id=IRD65MAG7YYBVEOPHERHYVPFA65YC7JQPJB6XKUQLHGKKMYTFLFQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.271407435 (13h4m47.860752422s) [noise=true , address=104.36.82.53:4122 , piece_id=DE5JKFJWUT4MIKUUQHWMXBTSK2MKDSTJ5VGJ5S2ALJXUTEJ3LUMA, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.271531252 (13h4m47.860876239s) [noise=true , address=104.36.82.59:4326 , piece_id=LJ5XKKU2EVRDGH5E4YAE5BRF5EIFM2BHG7BYMHOCQXIWUNS7DXRA, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.271734040 (13h4m47.861079027s) [noise=true , address=79.127.201.214:4295 , piece_id=6MJQFTPQRQL3OBVOWJ36WYWD3PVJBBCCMISXCJKC636VZPG474PA, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.271963008 (13h4m47.861307995s) [noise=true , address=104.36.82.8:4989 , piece_id=LPXKK2B443AVS624U7MPUP4GUEMUFJYVUBZY5ZJGZPV32WYW6T6A, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.272159905 (13h4m47.861504882s) [noise=true , address=104.36.82.43:4848 , piece_id=KQKHRNP6BMU7GRFQXGW7U5KSNF53NGMJIRFNNE5M3Q6MHGQJUC4Q, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.272343486 (13h4m47.861688463s) [noise=true , address=104.36.82.38:4715 , piece_id=BJDDYECUSWJH675QPKG4IP5PSTOS4Y77OMKQWG2BGZLAQE77GKHQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.272525674 (13h4m47.861870661s) [noise=true , address=104.36.82.15:4277 , piece_id=NCLDOZRBEK4OF7C4Q5DY4RX3ND73SMUM6QIRL5FHVOFBQYKOKIOQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.272744403 (13h4m47.86208939s) [noise=true , address=79.127.205.234:4597 , piece_id=CI2MXO5CGMRNZRAPXUCIGASHDL4KCMGAWVNMJESCLF3HTBRHWAGQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.272934386 (13h4m47.862279373s) [noise=true , address=104.36.82.46:4483 , piece_id=ZDGTDBBYXOWDCNUHAA7BLEGD4RAZOR6QLETRO4MG3MAGDE6RVLQQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.273120443 (13h4m47.86246543s) [noise=true , address=104.36.82.44:4901 , piece_id=UP7742ZXVVCLWBHJPYVKZH2C3W4M5NWE5QHEMSZGRWJM4Q72FIDA, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.273299345 (13h4m47.862644322s) [noise=true , address=104.36.82.22:4512 , piece_id=3BXMDY3JO7ZVVX6GF3LUH5RTR2GHVF5VJ2U5XEUF62VIUIXRVWSQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:49.376334243 (13h4m48.96567923s) [upload=13556]: Next returned nil segment Aug 7 22:44:49.376348381 (13h4m48.965693368s) [upload=13556]: Waiting for error group managing segments... Aug 7 22:44:49.668631065 (13h4m49.257976052s) [noise=true , address=79.127.205.234:4597 , piece_id=CI2MXO5CGMRNZRAPXUCIGASHDL4KCMGAWVNMJESCLF3HTBRHWAGQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:49.675850678 (13h4m49.265195665s) [noise=true , address=193.149.164.143:4696 , piece_id=D74PAAUDHRBQJSBQ7T5R6QDKUDUS4R7425SO7H53NHS4QZPTGBMQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.149126209 (13h4m49.738471196s) [noise=true , address=79.127.201.214:4295 , piece_id=6MJQFTPQRQL3OBVOWJ36WYWD3PVJBBCCMISXCJKC636VZPG474PA, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.188374551 (13h4m49.777719528s) [noise=true , address=104.36.82.43:4848 , piece_id=KQKHRNP6BMU7GRFQXGW7U5KSNF53NGMJIRFNNE5M3Q6MHGQJUC4Q, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.192657957 (13h4m49.782002944s) [noise=true , address=104.36.82.22:4512 , piece_id=3BXMDY3JO7ZVVX6GF3LUH5RTR2GHVF5VJ2U5XEUF62VIUIXRVWSQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.260069984 (13h4m49.849414972s) [noise=true , address=104.36.82.38:4715 , piece_id=BJDDYECUSWJH675QPKG4IP5PSTOS4Y77OMKQWG2BGZLAQE77GKHQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.295276123 (13h4m49.88462111s) [noise=true , address=104.36.82.45:4938 , piece_id=NF4CEKTH5KGR2VLO64EISWJ64CRKFHKXBP3R77ZAZ3HVAFAYJJ4Q, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.314186263 (13h4m49.90353125s) [noise=true , address=104.36.82.59:4326 , piece_id=LJ5XKKU2EVRDGH5E4YAE5BRF5EIFM2BHG7BYMHOCQXIWUNS7DXRA, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.335401744 (13h4m49.924746731s) [noise=true , address=104.36.82.8:4989 , piece_id=LPXKK2B443AVS624U7MPUP4GUEMUFJYVUBZY5ZJGZPV32WYW6T6A, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.350133970 (13h4m49.939478957s) [noise=true , address=104.36.82.41:4768 , piece_id=GSW4YJR4SSW5ZKGPYI7L2KRRSHL3FTEFUE2RSARPDARD7CWOSXBA, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.370519655 (13h4m49.959864642s) [noise=true , address=104.36.82.53:4122 , piece_id=DE5JKFJWUT4MIKUUQHWMXBTSK2MKDSTJ5VGJ5S2ALJXUTEJ3LUMA, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.407688108 (13h4m49.997033095s) [noise=true , address=104.36.82.9:4040 , piece_id=7YWI2TUXLX4QSI6GRFGIGLZRU3RUEUMJNJRVGAJON3NVRIZJ27DA, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.484565775 (13h4m50.073910762s) [noise=true , address=104.36.82.15:4277 , piece_id=NCLDOZRBEK4OF7C4Q5DY4RX3ND73SMUM6QIRL5FHVOFBQYKOKIOQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.666687912 (13h4m50.256032899s) [noise=true , address=104.36.82.60:4343 , piece_id=7ZZ3C77QFADRAQU3ZQF3LRXC5S6NF3XBNFXZ6XICWDCI3HFZ6EUQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.666718511 (13h4m50.256063498s) [seg_pos={0 0}, upload=13556]: Segment reached optimal threshold of 14 pieces. Aug 7 22:44:50.667127924 (13h4m50.256472911s) [noise=true , address=104.36.82.2:4742 , piece_id=TREABDVVCEXW3MN7U6TE5CWVGQXWW2Z5REAV7PEZWYRYNBK27ECQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WsGo4te3PENFo9YRioYFrhkeRQvztnLVZJV4i38TV3TjKqBFT): protocol: expected piece hash; context canceled; EOF Aug 7 22:44:50.667260107 (13h4m50.256605094s) [noise=true , address=104.36.82.46:4483 , piece_id=ZDGTDBBYXOWDCNUHAA7BLEGD4RAZOR6QLETRO4MG3MAGDE6RVLQQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: write tcp 188.166.225.170:58454->104.36.82.46:4483: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:44:50.667317968 (13h4m50.256662945s) [noise=true , address=104.36.82.44:4893 , piece_id=IRD65MAG7YYBVEOPHERHYVPFA65YC7JQPJB6XKUQLHGKKMYTFLFQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ): protocol: rpc: dial tcp 104.36.82.44:4893: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:44:50.667373113 (13h4m50.256718091s) [noise=true , address=104.36.82.44:4901 , piece_id=UP7742ZXVVCLWBHJPYVKZH2C3W4M5NWE5QHEMSZGRWJM4Q72FIDA, seg_pos={0 0}, upload=13556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: rpc: dial tcp 104.36.82.44:4901: operation was canceled; conn closed: op Aug 7 22:44:50.667382952 (13h4m50.256727939s) [seg_pos={0 0}, upload=13556]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:44:50.667435813 (13h4m50.2567808s) [upload=13556]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:44:50.667439520 (13h4m50.256784507s) [upload=13556]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:44:50.982639800 (13h4m50.571984787s) [upload=13556]: Done uploading segments... Aug 7 22:44:50.982656853 (13h4m50.572001841s) [upload=13556]: Upload finished. err: <nil> Duration: 7.054310441s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 19 out of 60 is failed (threshold=3) [..............XXXXXXXXXXXXXXXX.XX..X........................] duration: 1.827837086s FAIL (2025-08-07 22:45:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7 22:22:45.818723517 (12h42m45.408068504s) [upload=13188]: Done starting upload Aug 7 22:22:45.818774856 (12h42m45.408119843s) [upload=13188]: Uploading segments... Aug 7 22:22:45.818807999 (12h42m45.408152996s) [upload=13188]: Got next segment. Inline: true Aug 7 22:22:45.818812908 (12h42m45.408157895s) [upload=13188]: Waiting for error group managing segments... Aug 7 22:22:45.818870779 (12h42m45.408215776s) [upload=13188]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:45.818879214 (12h42m45.408224202s) [upload=13188]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:45.818883494 (12h42m45.408228481s) [upload=13188]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:15.175828623 (12h43m14.76517361s) [upload=13188]: Done uploading segments... Aug 7 22:23:15.175844935 (12h43m14.765189912s) [upload=13188]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:16.350710694 (12h43m15.940055691s) [upload=13195]: Starting upload Aug 7 22:23:16.350768756 (12h43m15.940113733s) [upload=13195]: Done starting upload Aug 7 22:23:16.350814804 (12h43m15.940159791s) [upload=13195]: Uploading segments... Aug 7 22:23:16.350849079 (12h43m15.940194056s) [upload=13195]: Got next segment. Inline: true Aug 7 22:23:16.350852777 (12h43m15.940197764s) [upload=13195]: Waiting for error group managing segments... Aug 7 22:23:16.350884928 (12h43m15.940229915s) [upload=13195]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:16.350888735 (12h43m15.940233722s) [upload=13195]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:16.350891020 (12h43m15.940236007s) [upload=13195]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:45.695642688 (12h43m45.284987675s) [upload=13195]: Done uploading segments... Aug 7 22:23:45.695657867 (12h43m45.285002854s) [upload=13195]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:08.254775832 (12h44m7.844120819s) [upload=13210]: Starting upload Aug 7 22:24:08.254857087 (12h44m7.844202074s) [upload=13210]: Done starting upload Aug 7 22:24:08.254936559 (12h44m7.844281546s) [upload=13210]: Uploading segments... Aug 7 22:24:08.255031861 (12h44m7.844376858s) [upload=13210]: Got next segment. Inline: true Aug 7 22:24:08.255037032 (12h44m7.844382019s) [upload=13210]: Waiting for error group managing segments... Aug 7 22:24:08.255078220 (12h44m7.844423207s) [upload=13210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:08.255093450 (12h44m7.844438437s) [upload=13210]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:08.255096435 (12h44m7.844441422s) [upload=13210]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:37.608974587 (12h44m37.198319574s) [upload=13210]: Done uploading segments... Aug 7 22:24:37.609189158 (12h44m37.198534195s) [upload=13210]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:19.078840222 (12h45m18.668185229s) [upload=13227]: Starting upload Aug 7 22:25:19.078945333 (12h45m18.66829035s) [upload=13227]: Done starting upload Aug 7 22:25:19.079016760 (12h45m18.668361767s) [upload=13227]: Uploading segments... Aug 7 22:25:19.079058389 (12h45m18.668403377s) [upload=13227]: Got next segment. Inline: true Aug 7 22:25:19.079064030 (12h45m18.668409007s) [upload=13227]: Waiting for error group managing segments... Aug 7 22:25:19.079115388 (12h45m18.668460375s) [upload=13227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:19.079121570 (12h45m18.668466557s) [upload=13227]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:19.079126009 (12h45m18.668471077s) [upload=13227]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:48.426015265 (12h45m48.015360252s) [upload=13227]: Done uploading segments... Aug 7 22:25:48.426036394 (12h45m48.015381381s) [upload=13227]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:26:18.229360345 (12h46m17.818705332s) [upload=13243]: Starting upload Aug 7 22:26:18.229421673 (12h46m17.81876665s) [upload=13243]: Done starting upload Aug 7 22:26:18.229465397 (12h46m17.818810384s) [upload=13243]: Uploading segments... Aug 7 22:26:18.229495053 (12h46m17.81884004s) [upload=13243]: Got next segment. Inline: true Aug 7 22:26:18.229498680 (12h46m17.818843667s) [upload=13243]: Waiting for error group managing segments... Aug 7 22:26:18.229531593 (12h46m17.81887658s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:18.229535170 (12h46m17.818880157s) [upload=13243]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:18.229537614 (12h46m17.818882601s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:18.493923010 (12h46m18.083268007s) [upload=13243]: Done uploading segments... Aug 7 22:26:18.493948349 (12h46m18.083293346s) [upload=13243]: Upload finished. err: <nil> Aug 7 22:26:34.238583280 (12h46m33.827928267s) [upload=13246]: Starting upload Aug 7 22:26:34.238671890 (12h46m33.828016897s) [upload=13246]: Done starting upload Aug 7 22:26:34.238753585 (12h46m33.828098573s) [upload=13246]: Uploading segments... Aug 7 22:26:34.238795296 (12h46m33.828140283s) [upload=13246]: Got next segment. Inline: true Aug 7 22:26:34.238800546 (12h46m33.828145533s) [upload=13246]: Waiting for error group managing segments... Aug 7 22:26:34.238842697 (12h46m33.828187684s) [upload=13246]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:34.238847656 (12h46m33.828192633s) [upload=13246]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:34.238851193 (12h46m33.82819618s) [upload=13246]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:03.590020724 (12h47m3.179365711s) [upload=13246]: Done uploading segments... Aug 7 22:27:03.590046854 (12h47m3.179391861s) [upload=13246]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:08.455168905 (12h47m8.044513892s) [upload=13253]: Starting upload Aug 7 22:27:08.455224000 (12h47m8.044568977s) [upload=13253]: Done starting upload Aug 7 22:27:08.455266011 (12h47m8.044610998s) [upload=13253]: Uploading segments... Aug 7 22:27:08.455292903 (12h47m8.04463789s) [upload=13253]: Got next segment. Inline: true Aug 7 22:27:08.455296630 (12h47m8.044641607s) [upload=13253]: Waiting for error group managing segments... Aug 7 22:27:08.455327689 (12h47m8.044672676s) [upload=13253]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:08.455331626 (12h47m8.044676613s) [upload=13253]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:08.455334372 (12h47m8.044679349s) [upload=13253]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:37.813944606 (12h47m37.403289643s) [upload=13253]: Done uploading segments... Aug 7 22:27:37.813971608 (12h47m37.403316615s) [upload=13253]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:28:03.718747816 (12h48m3.308092803s) [upload=13266]: Starting upload Aug 7 22:28:03.718803663 (12h48m3.30814865s) [upload=13266]: Done starting upload Aug 7 22:28:03.718851525 (12h48m3.308196512s) [upload=13266]: Uploading segments... Aug 7 22:28:03.718879849 (12h48m3.308224836s) [upload=13266]: Got next segment. Inline: true Aug 7 22:28:03.718883686 (12h48m3.308228673s) [upload=13266]: Waiting for error group managing segments... Aug 7 22:28:03.718918023 (12h48m3.30826301s) [upload=13266]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:03.718922461 (12h48m3.308267448s) [upload=13266]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:03.718924966 (12h48m3.308269953s) [upload=13266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:04.001556505 (12h48m3.590901502s) [upload=13266]: Done uploading segments... Aug 7 22:28:04.001574169 (12h48m3.590919156s) [upload=13266]: Upload finished. err: <nil> Aug 7 22:28:53.653874975 (12h48m53.243219952s) [upload=13280]: Starting upload Aug 7 22:28:53.653923989 (12h48m53.243268976s) [upload=13280]: Done starting upload Aug 7 22:28:53.653970338 (12h48m53.243315315s) [upload=13280]: Uploading segments... Aug 7 22:28:53.654001658 (12h48m53.243346645s) [upload=13280]: Got next segment. Inline: true Aug 7 22:28:53.654005195 (12h48m53.243350182s) [upload=13280]: Waiting for error group managing segments... Aug 7 22:28:53.654037516 (12h48m53.243382503s) [upload=13280]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:53.654041253 (12h48m53.24338623s) [upload=13280]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:53.654043598 (12h48m53.243388575s) [upload=13280]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:53.931677747 (12h48m53.521022724s) [upload=13280]: Done uploading segments... Aug 7 22:28:53.931693927 (12h48m53.521038914s) [upload=13280]: Upload finished. err: <nil> Aug 7 22:29:13.069018260 (12h49m12.658363247s) [upload=13284]: Starting upload Aug 7 22:29:13.069075108 (12h49m12.658420096s) [upload=13284]: Done starting upload Aug 7 22:29:13.069124593 (12h49m12.65846958s) [upload=13284]: Uploading segments... Aug 7 22:29:13.069159800 (12h49m12.658504787s) [upload=13284]: Got next segment. Inline: true Aug 7 22:29:13.069163377 (12h49m12.658508364s) [upload=13284]: Waiting for error group managing segments... Aug 7 22:29:13.069195007 (12h49m12.658539995s) [upload=13284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:29:13.069198645 (12h49m12.658543632s) [upload=13284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:29:13.069201010 (12h49m12.658545997s) [upload=13284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:42.426836268 (12h49m42.016181256s) [upload=13284]: Done uploading segments... Aug 7 22:29:42.426899691 (12h49m42.016244678s) [upload=13284]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.827863777s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 21 out of 60 is failed (threshold=3) [...............XXXXXXXXXXXXXXXXXXX.X..X.....................] duration: 2.018833353s FAIL (2025-08-07 22:45:04 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.018864432s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 11 out of 60 is failed (threshold=3) [............................XXXXXXXXX..X.X..................] duration: 6.178767921s FAIL (2025-08-07 22:45:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=13236]: Begin upload segment... Aug 7 22:25:52.992307080 (12h45m52.581652067s) [seg_pos={0 0}, upload=13236]: Done begin upload segment. Aug 7 22:25:52.992331798 (12h45m52.581676775s) [noise=true , address=104.36.82.13:4213 , piece_id=5W5FD4WMMSZ7BKHFQI4GYOODXZ2P2MMAIQNK6CD6XILH22C53JGQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992683770 (12h45m52.582028757s) [noise=true , address=104.36.82.9:4024 , piece_id=VBFOHRWDKWRY2774KJVK3JXJSMG4CPVDBBECCSZVJHR4274KLRMA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992785475 (12h45m52.582130462s) [noise=true , address=104.36.82.22:4527 , piece_id=JBNJOSETDAB6VG7PGVXIBUP6GPF2FBPJURBZNBR3BJOGMRK3EIQA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992942396 (12h45m52.582287373s) [noise=true , address=45.91.116.11:4124 , piece_id=TLATR2WNDVRR3WWTT3FZARUJ2QWWQFZBDYXLZ7PUPFOAEGJI4BTQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993132439 (12h45m52.582477457s) [noise=true , address=104.36.82.5:4855 , piece_id=6EW7GNJ6KVWKNFHGFZVPEYPD4TDIVVZUXRW6AGJ3IGVANEAHYDUA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993337862 (12h45m52.582682849s) [noise=true , address=64.49.13.170:4717 , piece_id=52FN65VRQ24LBOGHJ22EQQM3MNI3EB2UXB57P5LD5GPLSABNS22Q, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993511144 (12h45m52.582856131s) [noise=true , address=104.36.82.60:4361 , piece_id=EPWXHNXPVAUX75FUAXY57QZDKRXIEDHRXFL6PEBQGXY2A3Q4KNHA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993715204 (12h45m52.583060191s) [noise=true , address=104.36.82.58:4281 , piece_id=5UH27M25HE2LEZJ53H7QBKU6I6N2JE6F4PZTLOE6XP3GEDCZLEEQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993820806 (12h45m52.583165784s) [noise=true , address=45.91.116.9:4284 , piece_id=LT3RP2EBVHIFIMAIGM3DW6QMPNLKZCK5FD4GQEQI6BFWYD56ZUBQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994068250 (12h45m52.583413237s) [noise=true , address=79.127.205.233:4633 , piece_id=LBO5XLAXBRDURMDMN6MZ4HLVSG4TTUZPWCFTOTB6QMWAS5IIRLBA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994092416 (12h45m52.583437393s) [noise=true , address=104.36.82.60:4346 , piece_id=3NU3QBO5DAW5ZB7NTAHWP4C7RE4BYDPHZ77JOUQT3R3CBTOMONLA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994295394 (12h45m52.583640371s) [noise=true , address=45.91.116.7:4037 , piece_id=2NDGWBVYO3D2RYJ57QSXT632TT24SMRNWA3YZ2YW2BN6TBBLMRQA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994307056 (12h45m52.583652033s) [noise=true , address=104.36.82.46:4505 , piece_id=A7XPGAYUTNB26A6DJQNQ4NZ4FG27FUBQNEH5AQ7IFLYR3DLGG6SQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994547126 (12h45m52.583892103s) [noise=true , address=104.36.82.61:4389 , piece_id=WBB2Y7YYCEML23D3CKJHGIEOUELRTVG3NXVGI4JDD7XPNQ4STENQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994782436 (12h45m52.584127423s) [noise=true , address=64.49.13.154:4934 , piece_id=LDQTK4SFJPTL7NAY5XXKBIFCVLNRSLHLACTJ65MWCHFSBV4EKJYA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998117076 (12h45m52.587462053s) [noise=true , address=104.36.82.56:4252 , piece_id=XQWAMPOYTVGKYEHIEEEIDDST5BOIN2XCT7B3SFUR4U7PQBROEELQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998596283 (12h45m52.58794127s) [noise=true , address=104.36.82.15:4261 , piece_id=C3RUC2F5LVRKH4MYTXDYOAP5KJGRZ6TAX57XULVBAKOS7ZK4QONQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998917628 (12h45m52.588262615s) [noise=true , address=104.36.82.23:4473 , piece_id=N54OVKSO4KKX4NPORULBLAPVE5UT5RU6BBIEJCCMKG5553GVY6JA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:54.121532197 (12h45m53.710877184s) [upload=13236]: Next returned nil segment Aug 7 22:25:54.121549150 (12h45m53.710894137s) [upload=13236]: Waiting for error group managing segments... Aug 7 22:25:54.447812563 (12h45m54.03715755s) [noise=true , address=79.127.205.233:4633 , piece_id=LBO5XLAXBRDURMDMN6MZ4HLVSG4TTUZPWCFTOTB6QMWAS5IIRLBA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:54.450658769 (12h45m54.040003756s) [noise=true , address=64.49.13.170:4717 , piece_id=52FN65VRQ24LBOGHJ22EQQM3MNI3EB2UXB57P5LD5GPLSABNS22Q, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:54.505775361 (12h45m54.095120349s) [noise=true , address=64.49.13.154:4934 , piece_id=LDQTK4SFJPTL7NAY5XXKBIFCVLNRSLHLACTJ65MWCHFSBV4EKJYA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.074032067 (12h45m54.663377054s) [noise=true , address=104.36.82.60:4346 , piece_id=3NU3QBO5DAW5ZB7NTAHWP4C7RE4BYDPHZ77JOUQT3R3CBTOMONLA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.160612428 (12h45m54.749957415s) [noise=true , address=104.36.82.61:4389 , piece_id=WBB2Y7YYCEML23D3CKJHGIEOUELRTVG3NXVGI4JDD7XPNQ4STENQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.161069252 (12h45m54.750414239s) [noise=true , address=104.36.82.22:4527 , piece_id=JBNJOSETDAB6VG7PGVXIBUP6GPF2FBPJURBZNBR3BJOGMRK3EIQA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.172460447 (12h45m54.761805434s) [noise=true , address=104.36.82.60:4361 , piece_id=EPWXHNXPVAUX75FUAXY57QZDKRXIEDHRXFL6PEBQGXY2A3Q4KNHA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.217919931 (12h45m54.807264908s) [noise=true , address=104.36.82.56:4252 , piece_id=XQWAMPOYTVGKYEHIEEEIDDST5BOIN2XCT7B3SFUR4U7PQBROEELQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.226637230 (12h45m54.815982217s) [noise=true , address=104.36.82.58:4281 , piece_id=5UH27M25HE2LEZJ53H7QBKU6I6N2JE6F4PZTLOE6XP3GEDCZLEEQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.271492068 (12h45m54.860837055s) [noise=true , address=45.91.116.11:4124 , piece_id=TLATR2WNDVRR3WWTT3FZARUJ2QWWQFZBDYXLZ7PUPFOAEGJI4BTQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.281497411 (12h45m54.870842388s) [noise=true , address=45.91.116.9:4284 , piece_id=LT3RP2EBVHIFIMAIGM3DW6QMPNLKZCK5FD4GQEQI6BFWYD56ZUBQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.402216807 (12h45m54.991561794s) [noise=true , address=104.36.82.9:4024 , piece_id=VBFOHRWDKWRY2774KJVK3JXJSMG4CPVDBBECCSZVJHR4274KLRMA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.632857570 (12h45m55.222202558s) [noise=true , address=104.36.82.5:4855 , piece_id=6EW7GNJ6KVWKNFHGFZVPEYPD4TDIVVZUXRW6AGJ3IGVANEAHYDUA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:56.838313689 (12h45m56.427658676s) [noise=true , address=104.36.82.13:4213 , piece_id=5W5FD4WMMSZ7BKHFQI4GYOODXZ2P2MMAIQNK6CD6XILH22C53JGQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:56.838338066 (12h45m56.427683053s) [seg_pos={0 0}, upload=13236]: Segment reached optimal threshold of 14 pieces. Aug 7 22:25:56.838585389 (12h45m56.427930366s) [noise=true , address=45.91.116.7:4037 , piece_id=2NDGWBVYO3D2RYJ57QSXT632TT24SMRNWA3YZ2YW2BN6TBBLMRQA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; write tcp 188.166.225.170:55570->45.91.116.7:4037: use of closed network connection; write tcp 188.166.225.170:55570->45.91.116.7:4037: use of closed network connection; EOF Aug 7 22:25:56.838630465 (12h45m56.427975452s) [noise=true , address=104.36.82.46:4505 , piece_id=A7XPGAYUTNB26A6DJQNQ4NZ4FG27FUBQNEH5AQ7IFLYR3DLGG6SQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf): protocol: write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection; write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection Aug 7 22:25:56.838682564 (12h45m56.428027551s) [noise=true , address=104.36.82.23:4473 , piece_id=N54OVKSO4KKX4NPORULBLAPVE5UT5RU6BBIEJCCMKG5553GVY6JA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC): protocol: write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection; write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection Aug 7 22:25:56.838697754 (12h45m56.428042731s) [noise=true , address=104.36.82.15:4261 , piece_id=C3RUC2F5LVRKH4MYTXDYOAP5KJGRZ6TAX57XULVBAKOS7ZK4QONQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i1jcJdEP7Ch9uaGcjDaVxpodx55dJEHUKFbcr1GeKQ9waWjYz): protocol: expected piece hash; context canceled; EOF Aug 7 22:25:56.838720558 (12h45m56.428065535s) [seg_pos={0 0}, upload=13236]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:25:56.838823714 (12h45m56.428168701s) [upload=13236]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:25:56.838828013 (12h45m56.428173s) [upload=13236]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:30:17.177125535 (12h50m16.766470522s) [upload=13236]: Done uploading segments... Aug 7 22:30:17.177153268 (12h50m16.766498255s) [upload=13236]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.178816705s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........X..............X.........X....X.................X] duration: 7.546197265s FAIL (2025-08-07 22:44:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 81af0bd8b34b2e69389f7db10076d194a3d6fbdd58cd01527d1b66bf7a00fe45 != before 7800c7ceda312caa58fa18fda98736593dbd42db38936a6dbdeb420eade030ff ======LOGS====== pload=13556]: Done starting upload Aug 7 22:44:47.819148413 (13h4m47.4084934s) [upload=13556]: Uploading segments... Aug 7 22:44:47.819394214 (13h4m47.408739201s) [upload=13556]: Got next segment. Inline: false Aug 7 22:44:47.819437876 (13h4m47.408782864s) [upload=13556]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:44:47.819442115 (13h4m47.408787092s) [upload=13556]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:44:48.269086796 (13h4m47.858431783s) [seg_pos={0 0}, upload=13556]: Begin upload segment... Aug 7 22:44:48.269139916 (13h4m47.858484903s) [seg_pos={0 0}, upload=13556]: Done begin upload segment. Aug 7 22:44:48.269169674 (13h4m47.858514661s) [noise=true , address=104.36.82.45:4938 , piece_id=NF4CEKTH5KGR2VLO64EISWJ64CRKFHKXBP3R77ZAZ3HVAFAYJJ4Q, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.269176958 (13h4m47.858521965s) [noise=true , address=104.36.82.60:4343 , piece_id=7ZZ3C77QFADRAQU3ZQF3LRXC5S6NF3XBNFXZ6XICWDCI3HFZ6EUQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.269619725 (13h4m47.858964712s) [noise=true , address=104.36.82.41:4768 , piece_id=GSW4YJR4SSW5ZKGPYI7L2KRRSHL3FTEFUE2RSARPDARD7CWOSXBA, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.269688626 (13h4m47.859033623s) [noise=true , address=104.36.82.9:4040 , piece_id=7YWI2TUXLX4QSI6GRFGIGLZRU3RUEUMJNJRVGAJON3NVRIZJ27DA, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.269932232 (13h4m47.85927722s) [noise=true , address=193.149.164.143:4696 , piece_id=D74PAAUDHRBQJSBQ7T5R6QDKUDUS4R7425SO7H53NHS4QZPTGBMQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.270291710 (13h4m47.859636717s) [noise=true , address=104.36.82.2:4742 , piece_id=TREABDVVCEXW3MN7U6TE5CWVGQXWW2Z5REAV7PEZWYRYNBK27ECQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.271203444 (13h4m47.860548432s) [noise=true , address=104.36.82.44:4893 , piece_id=IRD65MAG7YYBVEOPHERHYVPFA65YC7JQPJB6XKUQLHGKKMYTFLFQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.271407435 (13h4m47.860752422s) [noise=true , address=104.36.82.53:4122 , piece_id=DE5JKFJWUT4MIKUUQHWMXBTSK2MKDSTJ5VGJ5S2ALJXUTEJ3LUMA, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.271531252 (13h4m47.860876239s) [noise=true , address=104.36.82.59:4326 , piece_id=LJ5XKKU2EVRDGH5E4YAE5BRF5EIFM2BHG7BYMHOCQXIWUNS7DXRA, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.271734040 (13h4m47.861079027s) [noise=true , address=79.127.201.214:4295 , piece_id=6MJQFTPQRQL3OBVOWJ36WYWD3PVJBBCCMISXCJKC636VZPG474PA, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.271963008 (13h4m47.861307995s) [noise=true , address=104.36.82.8:4989 , piece_id=LPXKK2B443AVS624U7MPUP4GUEMUFJYVUBZY5ZJGZPV32WYW6T6A, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.272159905 (13h4m47.861504882s) [noise=true , address=104.36.82.43:4848 , piece_id=KQKHRNP6BMU7GRFQXGW7U5KSNF53NGMJIRFNNE5M3Q6MHGQJUC4Q, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.272343486 (13h4m47.861688463s) [noise=true , address=104.36.82.38:4715 , piece_id=BJDDYECUSWJH675QPKG4IP5PSTOS4Y77OMKQWG2BGZLAQE77GKHQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.272525674 (13h4m47.861870661s) [noise=true , address=104.36.82.15:4277 , piece_id=NCLDOZRBEK4OF7C4Q5DY4RX3ND73SMUM6QIRL5FHVOFBQYKOKIOQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.272744403 (13h4m47.86208939s) [noise=true , address=79.127.205.234:4597 , piece_id=CI2MXO5CGMRNZRAPXUCIGASHDL4KCMGAWVNMJESCLF3HTBRHWAGQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.272934386 (13h4m47.862279373s) [noise=true , address=104.36.82.46:4483 , piece_id=ZDGTDBBYXOWDCNUHAA7BLEGD4RAZOR6QLETRO4MG3MAGDE6RVLQQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.273120443 (13h4m47.86246543s) [noise=true , address=104.36.82.44:4901 , piece_id=UP7742ZXVVCLWBHJPYVKZH2C3W4M5NWE5QHEMSZGRWJM4Q72FIDA, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:48.273299345 (13h4m47.862644322s) [noise=true , address=104.36.82.22:4512 , piece_id=3BXMDY3JO7ZVVX6GF3LUH5RTR2GHVF5VJ2U5XEUF62VIUIXRVWSQ, seg_pos={0 0}, upload=13556]: Uploading piece... Aug 7 22:44:49.376334243 (13h4m48.96567923s) [upload=13556]: Next returned nil segment Aug 7 22:44:49.376348381 (13h4m48.965693368s) [upload=13556]: Waiting for error group managing segments... Aug 7 22:44:49.668631065 (13h4m49.257976052s) [noise=true , address=79.127.205.234:4597 , piece_id=CI2MXO5CGMRNZRAPXUCIGASHDL4KCMGAWVNMJESCLF3HTBRHWAGQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:49.675850678 (13h4m49.265195665s) [noise=true , address=193.149.164.143:4696 , piece_id=D74PAAUDHRBQJSBQ7T5R6QDKUDUS4R7425SO7H53NHS4QZPTGBMQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.149126209 (13h4m49.738471196s) [noise=true , address=79.127.201.214:4295 , piece_id=6MJQFTPQRQL3OBVOWJ36WYWD3PVJBBCCMISXCJKC636VZPG474PA, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.188374551 (13h4m49.777719528s) [noise=true , address=104.36.82.43:4848 , piece_id=KQKHRNP6BMU7GRFQXGW7U5KSNF53NGMJIRFNNE5M3Q6MHGQJUC4Q, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.192657957 (13h4m49.782002944s) [noise=true , address=104.36.82.22:4512 , piece_id=3BXMDY3JO7ZVVX6GF3LUH5RTR2GHVF5VJ2U5XEUF62VIUIXRVWSQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.260069984 (13h4m49.849414972s) [noise=true , address=104.36.82.38:4715 , piece_id=BJDDYECUSWJH675QPKG4IP5PSTOS4Y77OMKQWG2BGZLAQE77GKHQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.295276123 (13h4m49.88462111s) [noise=true , address=104.36.82.45:4938 , piece_id=NF4CEKTH5KGR2VLO64EISWJ64CRKFHKXBP3R77ZAZ3HVAFAYJJ4Q, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.314186263 (13h4m49.90353125s) [noise=true , address=104.36.82.59:4326 , piece_id=LJ5XKKU2EVRDGH5E4YAE5BRF5EIFM2BHG7BYMHOCQXIWUNS7DXRA, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.335401744 (13h4m49.924746731s) [noise=true , address=104.36.82.8:4989 , piece_id=LPXKK2B443AVS624U7MPUP4GUEMUFJYVUBZY5ZJGZPV32WYW6T6A, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.350133970 (13h4m49.939478957s) [noise=true , address=104.36.82.41:4768 , piece_id=GSW4YJR4SSW5ZKGPYI7L2KRRSHL3FTEFUE2RSARPDARD7CWOSXBA, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.370519655 (13h4m49.959864642s) [noise=true , address=104.36.82.53:4122 , piece_id=DE5JKFJWUT4MIKUUQHWMXBTSK2MKDSTJ5VGJ5S2ALJXUTEJ3LUMA, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.407688108 (13h4m49.997033095s) [noise=true , address=104.36.82.9:4040 , piece_id=7YWI2TUXLX4QSI6GRFGIGLZRU3RUEUMJNJRVGAJON3NVRIZJ27DA, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.484565775 (13h4m50.073910762s) [noise=true , address=104.36.82.15:4277 , piece_id=NCLDOZRBEK4OF7C4Q5DY4RX3ND73SMUM6QIRL5FHVOFBQYKOKIOQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.666687912 (13h4m50.256032899s) [noise=true , address=104.36.82.60:4343 , piece_id=7ZZ3C77QFADRAQU3ZQF3LRXC5S6NF3XBNFXZ6XICWDCI3HFZ6EUQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: <nil> Aug 7 22:44:50.666718511 (13h4m50.256063498s) [seg_pos={0 0}, upload=13556]: Segment reached optimal threshold of 14 pieces. Aug 7 22:44:50.667127924 (13h4m50.256472911s) [noise=true , address=104.36.82.2:4742 , piece_id=TREABDVVCEXW3MN7U6TE5CWVGQXWW2Z5REAV7PEZWYRYNBK27ECQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WsGo4te3PENFo9YRioYFrhkeRQvztnLVZJV4i38TV3TjKqBFT): protocol: expected piece hash; context canceled; EOF Aug 7 22:44:50.667260107 (13h4m50.256605094s) [noise=true , address=104.36.82.46:4483 , piece_id=ZDGTDBBYXOWDCNUHAA7BLEGD4RAZOR6QLETRO4MG3MAGDE6RVLQQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: write tcp 188.166.225.170:58454->104.36.82.46:4483: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:44:50.667317968 (13h4m50.256662945s) [noise=true , address=104.36.82.44:4893 , piece_id=IRD65MAG7YYBVEOPHERHYVPFA65YC7JQPJB6XKUQLHGKKMYTFLFQ, seg_pos={0 0}, upload=13556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ): protocol: rpc: dial tcp 104.36.82.44:4893: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:44:50.667373113 (13h4m50.256718091s) [noise=true , address=104.36.82.44:4901 , piece_id=UP7742ZXVVCLWBHJPYVKZH2C3W4M5NWE5QHEMSZGRWJM4Q72FIDA, seg_pos={0 0}, upload=13556]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: rpc: dial tcp 104.36.82.44:4901: operation was canceled; conn closed: op Aug 7 22:44:50.667382952 (13h4m50.256727939s) [seg_pos={0 0}, upload=13556]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:44:50.667435813 (13h4m50.2567808s) [upload=13556]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:44:50.667439520 (13h4m50.256784507s) [upload=13556]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:44:50.982639800 (13h4m50.571984787s) [upload=13556]: Done uploading segments... Aug 7 22:44:50.982656853 (13h4m50.572001841s) [upload=13556]: Upload finished. err: <nil> Duration: 7.546227532s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.....................XXXXXXXXXXXXXXXXX......................] duration: 6.776646193s FAIL (2025-08-07 22:44:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :32.805219872 (12h43m32.394564859s) [upload=13200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:32.805223589 (12h43m32.394568576s) [upload=13200]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:24:02.160684947 (12h44m1.750029944s) [upload=13200]: Next segment err: context deadline exceeded Aug 7 22:24:02.160927551 (12h44m1.750272538s) [upload=13200]: Done uploading segments... Aug 7 22:24:02.160942019 (12h44m1.750287007s) [upload=13200]: Upload finished. err: context deadline exceeded Aug 7 22:24:12.500861314 (12h44m12.090206301s) [upload=13211]: Starting upload Aug 7 22:24:12.500912582 (12h44m12.090257569s) [upload=13211]: Done starting upload Aug 7 22:24:12.500973068 (12h44m12.090318045s) [upload=13211]: Uploading segments... Aug 7 22:24:12.501078069 (12h44m12.090423056s) [upload=13211]: Got next segment. Inline: false Aug 7 22:24:12.501106834 (12h44m12.090451821s) [upload=13211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:12.501110571 (12h44m12.090455558s) [upload=13211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:24:25.371267525 (12h44m24.960612512s) [seg_pos={0 0}, upload=13211]: Begin upload segment... Aug 7 22:24:25.371330936 (12h44m24.960675953s) [seg_pos={0 0}, upload=13211]: Done begin upload segment. Aug 7 22:24:25.371383426 (12h44m24.960728413s) [noise=true , address=77.90.31.14:4661 , piece_id=73RH6546ZSA46KNTPMI3YPHM65Y3DJG5PCAAPGYJJFXDQ6KM5MRA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.371396932 (12h44m24.96074194s) [noise=true , address=77.90.31.19:4841 , piece_id=GPGSM3FHS76WAHMKYCXKIU57NEENBDZFAMIBSFP6JWV5K33BDCOQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.371863044 (12h44m24.961208021s) [noise=true , address=45.91.116.9:4261 , piece_id=F3JBOLQO76A5WQKYNQAKPAWVWZO5U5EYX7ZZIKQMOAUZUQCRZ7GA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372024263 (12h44m24.96136925s) [noise=true , address=45.91.116.12:4225 , piece_id=47OO6QUMLHWH54IAXRLTQEN56Z5ITGXCGKSTQFGSHTF32X3Z7HSQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372219627 (12h44m24.961564614s) [noise=true , address=104.36.82.2:4729 , piece_id=JYG3SHRCFAE37242ADYQA2CXJRRGO5APTOBZ5725GMZ2FRMPYLKA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372406654 (12h44m24.961751641s) [noise=true , address=45.91.116.7:4009 , piece_id=N25P6VDHR6RAW44ZNMAPNYJUX4AUV2UI3ROEX67KTWLK7RRB4K7Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372852217 (12h44m24.962197204s) [noise=true , address=45.91.116.6:4973 , piece_id=WLZJLHSER36K377KZCHHCLWNANIVKGHVKF2I3FBC5UDDLSIUCHFQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373233576 (12h44m24.962578563s) [noise=true , address=45.91.116.4:4937 , piece_id=ALO2UUBMXABZGWARBYTKGYY5DHLJEGULYGIZ7E4JQCYNFGP4JG7Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373581912 (12h44m24.9629269s) [noise=true , address=45.91.116.5:4299 , piece_id=LMJE5NXY2LTDFI3BZY2YGMAXFURMI5NOSZWW2ABZK4NFKQK5JROQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373901074 (12h44m24.963246061s) [noise=true , address=104.36.82.11:4113 , piece_id=7UYIPJZWWCEIMHEOPGC3DKXXTULAHTNF5QMPJJYIHXTKPSDOMM5A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374042614 (12h44m24.963387601s) [noise=true , address=77.90.31.17:4772 , piece_id=UEGCYD3SF4UF3CHRT5GEAZ6B77VKMG7S7PMSSRLZT6IP4ZKOTHMA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374483207 (12h44m24.963828194s) [noise=true , address=104.36.82.7:4912 , piece_id=H6DSVHM5ZX2GWEPHGRDH2WEDRHCLEYLZ62ISEHBLEITT4NGNDT7A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374810343 (12h44m24.96415533s) [noise=true , address=45.91.116.4:4939 , piece_id=S4PTGHW7FTLW5Q7W2VQZHLAI4YQZ25U35J7LUYSS5EXK552GUE2A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375117380 (12h44m24.964462368s) [noise=true , address=77.90.31.11:4555 , piece_id=CTQZTDFEGF756WLYI6U6NGMGBLFOKXZX3QWTQE7KBAITW43WVIJA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375526834 (12h44m24.964871821s) [noise=true , address=45.91.116.10:4082 , piece_id=4QZEPLP5I47FAOHIYQWJEBODVUEWTM25IQ652LDWLFZERCPTEXHA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375955474 (12h44m24.965300461s) [noise=true , address=104.36.82.10:4078 , piece_id=QKGPABRCDU26EDHDNRMCHMUQ3XTOQSN47WG5RKF6S4LD7OG2HY6Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:27.072833335 (12h44m26.662178332s) [upload=13211]: Next returned nil segment Aug 7 22:24:27.072902037 (12h44m26.662247034s) [upload=13211]: Waiting for error group managing segments... Aug 7 22:24:27.436037535 (12h44m27.025382522s) [noise=true , address=104.36.82.7:4912 , piece_id=H6DSVHM5ZX2GWEPHGRDH2WEDRHCLEYLZ62ISEHBLEITT4NGNDT7A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.446916249 (12h44m27.036261236s) [noise=true , address=104.36.82.2:4729 , piece_id=JYG3SHRCFAE37242ADYQA2CXJRRGO5APTOBZ5725GMZ2FRMPYLKA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.566730713 (12h44m27.15607569s) [noise=true , address=45.91.116.9:4261 , piece_id=F3JBOLQO76A5WQKYNQAKPAWVWZO5U5EYX7ZZIKQMOAUZUQCRZ7GA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.594802733 (12h44m27.18414772s) [noise=true , address=45.91.116.4:4937 , piece_id=ALO2UUBMXABZGWARBYTKGYY5DHLJEGULYGIZ7E4JQCYNFGP4JG7Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.613347363 (12h44m27.20269235s) [noise=true , address=77.90.31.14:4661 , piece_id=73RH6546ZSA46KNTPMI3YPHM65Y3DJG5PCAAPGYJJFXDQ6KM5MRA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.623084653 (12h44m27.21242964s) [noise=true , address=45.91.116.10:4082 , piece_id=4QZEPLP5I47FAOHIYQWJEBODVUEWTM25IQ652LDWLFZERCPTEXHA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.661186031 (12h44m27.250531018s) [noise=true , address=77.90.31.19:4841 , piece_id=GPGSM3FHS76WAHMKYCXKIU57NEENBDZFAMIBSFP6JWV5K33BDCOQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.698198064 (12h44m27.287543051s) [noise=true , address=45.91.116.6:4973 , piece_id=WLZJLHSER36K377KZCHHCLWNANIVKGHVKF2I3FBC5UDDLSIUCHFQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.724830297 (12h44m27.314175285s) [noise=true , address=45.91.116.4:4939 , piece_id=S4PTGHW7FTLW5Q7W2VQZHLAI4YQZ25U35J7LUYSS5EXK552GUE2A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.734317339 (12h44m27.323662326s) [noise=true , address=45.91.116.5:4299 , piece_id=LMJE5NXY2LTDFI3BZY2YGMAXFURMI5NOSZWW2ABZK4NFKQK5JROQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.908078026 (12h44m27.497423013s) [noise=true , address=77.90.31.17:4772 , piece_id=UEGCYD3SF4UF3CHRT5GEAZ6B77VKMG7S7PMSSRLZT6IP4ZKOTHMA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.985050925 (12h44m27.574395912s) [noise=true , address=45.91.116.7:4009 , piece_id=N25P6VDHR6RAW44ZNMAPNYJUX4AUV2UI3ROEX67KTWLK7RRB4K7Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:28.048748935 (12h44m27.638093922s) [noise=true , address=77.90.31.11:4555 , piece_id=CTQZTDFEGF756WLYI6U6NGMGBLFOKXZX3QWTQE7KBAITW43WVIJA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:28.048775115 (12h44m27.638120102s) [seg_pos={0 0}, upload=13211]: Segment reached optimal threshold of 13 pieces. Aug 7 22:24:28.048900375 (12h44m27.638245362s) [noise=true , address=45.91.116.12:4225 , piece_id=47OO6QUMLHWH54IAXRLTQEN56Z5ITGXCGKSTQFGSHTF32X3Z7HSQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Aug 7 22:24:28.049006828 (12h44m27.638351815s) [noise=true , address=104.36.82.11:4113 , piece_id=7UYIPJZWWCEIMHEOPGC3DKXXTULAHTNF5QMPJJYIHXTKPSDOMM5A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection; write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection Aug 7 22:24:28.049050902 (12h44m27.638395889s) [noise=true , address=104.36.82.10:4078 , piece_id=QKGPABRCDU26EDHDNRMCHMUQ3XTOQSN47WG5RKF6S4LD7OG2HY6Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection; write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection Aug 7 22:24:28.049061182 (12h44m27.638406169s) [seg_pos={0 0}, upload=13211]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 22:24:28.049126247 (12h44m27.638471224s) [upload=13211]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:24:28.049130154 (12h44m27.638475141s) [upload=13211]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:23.601003390 (12h47m23.190348377s) [upload=13211]: Done uploading segments... Aug 7 22:27:23.601027807 (12h47m23.190372794s) [upload=13211]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.776667202s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 238.285142ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 22:44:44 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 238.30965ms edge gateway vhost bgp us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 24 out of 60 is failed (threshold=3) [...............XXXXXXXXXXXXXXXXXXX.XXX.X.X..................] duration: 2.492551252s FAIL (2025-08-07 22:44:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.49257104s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [...................XXXXXXXXXXXXXXXXXXX...X..................] duration: 6.359831335s FAIL (2025-08-07 22:44:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.359859439s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 17 out of 60 is failed (threshold=3) [..................XXXXXXXXXXXXXXXX.X........................] duration: 1.911280488s FAIL (2025-08-07 22:44:38 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== starting upload Aug 7 22:19:13.035284043 (12h39m12.62462903s) [upload=13134]: Uploading segments... Aug 7 22:19:13.035347064 (12h39m12.624692051s) [upload=13134]: Got next segment. Inline: true Aug 7 22:19:13.035351633 (12h39m12.62469662s) [upload=13134]: Waiting for error group managing segments... Aug 7 22:19:13.035393122 (12h39m12.624738099s) [upload=13134]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:19:13.035397750 (12h39m12.624742739s) [upload=13134]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:19:13.035400156 (12h39m12.624745143s) [upload=13134]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:20:12.392096704 (12h40m11.981441692s) [upload=13134]: Done uploading segments... Aug 7 22:20:12.392135208 (12h40m11.981480225s) [upload=13134]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:20:13.302960454 (12h40m12.892305441s) [upload=13150]: Starting upload Aug 7 22:20:13.303052761 (12h40m12.892397748s) [upload=13150]: Done starting upload Aug 7 22:20:13.303103337 (12h40m12.892448325s) [upload=13150]: Uploading segments... Aug 7 22:20:13.303136621 (12h40m12.892481608s) [upload=13150]: Got next segment. Inline: true Aug 7 22:20:13.303140047 (12h40m12.892485034s) [upload=13150]: Waiting for error group managing segments... Aug 7 22:20:13.303175805 (12h40m12.892520793s) [upload=13150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:13.303181377 (12h40m12.892526364s) [upload=13150]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:20:13.303183922 (12h40m12.892528909s) [upload=13150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:21:12.655202481 (12h41m12.244547468s) [upload=13150]: Done uploading segments... Aug 7 22:21:12.655262385 (12h41m12.244607372s) [upload=13150]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:21:28.179922232 (12h41m27.769267219s) [upload=13167]: Starting upload Aug 7 22:21:28.180014519 (12h41m27.769359506s) [upload=13167]: Done starting upload Aug 7 22:21:28.180090304 (12h41m27.769435291s) [upload=13167]: Uploading segments... Aug 7 22:21:28.180123167 (12h41m27.769468154s) [upload=13167]: Got next segment. Inline: true Aug 7 22:21:28.180126524 (12h41m27.769471511s) [upload=13167]: Waiting for error group managing segments... Aug 7 22:21:28.180185667 (12h41m27.769530654s) [upload=13167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:21:28.180189734 (12h41m27.769534721s) [upload=13167]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:21:28.180192199 (12h41m27.769537186s) [upload=13167]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:27.535085641 (12h42m27.124430628s) [upload=13167]: Done uploading segments... Aug 7 22:22:27.535157829 (12h42m27.124502816s) [upload=13167]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:22:28.402559244 (12h42m27.991904221s) [upload=13183]: Starting upload Aug 7 22:22:28.402660068 (12h42m27.992005055s) [upload=13183]: Done starting upload Aug 7 22:22:28.402728308 (12h42m27.992073335s) [upload=13183]: Uploading segments... Aug 7 22:22:28.402775298 (12h42m27.992120335s) [upload=13183]: Got next segment. Inline: true Aug 7 22:22:28.402781640 (12h42m27.992126627s) [upload=13183]: Waiting for error group managing segments... Aug 7 22:22:28.402833088 (12h42m27.992178146s) [upload=13183]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:28.402840001 (12h42m27.992184988s) [upload=13183]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:28.402845021 (12h42m27.992190008s) [upload=13183]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:27.752834209 (12h43m27.342179246s) [upload=13183]: Done uploading segments... Aug 7 22:23:27.752884696 (12h43m27.342229693s) [upload=13183]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:28.630476036 (12h43m28.219821014s) [upload=13198]: Starting upload Aug 7 22:23:28.630528016 (12h43m28.219873003s) [upload=13198]: Done starting upload Aug 7 22:23:28.630599864 (12h43m28.219944851s) [upload=13198]: Uploading segments... Aug 7 22:23:28.630637285 (12h43m28.219982272s) [upload=13198]: Got next segment. Inline: true Aug 7 22:23:28.630640952 (12h43m28.219985939s) [upload=13198]: Waiting for error group managing segments... Aug 7 22:23:28.630673745 (12h43m28.220018732s) [upload=13198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:28.630677352 (12h43m28.220022339s) [upload=13198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:28.630679597 (12h43m28.220024584s) [upload=13198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:27.987023038 (12h44m27.576368025s) [upload=13198]: Done uploading segments... Aug 7 22:24:27.987035291 (12h44m27.576380268s) [upload=13198]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:28.848154918 (12h44m28.437499895s) [upload=13216]: Starting upload Aug 7 22:24:28.848218670 (12h44m28.437563657s) [upload=13216]: Done starting upload Aug 7 22:24:28.848264347 (12h44m28.437609324s) [upload=13216]: Uploading segments... Aug 7 22:24:28.848331396 (12h44m28.437676373s) [upload=13216]: Got next segment. Inline: true Aug 7 22:24:28.848335494 (12h44m28.437680481s) [upload=13216]: Waiting for error group managing segments... Aug 7 22:24:28.848369038 (12h44m28.437714025s) [upload=13216]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:28.848373045 (12h44m28.437718032s) [upload=13216]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:28.848375189 (12h44m28.437720176s) [upload=13216]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:28.201395574 (12h45m27.790740561s) [upload=13216]: Done uploading segments... Aug 7 22:25:28.201413839 (12h45m27.790758816s) [upload=13216]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:34.254413826 (12h45m33.843758814s) [upload=13234]: Starting upload Aug 7 22:25:34.254496073 (12h45m33.84384106s) [upload=13234]: Done starting upload Aug 7 22:25:34.254567009 (12h45m33.843911996s) [upload=13234]: Uploading segments... Aug 7 22:25:34.254604511 (12h45m33.843949508s) [upload=13234]: Got next segment. Inline: true Aug 7 22:25:34.254609231 (12h45m33.843954218s) [upload=13234]: Waiting for error group managing segments... Aug 7 22:25:34.254647023 (12h45m33.84399201s) [upload=13234]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:34.254651561 (12h45m33.843996558s) [upload=13234]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:34.254654356 (12h45m33.843999343s) [upload=13234]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:33.597318767 (12h46m33.186663754s) [upload=13234]: Done uploading segments... Aug 7 22:26:33.597351351 (12h46m33.186696338s) [upload=13234]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:26:34.495602477 (12h46m34.084947454s) [upload=13247]: Starting upload Aug 7 22:26:34.495652543 (12h46m34.08499753s) [upload=13247]: Done starting upload Aug 7 22:26:34.495718250 (12h46m34.085063237s) [upload=13247]: Uploading segments... Aug 7 22:26:34.495750731 (12h46m34.085095719s) [upload=13247]: Got next segment. Inline: true Aug 7 22:26:34.495754388 (12h46m34.085099375s) [upload=13247]: Waiting for error group managing segments... Aug 7 22:26:34.495787031 (12h46m34.085132018s) [upload=13247]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:34.495791108 (12h46m34.085136095s) [upload=13247]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:34.495793463 (12h46m34.08513844s) [upload=13247]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:33.854971690 (12h47m33.444316667s) [upload=13247]: Done uploading segments... Aug 7 22:27:33.855000124 (12h47m33.444345111s) [upload=13247]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:47.578421960 (12h47m47.167766947s) [upload=13264]: Starting upload Aug 7 22:27:47.578474811 (12h47m47.167819798s) [upload=13264]: Done starting upload Aug 7 22:27:47.578529386 (12h47m47.167874363s) [upload=13264]: Uploading segments... Aug 7 22:27:47.578558632 (12h47m47.167903619s) [upload=13264]: Got next segment. Inline: true Aug 7 22:27:47.578561968 (12h47m47.167906945s) [upload=13264]: Waiting for error group managing segments... Aug 7 22:27:47.578595061 (12h47m47.167940048s) [upload=13264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:47.578599049 (12h47m47.167944036s) [upload=13264]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:47.578601603 (12h47m47.16794659s) [upload=13264]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:47.846848491 (12h47m47.436193478s) [upload=13264]: Done uploading segments... Aug 7 22:27:47.846867718 (12h47m47.436212695s) [upload=13264]: Upload finished. err: <nil> Aug 7 22:28:34.302216720 (12h48m33.891561707s) [upload=13274]: Starting upload Aug 7 22:28:34.302270392 (12h48m33.891615379s) [upload=13274]: Done starting upload Aug 7 22:28:34.302346498 (12h48m33.891691485s) [upload=13274]: Uploading segments... Aug 7 22:28:34.302375654 (12h48m33.891720641s) [upload=13274]: Got next segment. Inline: true Aug 7 22:28:34.302379070 (12h48m33.891724057s) [upload=13274]: Waiting for error group managing segments... Aug 7 22:28:34.302409759 (12h48m33.891754746s) [upload=13274]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:34.302413246 (12h48m33.891758234s) [upload=13274]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:34.302415511 (12h48m33.891760488s) [upload=13274]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:29.866705738 (12h49m29.456050725s) [upload=13274]: Done uploading segments... Aug 7 22:29:29.866723503 (12h49m29.45606849s) [upload=13274]: Upload finished. err: <nil> Duration: 1.91154355s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 6 out of 60 is failed (threshold=3) [XX..X..............................XXX......................] duration: 5.733117506s FAIL (2025-08-07 22:44:36 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.733146269s core uplink us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 651.387748ms target: https://us1.storj.io/login FAIL (2025-08-07 22:44:35 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 651.413607ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 19 out of 60 is failed (threshold=3) [...............XXXXXXXXXXXXXXXX.XX..X.......................] duration: 1.858160423s FAIL (2025-08-07 22:44:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7 22:22:45.818723517 (12h42m45.408068504s) [upload=13188]: Done starting upload Aug 7 22:22:45.818774856 (12h42m45.408119843s) [upload=13188]: Uploading segments... Aug 7 22:22:45.818807999 (12h42m45.408152996s) [upload=13188]: Got next segment. Inline: true Aug 7 22:22:45.818812908 (12h42m45.408157895s) [upload=13188]: Waiting for error group managing segments... Aug 7 22:22:45.818870779 (12h42m45.408215776s) [upload=13188]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:45.818879214 (12h42m45.408224202s) [upload=13188]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:45.818883494 (12h42m45.408228481s) [upload=13188]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:15.175828623 (12h43m14.76517361s) [upload=13188]: Done uploading segments... Aug 7 22:23:15.175844935 (12h43m14.765189912s) [upload=13188]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:16.350710694 (12h43m15.940055691s) [upload=13195]: Starting upload Aug 7 22:23:16.350768756 (12h43m15.940113733s) [upload=13195]: Done starting upload Aug 7 22:23:16.350814804 (12h43m15.940159791s) [upload=13195]: Uploading segments... Aug 7 22:23:16.350849079 (12h43m15.940194056s) [upload=13195]: Got next segment. Inline: true Aug 7 22:23:16.350852777 (12h43m15.940197764s) [upload=13195]: Waiting for error group managing segments... Aug 7 22:23:16.350884928 (12h43m15.940229915s) [upload=13195]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:16.350888735 (12h43m15.940233722s) [upload=13195]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:16.350891020 (12h43m15.940236007s) [upload=13195]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:45.695642688 (12h43m45.284987675s) [upload=13195]: Done uploading segments... Aug 7 22:23:45.695657867 (12h43m45.285002854s) [upload=13195]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:08.254775832 (12h44m7.844120819s) [upload=13210]: Starting upload Aug 7 22:24:08.254857087 (12h44m7.844202074s) [upload=13210]: Done starting upload Aug 7 22:24:08.254936559 (12h44m7.844281546s) [upload=13210]: Uploading segments... Aug 7 22:24:08.255031861 (12h44m7.844376858s) [upload=13210]: Got next segment. Inline: true Aug 7 22:24:08.255037032 (12h44m7.844382019s) [upload=13210]: Waiting for error group managing segments... Aug 7 22:24:08.255078220 (12h44m7.844423207s) [upload=13210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:08.255093450 (12h44m7.844438437s) [upload=13210]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:08.255096435 (12h44m7.844441422s) [upload=13210]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:37.608974587 (12h44m37.198319574s) [upload=13210]: Done uploading segments... Aug 7 22:24:37.609189158 (12h44m37.198534195s) [upload=13210]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:19.078840222 (12h45m18.668185229s) [upload=13227]: Starting upload Aug 7 22:25:19.078945333 (12h45m18.66829035s) [upload=13227]: Done starting upload Aug 7 22:25:19.079016760 (12h45m18.668361767s) [upload=13227]: Uploading segments... Aug 7 22:25:19.079058389 (12h45m18.668403377s) [upload=13227]: Got next segment. Inline: true Aug 7 22:25:19.079064030 (12h45m18.668409007s) [upload=13227]: Waiting for error group managing segments... Aug 7 22:25:19.079115388 (12h45m18.668460375s) [upload=13227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:19.079121570 (12h45m18.668466557s) [upload=13227]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:19.079126009 (12h45m18.668471077s) [upload=13227]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:48.426015265 (12h45m48.015360252s) [upload=13227]: Done uploading segments... Aug 7 22:25:48.426036394 (12h45m48.015381381s) [upload=13227]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:26:18.229360345 (12h46m17.818705332s) [upload=13243]: Starting upload Aug 7 22:26:18.229421673 (12h46m17.81876665s) [upload=13243]: Done starting upload Aug 7 22:26:18.229465397 (12h46m17.818810384s) [upload=13243]: Uploading segments... Aug 7 22:26:18.229495053 (12h46m17.81884004s) [upload=13243]: Got next segment. Inline: true Aug 7 22:26:18.229498680 (12h46m17.818843667s) [upload=13243]: Waiting for error group managing segments... Aug 7 22:26:18.229531593 (12h46m17.81887658s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:18.229535170 (12h46m17.818880157s) [upload=13243]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:18.229537614 (12h46m17.818882601s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:18.493923010 (12h46m18.083268007s) [upload=13243]: Done uploading segments... Aug 7 22:26:18.493948349 (12h46m18.083293346s) [upload=13243]: Upload finished. err: <nil> Aug 7 22:26:34.238583280 (12h46m33.827928267s) [upload=13246]: Starting upload Aug 7 22:26:34.238671890 (12h46m33.828016897s) [upload=13246]: Done starting upload Aug 7 22:26:34.238753585 (12h46m33.828098573s) [upload=13246]: Uploading segments... Aug 7 22:26:34.238795296 (12h46m33.828140283s) [upload=13246]: Got next segment. Inline: true Aug 7 22:26:34.238800546 (12h46m33.828145533s) [upload=13246]: Waiting for error group managing segments... Aug 7 22:26:34.238842697 (12h46m33.828187684s) [upload=13246]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:34.238847656 (12h46m33.828192633s) [upload=13246]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:34.238851193 (12h46m33.82819618s) [upload=13246]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:03.590020724 (12h47m3.179365711s) [upload=13246]: Done uploading segments... Aug 7 22:27:03.590046854 (12h47m3.179391861s) [upload=13246]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:08.455168905 (12h47m8.044513892s) [upload=13253]: Starting upload Aug 7 22:27:08.455224000 (12h47m8.044568977s) [upload=13253]: Done starting upload Aug 7 22:27:08.455266011 (12h47m8.044610998s) [upload=13253]: Uploading segments... Aug 7 22:27:08.455292903 (12h47m8.04463789s) [upload=13253]: Got next segment. Inline: true Aug 7 22:27:08.455296630 (12h47m8.044641607s) [upload=13253]: Waiting for error group managing segments... Aug 7 22:27:08.455327689 (12h47m8.044672676s) [upload=13253]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:08.455331626 (12h47m8.044676613s) [upload=13253]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:08.455334372 (12h47m8.044679349s) [upload=13253]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:37.813944606 (12h47m37.403289643s) [upload=13253]: Done uploading segments... Aug 7 22:27:37.813971608 (12h47m37.403316615s) [upload=13253]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:28:03.718747816 (12h48m3.308092803s) [upload=13266]: Starting upload Aug 7 22:28:03.718803663 (12h48m3.30814865s) [upload=13266]: Done starting upload Aug 7 22:28:03.718851525 (12h48m3.308196512s) [upload=13266]: Uploading segments... Aug 7 22:28:03.718879849 (12h48m3.308224836s) [upload=13266]: Got next segment. Inline: true Aug 7 22:28:03.718883686 (12h48m3.308228673s) [upload=13266]: Waiting for error group managing segments... Aug 7 22:28:03.718918023 (12h48m3.30826301s) [upload=13266]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:03.718922461 (12h48m3.308267448s) [upload=13266]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:03.718924966 (12h48m3.308269953s) [upload=13266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:04.001556505 (12h48m3.590901502s) [upload=13266]: Done uploading segments... Aug 7 22:28:04.001574169 (12h48m3.590919156s) [upload=13266]: Upload finished. err: <nil> Aug 7 22:28:53.653874975 (12h48m53.243219952s) [upload=13280]: Starting upload Aug 7 22:28:53.653923989 (12h48m53.243268976s) [upload=13280]: Done starting upload Aug 7 22:28:53.653970338 (12h48m53.243315315s) [upload=13280]: Uploading segments... Aug 7 22:28:53.654001658 (12h48m53.243346645s) [upload=13280]: Got next segment. Inline: true Aug 7 22:28:53.654005195 (12h48m53.243350182s) [upload=13280]: Waiting for error group managing segments... Aug 7 22:28:53.654037516 (12h48m53.243382503s) [upload=13280]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:53.654041253 (12h48m53.24338623s) [upload=13280]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:53.654043598 (12h48m53.243388575s) [upload=13280]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:53.931677747 (12h48m53.521022724s) [upload=13280]: Done uploading segments... Aug 7 22:28:53.931693927 (12h48m53.521038914s) [upload=13280]: Upload finished. err: <nil> Aug 7 22:29:13.069018260 (12h49m12.658363247s) [upload=13284]: Starting upload Aug 7 22:29:13.069075108 (12h49m12.658420096s) [upload=13284]: Done starting upload Aug 7 22:29:13.069124593 (12h49m12.65846958s) [upload=13284]: Uploading segments... Aug 7 22:29:13.069159800 (12h49m12.658504787s) [upload=13284]: Got next segment. Inline: true Aug 7 22:29:13.069163377 (12h49m12.658508364s) [upload=13284]: Waiting for error group managing segments... Aug 7 22:29:13.069195007 (12h49m12.658539995s) [upload=13284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:29:13.069198645 (12h49m12.658543632s) [upload=13284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:29:13.069201010 (12h49m12.658545997s) [upload=13284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:42.426836268 (12h49m42.016181256s) [upload=13284]: Done uploading segments... Aug 7 22:29:42.426899691 (12h49m42.016244678s) [upload=13284]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.858176615s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 11 out of 60 is failed (threshold=3) [.............................XXXXXXXXX..X.X.................] duration: 6.435083384s FAIL (2025-08-07 22:44:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=13236]: Begin upload segment... Aug 7 22:25:52.992307080 (12h45m52.581652067s) [seg_pos={0 0}, upload=13236]: Done begin upload segment. Aug 7 22:25:52.992331798 (12h45m52.581676775s) [noise=true , address=104.36.82.13:4213 , piece_id=5W5FD4WMMSZ7BKHFQI4GYOODXZ2P2MMAIQNK6CD6XILH22C53JGQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992683770 (12h45m52.582028757s) [noise=true , address=104.36.82.9:4024 , piece_id=VBFOHRWDKWRY2774KJVK3JXJSMG4CPVDBBECCSZVJHR4274KLRMA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992785475 (12h45m52.582130462s) [noise=true , address=104.36.82.22:4527 , piece_id=JBNJOSETDAB6VG7PGVXIBUP6GPF2FBPJURBZNBR3BJOGMRK3EIQA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992942396 (12h45m52.582287373s) [noise=true , address=45.91.116.11:4124 , piece_id=TLATR2WNDVRR3WWTT3FZARUJ2QWWQFZBDYXLZ7PUPFOAEGJI4BTQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993132439 (12h45m52.582477457s) [noise=true , address=104.36.82.5:4855 , piece_id=6EW7GNJ6KVWKNFHGFZVPEYPD4TDIVVZUXRW6AGJ3IGVANEAHYDUA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993337862 (12h45m52.582682849s) [noise=true , address=64.49.13.170:4717 , piece_id=52FN65VRQ24LBOGHJ22EQQM3MNI3EB2UXB57P5LD5GPLSABNS22Q, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993511144 (12h45m52.582856131s) [noise=true , address=104.36.82.60:4361 , piece_id=EPWXHNXPVAUX75FUAXY57QZDKRXIEDHRXFL6PEBQGXY2A3Q4KNHA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993715204 (12h45m52.583060191s) [noise=true , address=104.36.82.58:4281 , piece_id=5UH27M25HE2LEZJ53H7QBKU6I6N2JE6F4PZTLOE6XP3GEDCZLEEQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993820806 (12h45m52.583165784s) [noise=true , address=45.91.116.9:4284 , piece_id=LT3RP2EBVHIFIMAIGM3DW6QMPNLKZCK5FD4GQEQI6BFWYD56ZUBQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994068250 (12h45m52.583413237s) [noise=true , address=79.127.205.233:4633 , piece_id=LBO5XLAXBRDURMDMN6MZ4HLVSG4TTUZPWCFTOTB6QMWAS5IIRLBA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994092416 (12h45m52.583437393s) [noise=true , address=104.36.82.60:4346 , piece_id=3NU3QBO5DAW5ZB7NTAHWP4C7RE4BYDPHZ77JOUQT3R3CBTOMONLA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994295394 (12h45m52.583640371s) [noise=true , address=45.91.116.7:4037 , piece_id=2NDGWBVYO3D2RYJ57QSXT632TT24SMRNWA3YZ2YW2BN6TBBLMRQA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994307056 (12h45m52.583652033s) [noise=true , address=104.36.82.46:4505 , piece_id=A7XPGAYUTNB26A6DJQNQ4NZ4FG27FUBQNEH5AQ7IFLYR3DLGG6SQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994547126 (12h45m52.583892103s) [noise=true , address=104.36.82.61:4389 , piece_id=WBB2Y7YYCEML23D3CKJHGIEOUELRTVG3NXVGI4JDD7XPNQ4STENQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994782436 (12h45m52.584127423s) [noise=true , address=64.49.13.154:4934 , piece_id=LDQTK4SFJPTL7NAY5XXKBIFCVLNRSLHLACTJ65MWCHFSBV4EKJYA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998117076 (12h45m52.587462053s) [noise=true , address=104.36.82.56:4252 , piece_id=XQWAMPOYTVGKYEHIEEEIDDST5BOIN2XCT7B3SFUR4U7PQBROEELQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998596283 (12h45m52.58794127s) [noise=true , address=104.36.82.15:4261 , piece_id=C3RUC2F5LVRKH4MYTXDYOAP5KJGRZ6TAX57XULVBAKOS7ZK4QONQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998917628 (12h45m52.588262615s) [noise=true , address=104.36.82.23:4473 , piece_id=N54OVKSO4KKX4NPORULBLAPVE5UT5RU6BBIEJCCMKG5553GVY6JA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:54.121532197 (12h45m53.710877184s) [upload=13236]: Next returned nil segment Aug 7 22:25:54.121549150 (12h45m53.710894137s) [upload=13236]: Waiting for error group managing segments... Aug 7 22:25:54.447812563 (12h45m54.03715755s) [noise=true , address=79.127.205.233:4633 , piece_id=LBO5XLAXBRDURMDMN6MZ4HLVSG4TTUZPWCFTOTB6QMWAS5IIRLBA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:54.450658769 (12h45m54.040003756s) [noise=true , address=64.49.13.170:4717 , piece_id=52FN65VRQ24LBOGHJ22EQQM3MNI3EB2UXB57P5LD5GPLSABNS22Q, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:54.505775361 (12h45m54.095120349s) [noise=true , address=64.49.13.154:4934 , piece_id=LDQTK4SFJPTL7NAY5XXKBIFCVLNRSLHLACTJ65MWCHFSBV4EKJYA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.074032067 (12h45m54.663377054s) [noise=true , address=104.36.82.60:4346 , piece_id=3NU3QBO5DAW5ZB7NTAHWP4C7RE4BYDPHZ77JOUQT3R3CBTOMONLA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.160612428 (12h45m54.749957415s) [noise=true , address=104.36.82.61:4389 , piece_id=WBB2Y7YYCEML23D3CKJHGIEOUELRTVG3NXVGI4JDD7XPNQ4STENQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.161069252 (12h45m54.750414239s) [noise=true , address=104.36.82.22:4527 , piece_id=JBNJOSETDAB6VG7PGVXIBUP6GPF2FBPJURBZNBR3BJOGMRK3EIQA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.172460447 (12h45m54.761805434s) [noise=true , address=104.36.82.60:4361 , piece_id=EPWXHNXPVAUX75FUAXY57QZDKRXIEDHRXFL6PEBQGXY2A3Q4KNHA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.217919931 (12h45m54.807264908s) [noise=true , address=104.36.82.56:4252 , piece_id=XQWAMPOYTVGKYEHIEEEIDDST5BOIN2XCT7B3SFUR4U7PQBROEELQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.226637230 (12h45m54.815982217s) [noise=true , address=104.36.82.58:4281 , piece_id=5UH27M25HE2LEZJ53H7QBKU6I6N2JE6F4PZTLOE6XP3GEDCZLEEQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.271492068 (12h45m54.860837055s) [noise=true , address=45.91.116.11:4124 , piece_id=TLATR2WNDVRR3WWTT3FZARUJ2QWWQFZBDYXLZ7PUPFOAEGJI4BTQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.281497411 (12h45m54.870842388s) [noise=true , address=45.91.116.9:4284 , piece_id=LT3RP2EBVHIFIMAIGM3DW6QMPNLKZCK5FD4GQEQI6BFWYD56ZUBQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.402216807 (12h45m54.991561794s) [noise=true , address=104.36.82.9:4024 , piece_id=VBFOHRWDKWRY2774KJVK3JXJSMG4CPVDBBECCSZVJHR4274KLRMA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.632857570 (12h45m55.222202558s) [noise=true , address=104.36.82.5:4855 , piece_id=6EW7GNJ6KVWKNFHGFZVPEYPD4TDIVVZUXRW6AGJ3IGVANEAHYDUA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:56.838313689 (12h45m56.427658676s) [noise=true , address=104.36.82.13:4213 , piece_id=5W5FD4WMMSZ7BKHFQI4GYOODXZ2P2MMAIQNK6CD6XILH22C53JGQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:56.838338066 (12h45m56.427683053s) [seg_pos={0 0}, upload=13236]: Segment reached optimal threshold of 14 pieces. Aug 7 22:25:56.838585389 (12h45m56.427930366s) [noise=true , address=45.91.116.7:4037 , piece_id=2NDGWBVYO3D2RYJ57QSXT632TT24SMRNWA3YZ2YW2BN6TBBLMRQA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; write tcp 188.166.225.170:55570->45.91.116.7:4037: use of closed network connection; write tcp 188.166.225.170:55570->45.91.116.7:4037: use of closed network connection; EOF Aug 7 22:25:56.838630465 (12h45m56.427975452s) [noise=true , address=104.36.82.46:4505 , piece_id=A7XPGAYUTNB26A6DJQNQ4NZ4FG27FUBQNEH5AQ7IFLYR3DLGG6SQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf): protocol: write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection; write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection Aug 7 22:25:56.838682564 (12h45m56.428027551s) [noise=true , address=104.36.82.23:4473 , piece_id=N54OVKSO4KKX4NPORULBLAPVE5UT5RU6BBIEJCCMKG5553GVY6JA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC): protocol: write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection; write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection Aug 7 22:25:56.838697754 (12h45m56.428042731s) [noise=true , address=104.36.82.15:4261 , piece_id=C3RUC2F5LVRKH4MYTXDYOAP5KJGRZ6TAX57XULVBAKOS7ZK4QONQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i1jcJdEP7Ch9uaGcjDaVxpodx55dJEHUKFbcr1GeKQ9waWjYz): protocol: expected piece hash; context canceled; EOF Aug 7 22:25:56.838720558 (12h45m56.428065535s) [seg_pos={0 0}, upload=13236]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:25:56.838823714 (12h45m56.428168701s) [upload=13236]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:25:56.838828013 (12h45m56.428173s) [upload=13236]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:30:17.177125535 (12h50m16.766470522s) [upload=13236]: Done uploading segments... Aug 7 22:30:17.177153268 (12h50m16.766498255s) [upload=13236]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.435109596s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [......................XXXXXXXXXXXXXXXXX.....................] duration: 6.568607286s FAIL (2025-08-07 22:44:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :32.805219872 (12h43m32.394564859s) [upload=13200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:32.805223589 (12h43m32.394568576s) [upload=13200]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:24:02.160684947 (12h44m1.750029944s) [upload=13200]: Next segment err: context deadline exceeded Aug 7 22:24:02.160927551 (12h44m1.750272538s) [upload=13200]: Done uploading segments... Aug 7 22:24:02.160942019 (12h44m1.750287007s) [upload=13200]: Upload finished. err: context deadline exceeded Aug 7 22:24:12.500861314 (12h44m12.090206301s) [upload=13211]: Starting upload Aug 7 22:24:12.500912582 (12h44m12.090257569s) [upload=13211]: Done starting upload Aug 7 22:24:12.500973068 (12h44m12.090318045s) [upload=13211]: Uploading segments... Aug 7 22:24:12.501078069 (12h44m12.090423056s) [upload=13211]: Got next segment. Inline: false Aug 7 22:24:12.501106834 (12h44m12.090451821s) [upload=13211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:12.501110571 (12h44m12.090455558s) [upload=13211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:24:25.371267525 (12h44m24.960612512s) [seg_pos={0 0}, upload=13211]: Begin upload segment... Aug 7 22:24:25.371330936 (12h44m24.960675953s) [seg_pos={0 0}, upload=13211]: Done begin upload segment. Aug 7 22:24:25.371383426 (12h44m24.960728413s) [noise=true , address=77.90.31.14:4661 , piece_id=73RH6546ZSA46KNTPMI3YPHM65Y3DJG5PCAAPGYJJFXDQ6KM5MRA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.371396932 (12h44m24.96074194s) [noise=true , address=77.90.31.19:4841 , piece_id=GPGSM3FHS76WAHMKYCXKIU57NEENBDZFAMIBSFP6JWV5K33BDCOQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.371863044 (12h44m24.961208021s) [noise=true , address=45.91.116.9:4261 , piece_id=F3JBOLQO76A5WQKYNQAKPAWVWZO5U5EYX7ZZIKQMOAUZUQCRZ7GA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372024263 (12h44m24.96136925s) [noise=true , address=45.91.116.12:4225 , piece_id=47OO6QUMLHWH54IAXRLTQEN56Z5ITGXCGKSTQFGSHTF32X3Z7HSQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372219627 (12h44m24.961564614s) [noise=true , address=104.36.82.2:4729 , piece_id=JYG3SHRCFAE37242ADYQA2CXJRRGO5APTOBZ5725GMZ2FRMPYLKA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372406654 (12h44m24.961751641s) [noise=true , address=45.91.116.7:4009 , piece_id=N25P6VDHR6RAW44ZNMAPNYJUX4AUV2UI3ROEX67KTWLK7RRB4K7Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372852217 (12h44m24.962197204s) [noise=true , address=45.91.116.6:4973 , piece_id=WLZJLHSER36K377KZCHHCLWNANIVKGHVKF2I3FBC5UDDLSIUCHFQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373233576 (12h44m24.962578563s) [noise=true , address=45.91.116.4:4937 , piece_id=ALO2UUBMXABZGWARBYTKGYY5DHLJEGULYGIZ7E4JQCYNFGP4JG7Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373581912 (12h44m24.9629269s) [noise=true , address=45.91.116.5:4299 , piece_id=LMJE5NXY2LTDFI3BZY2YGMAXFURMI5NOSZWW2ABZK4NFKQK5JROQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373901074 (12h44m24.963246061s) [noise=true , address=104.36.82.11:4113 , piece_id=7UYIPJZWWCEIMHEOPGC3DKXXTULAHTNF5QMPJJYIHXTKPSDOMM5A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374042614 (12h44m24.963387601s) [noise=true , address=77.90.31.17:4772 , piece_id=UEGCYD3SF4UF3CHRT5GEAZ6B77VKMG7S7PMSSRLZT6IP4ZKOTHMA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374483207 (12h44m24.963828194s) [noise=true , address=104.36.82.7:4912 , piece_id=H6DSVHM5ZX2GWEPHGRDH2WEDRHCLEYLZ62ISEHBLEITT4NGNDT7A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374810343 (12h44m24.96415533s) [noise=true , address=45.91.116.4:4939 , piece_id=S4PTGHW7FTLW5Q7W2VQZHLAI4YQZ25U35J7LUYSS5EXK552GUE2A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375117380 (12h44m24.964462368s) [noise=true , address=77.90.31.11:4555 , piece_id=CTQZTDFEGF756WLYI6U6NGMGBLFOKXZX3QWTQE7KBAITW43WVIJA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375526834 (12h44m24.964871821s) [noise=true , address=45.91.116.10:4082 , piece_id=4QZEPLP5I47FAOHIYQWJEBODVUEWTM25IQ652LDWLFZERCPTEXHA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375955474 (12h44m24.965300461s) [noise=true , address=104.36.82.10:4078 , piece_id=QKGPABRCDU26EDHDNRMCHMUQ3XTOQSN47WG5RKF6S4LD7OG2HY6Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:27.072833335 (12h44m26.662178332s) [upload=13211]: Next returned nil segment Aug 7 22:24:27.072902037 (12h44m26.662247034s) [upload=13211]: Waiting for error group managing segments... Aug 7 22:24:27.436037535 (12h44m27.025382522s) [noise=true , address=104.36.82.7:4912 , piece_id=H6DSVHM5ZX2GWEPHGRDH2WEDRHCLEYLZ62ISEHBLEITT4NGNDT7A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.446916249 (12h44m27.036261236s) [noise=true , address=104.36.82.2:4729 , piece_id=JYG3SHRCFAE37242ADYQA2CXJRRGO5APTOBZ5725GMZ2FRMPYLKA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.566730713 (12h44m27.15607569s) [noise=true , address=45.91.116.9:4261 , piece_id=F3JBOLQO76A5WQKYNQAKPAWVWZO5U5EYX7ZZIKQMOAUZUQCRZ7GA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.594802733 (12h44m27.18414772s) [noise=true , address=45.91.116.4:4937 , piece_id=ALO2UUBMXABZGWARBYTKGYY5DHLJEGULYGIZ7E4JQCYNFGP4JG7Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.613347363 (12h44m27.20269235s) [noise=true , address=77.90.31.14:4661 , piece_id=73RH6546ZSA46KNTPMI3YPHM65Y3DJG5PCAAPGYJJFXDQ6KM5MRA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.623084653 (12h44m27.21242964s) [noise=true , address=45.91.116.10:4082 , piece_id=4QZEPLP5I47FAOHIYQWJEBODVUEWTM25IQ652LDWLFZERCPTEXHA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.661186031 (12h44m27.250531018s) [noise=true , address=77.90.31.19:4841 , piece_id=GPGSM3FHS76WAHMKYCXKIU57NEENBDZFAMIBSFP6JWV5K33BDCOQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.698198064 (12h44m27.287543051s) [noise=true , address=45.91.116.6:4973 , piece_id=WLZJLHSER36K377KZCHHCLWNANIVKGHVKF2I3FBC5UDDLSIUCHFQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.724830297 (12h44m27.314175285s) [noise=true , address=45.91.116.4:4939 , piece_id=S4PTGHW7FTLW5Q7W2VQZHLAI4YQZ25U35J7LUYSS5EXK552GUE2A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.734317339 (12h44m27.323662326s) [noise=true , address=45.91.116.5:4299 , piece_id=LMJE5NXY2LTDFI3BZY2YGMAXFURMI5NOSZWW2ABZK4NFKQK5JROQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.908078026 (12h44m27.497423013s) [noise=true , address=77.90.31.17:4772 , piece_id=UEGCYD3SF4UF3CHRT5GEAZ6B77VKMG7S7PMSSRLZT6IP4ZKOTHMA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.985050925 (12h44m27.574395912s) [noise=true , address=45.91.116.7:4009 , piece_id=N25P6VDHR6RAW44ZNMAPNYJUX4AUV2UI3ROEX67KTWLK7RRB4K7Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:28.048748935 (12h44m27.638093922s) [noise=true , address=77.90.31.11:4555 , piece_id=CTQZTDFEGF756WLYI6U6NGMGBLFOKXZX3QWTQE7KBAITW43WVIJA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:28.048775115 (12h44m27.638120102s) [seg_pos={0 0}, upload=13211]: Segment reached optimal threshold of 13 pieces. Aug 7 22:24:28.048900375 (12h44m27.638245362s) [noise=true , address=45.91.116.12:4225 , piece_id=47OO6QUMLHWH54IAXRLTQEN56Z5ITGXCGKSTQFGSHTF32X3Z7HSQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Aug 7 22:24:28.049006828 (12h44m27.638351815s) [noise=true , address=104.36.82.11:4113 , piece_id=7UYIPJZWWCEIMHEOPGC3DKXXTULAHTNF5QMPJJYIHXTKPSDOMM5A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection; write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection Aug 7 22:24:28.049050902 (12h44m27.638395889s) [noise=true , address=104.36.82.10:4078 , piece_id=QKGPABRCDU26EDHDNRMCHMUQ3XTOQSN47WG5RKF6S4LD7OG2HY6Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection; write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection Aug 7 22:24:28.049061182 (12h44m27.638406169s) [seg_pos={0 0}, upload=13211]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 22:24:28.049126247 (12h44m27.638471224s) [upload=13211]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:24:28.049130154 (12h44m27.638475141s) [upload=13211]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:23.601003390 (12h47m23.190348377s) [upload=13211]: Done uploading segments... Aug 7 22:27:23.601027807 (12h47m23.190372794s) [upload=13211]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.568646933s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [.....................XXXXXXXXXXXXXXXXX.X....................] duration: 5.659010649s FAIL (2025-08-07 22:44:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.659034303s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 17 out of 60 is failed (threshold=3) [...................XXXXXXXXXXXXXXXX.X.......................] duration: 1.944595646s FAIL (2025-08-07 22:44:15 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== starting upload Aug 7 22:19:13.035284043 (12h39m12.62462903s) [upload=13134]: Uploading segments... Aug 7 22:19:13.035347064 (12h39m12.624692051s) [upload=13134]: Got next segment. Inline: true Aug 7 22:19:13.035351633 (12h39m12.62469662s) [upload=13134]: Waiting for error group managing segments... Aug 7 22:19:13.035393122 (12h39m12.624738099s) [upload=13134]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:19:13.035397750 (12h39m12.624742739s) [upload=13134]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:19:13.035400156 (12h39m12.624745143s) [upload=13134]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:20:12.392096704 (12h40m11.981441692s) [upload=13134]: Done uploading segments... Aug 7 22:20:12.392135208 (12h40m11.981480225s) [upload=13134]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:20:13.302960454 (12h40m12.892305441s) [upload=13150]: Starting upload Aug 7 22:20:13.303052761 (12h40m12.892397748s) [upload=13150]: Done starting upload Aug 7 22:20:13.303103337 (12h40m12.892448325s) [upload=13150]: Uploading segments... Aug 7 22:20:13.303136621 (12h40m12.892481608s) [upload=13150]: Got next segment. Inline: true Aug 7 22:20:13.303140047 (12h40m12.892485034s) [upload=13150]: Waiting for error group managing segments... Aug 7 22:20:13.303175805 (12h40m12.892520793s) [upload=13150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:13.303181377 (12h40m12.892526364s) [upload=13150]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:20:13.303183922 (12h40m12.892528909s) [upload=13150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:21:12.655202481 (12h41m12.244547468s) [upload=13150]: Done uploading segments... Aug 7 22:21:12.655262385 (12h41m12.244607372s) [upload=13150]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:21:28.179922232 (12h41m27.769267219s) [upload=13167]: Starting upload Aug 7 22:21:28.180014519 (12h41m27.769359506s) [upload=13167]: Done starting upload Aug 7 22:21:28.180090304 (12h41m27.769435291s) [upload=13167]: Uploading segments... Aug 7 22:21:28.180123167 (12h41m27.769468154s) [upload=13167]: Got next segment. Inline: true Aug 7 22:21:28.180126524 (12h41m27.769471511s) [upload=13167]: Waiting for error group managing segments... Aug 7 22:21:28.180185667 (12h41m27.769530654s) [upload=13167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:21:28.180189734 (12h41m27.769534721s) [upload=13167]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:21:28.180192199 (12h41m27.769537186s) [upload=13167]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:27.535085641 (12h42m27.124430628s) [upload=13167]: Done uploading segments... Aug 7 22:22:27.535157829 (12h42m27.124502816s) [upload=13167]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:22:28.402559244 (12h42m27.991904221s) [upload=13183]: Starting upload Aug 7 22:22:28.402660068 (12h42m27.992005055s) [upload=13183]: Done starting upload Aug 7 22:22:28.402728308 (12h42m27.992073335s) [upload=13183]: Uploading segments... Aug 7 22:22:28.402775298 (12h42m27.992120335s) [upload=13183]: Got next segment. Inline: true Aug 7 22:22:28.402781640 (12h42m27.992126627s) [upload=13183]: Waiting for error group managing segments... Aug 7 22:22:28.402833088 (12h42m27.992178146s) [upload=13183]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:28.402840001 (12h42m27.992184988s) [upload=13183]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:28.402845021 (12h42m27.992190008s) [upload=13183]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:27.752834209 (12h43m27.342179246s) [upload=13183]: Done uploading segments... Aug 7 22:23:27.752884696 (12h43m27.342229693s) [upload=13183]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:28.630476036 (12h43m28.219821014s) [upload=13198]: Starting upload Aug 7 22:23:28.630528016 (12h43m28.219873003s) [upload=13198]: Done starting upload Aug 7 22:23:28.630599864 (12h43m28.219944851s) [upload=13198]: Uploading segments... Aug 7 22:23:28.630637285 (12h43m28.219982272s) [upload=13198]: Got next segment. Inline: true Aug 7 22:23:28.630640952 (12h43m28.219985939s) [upload=13198]: Waiting for error group managing segments... Aug 7 22:23:28.630673745 (12h43m28.220018732s) [upload=13198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:28.630677352 (12h43m28.220022339s) [upload=13198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:28.630679597 (12h43m28.220024584s) [upload=13198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:27.987023038 (12h44m27.576368025s) [upload=13198]: Done uploading segments... Aug 7 22:24:27.987035291 (12h44m27.576380268s) [upload=13198]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:28.848154918 (12h44m28.437499895s) [upload=13216]: Starting upload Aug 7 22:24:28.848218670 (12h44m28.437563657s) [upload=13216]: Done starting upload Aug 7 22:24:28.848264347 (12h44m28.437609324s) [upload=13216]: Uploading segments... Aug 7 22:24:28.848331396 (12h44m28.437676373s) [upload=13216]: Got next segment. Inline: true Aug 7 22:24:28.848335494 (12h44m28.437680481s) [upload=13216]: Waiting for error group managing segments... Aug 7 22:24:28.848369038 (12h44m28.437714025s) [upload=13216]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:28.848373045 (12h44m28.437718032s) [upload=13216]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:28.848375189 (12h44m28.437720176s) [upload=13216]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:28.201395574 (12h45m27.790740561s) [upload=13216]: Done uploading segments... Aug 7 22:25:28.201413839 (12h45m27.790758816s) [upload=13216]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:34.254413826 (12h45m33.843758814s) [upload=13234]: Starting upload Aug 7 22:25:34.254496073 (12h45m33.84384106s) [upload=13234]: Done starting upload Aug 7 22:25:34.254567009 (12h45m33.843911996s) [upload=13234]: Uploading segments... Aug 7 22:25:34.254604511 (12h45m33.843949508s) [upload=13234]: Got next segment. Inline: true Aug 7 22:25:34.254609231 (12h45m33.843954218s) [upload=13234]: Waiting for error group managing segments... Aug 7 22:25:34.254647023 (12h45m33.84399201s) [upload=13234]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:34.254651561 (12h45m33.843996558s) [upload=13234]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:34.254654356 (12h45m33.843999343s) [upload=13234]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:33.597318767 (12h46m33.186663754s) [upload=13234]: Done uploading segments... Aug 7 22:26:33.597351351 (12h46m33.186696338s) [upload=13234]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:26:34.495602477 (12h46m34.084947454s) [upload=13247]: Starting upload Aug 7 22:26:34.495652543 (12h46m34.08499753s) [upload=13247]: Done starting upload Aug 7 22:26:34.495718250 (12h46m34.085063237s) [upload=13247]: Uploading segments... Aug 7 22:26:34.495750731 (12h46m34.085095719s) [upload=13247]: Got next segment. Inline: true Aug 7 22:26:34.495754388 (12h46m34.085099375s) [upload=13247]: Waiting for error group managing segments... Aug 7 22:26:34.495787031 (12h46m34.085132018s) [upload=13247]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:34.495791108 (12h46m34.085136095s) [upload=13247]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:34.495793463 (12h46m34.08513844s) [upload=13247]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:33.854971690 (12h47m33.444316667s) [upload=13247]: Done uploading segments... Aug 7 22:27:33.855000124 (12h47m33.444345111s) [upload=13247]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:47.578421960 (12h47m47.167766947s) [upload=13264]: Starting upload Aug 7 22:27:47.578474811 (12h47m47.167819798s) [upload=13264]: Done starting upload Aug 7 22:27:47.578529386 (12h47m47.167874363s) [upload=13264]: Uploading segments... Aug 7 22:27:47.578558632 (12h47m47.167903619s) [upload=13264]: Got next segment. Inline: true Aug 7 22:27:47.578561968 (12h47m47.167906945s) [upload=13264]: Waiting for error group managing segments... Aug 7 22:27:47.578595061 (12h47m47.167940048s) [upload=13264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:47.578599049 (12h47m47.167944036s) [upload=13264]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:47.578601603 (12h47m47.16794659s) [upload=13264]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:47.846848491 (12h47m47.436193478s) [upload=13264]: Done uploading segments... Aug 7 22:27:47.846867718 (12h47m47.436212695s) [upload=13264]: Upload finished. err: <nil> Aug 7 22:28:34.302216720 (12h48m33.891561707s) [upload=13274]: Starting upload Aug 7 22:28:34.302270392 (12h48m33.891615379s) [upload=13274]: Done starting upload Aug 7 22:28:34.302346498 (12h48m33.891691485s) [upload=13274]: Uploading segments... Aug 7 22:28:34.302375654 (12h48m33.891720641s) [upload=13274]: Got next segment. Inline: true Aug 7 22:28:34.302379070 (12h48m33.891724057s) [upload=13274]: Waiting for error group managing segments... Aug 7 22:28:34.302409759 (12h48m33.891754746s) [upload=13274]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:34.302413246 (12h48m33.891758234s) [upload=13274]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:34.302415511 (12h48m33.891760488s) [upload=13274]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:29.866705738 (12h49m29.456050725s) [upload=13274]: Done uploading segments... Aug 7 22:29:29.866723503 (12h49m29.45606849s) [upload=13274]: Upload finished. err: <nil> Duration: 1.944621446s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 645.650482ms target: https://us1.storj.io/login FAIL (2025-08-07 22:44:11 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 645.706849ms core satellite us1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 236.82492ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 22:44:10 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 236.847642ms edge gateway vhost bgp us1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [..........................X..X..............................] duration: 696.346354ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 22:44:02 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 696.373966ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.XX..X..............................XXX.....................] duration: 5.893839068s FAIL (2025-08-07 22:43:59 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.893865819s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 21 out of 60 is failed (threshold=3) [................XXXXXXXXXXXXXXXXXXX.X..X....................] duration: 1.87441825s FAIL (2025-08-07 22:43:56 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.874442085s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 17 out of 60 is failed (threshold=3) [....................XXXXXXXXXXXXXXXX.X......................] duration: 1.880414302s FAIL (2025-08-07 22:43:53 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== starting upload Aug 7 22:19:13.035284043 (12h39m12.62462903s) [upload=13134]: Uploading segments... Aug 7 22:19:13.035347064 (12h39m12.624692051s) [upload=13134]: Got next segment. Inline: true Aug 7 22:19:13.035351633 (12h39m12.62469662s) [upload=13134]: Waiting for error group managing segments... Aug 7 22:19:13.035393122 (12h39m12.624738099s) [upload=13134]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:19:13.035397750 (12h39m12.624742739s) [upload=13134]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:19:13.035400156 (12h39m12.624745143s) [upload=13134]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:20:12.392096704 (12h40m11.981441692s) [upload=13134]: Done uploading segments... Aug 7 22:20:12.392135208 (12h40m11.981480225s) [upload=13134]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:20:13.302960454 (12h40m12.892305441s) [upload=13150]: Starting upload Aug 7 22:20:13.303052761 (12h40m12.892397748s) [upload=13150]: Done starting upload Aug 7 22:20:13.303103337 (12h40m12.892448325s) [upload=13150]: Uploading segments... Aug 7 22:20:13.303136621 (12h40m12.892481608s) [upload=13150]: Got next segment. Inline: true Aug 7 22:20:13.303140047 (12h40m12.892485034s) [upload=13150]: Waiting for error group managing segments... Aug 7 22:20:13.303175805 (12h40m12.892520793s) [upload=13150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:13.303181377 (12h40m12.892526364s) [upload=13150]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:20:13.303183922 (12h40m12.892528909s) [upload=13150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:21:12.655202481 (12h41m12.244547468s) [upload=13150]: Done uploading segments... Aug 7 22:21:12.655262385 (12h41m12.244607372s) [upload=13150]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:21:28.179922232 (12h41m27.769267219s) [upload=13167]: Starting upload Aug 7 22:21:28.180014519 (12h41m27.769359506s) [upload=13167]: Done starting upload Aug 7 22:21:28.180090304 (12h41m27.769435291s) [upload=13167]: Uploading segments... Aug 7 22:21:28.180123167 (12h41m27.769468154s) [upload=13167]: Got next segment. Inline: true Aug 7 22:21:28.180126524 (12h41m27.769471511s) [upload=13167]: Waiting for error group managing segments... Aug 7 22:21:28.180185667 (12h41m27.769530654s) [upload=13167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:21:28.180189734 (12h41m27.769534721s) [upload=13167]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:21:28.180192199 (12h41m27.769537186s) [upload=13167]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:27.535085641 (12h42m27.124430628s) [upload=13167]: Done uploading segments... Aug 7 22:22:27.535157829 (12h42m27.124502816s) [upload=13167]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:22:28.402559244 (12h42m27.991904221s) [upload=13183]: Starting upload Aug 7 22:22:28.402660068 (12h42m27.992005055s) [upload=13183]: Done starting upload Aug 7 22:22:28.402728308 (12h42m27.992073335s) [upload=13183]: Uploading segments... Aug 7 22:22:28.402775298 (12h42m27.992120335s) [upload=13183]: Got next segment. Inline: true Aug 7 22:22:28.402781640 (12h42m27.992126627s) [upload=13183]: Waiting for error group managing segments... Aug 7 22:22:28.402833088 (12h42m27.992178146s) [upload=13183]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:28.402840001 (12h42m27.992184988s) [upload=13183]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:28.402845021 (12h42m27.992190008s) [upload=13183]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:27.752834209 (12h43m27.342179246s) [upload=13183]: Done uploading segments... Aug 7 22:23:27.752884696 (12h43m27.342229693s) [upload=13183]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:28.630476036 (12h43m28.219821014s) [upload=13198]: Starting upload Aug 7 22:23:28.630528016 (12h43m28.219873003s) [upload=13198]: Done starting upload Aug 7 22:23:28.630599864 (12h43m28.219944851s) [upload=13198]: Uploading segments... Aug 7 22:23:28.630637285 (12h43m28.219982272s) [upload=13198]: Got next segment. Inline: true Aug 7 22:23:28.630640952 (12h43m28.219985939s) [upload=13198]: Waiting for error group managing segments... Aug 7 22:23:28.630673745 (12h43m28.220018732s) [upload=13198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:28.630677352 (12h43m28.220022339s) [upload=13198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:28.630679597 (12h43m28.220024584s) [upload=13198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:27.987023038 (12h44m27.576368025s) [upload=13198]: Done uploading segments... Aug 7 22:24:27.987035291 (12h44m27.576380268s) [upload=13198]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:28.848154918 (12h44m28.437499895s) [upload=13216]: Starting upload Aug 7 22:24:28.848218670 (12h44m28.437563657s) [upload=13216]: Done starting upload Aug 7 22:24:28.848264347 (12h44m28.437609324s) [upload=13216]: Uploading segments... Aug 7 22:24:28.848331396 (12h44m28.437676373s) [upload=13216]: Got next segment. Inline: true Aug 7 22:24:28.848335494 (12h44m28.437680481s) [upload=13216]: Waiting for error group managing segments... Aug 7 22:24:28.848369038 (12h44m28.437714025s) [upload=13216]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:28.848373045 (12h44m28.437718032s) [upload=13216]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:28.848375189 (12h44m28.437720176s) [upload=13216]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:28.201395574 (12h45m27.790740561s) [upload=13216]: Done uploading segments... Aug 7 22:25:28.201413839 (12h45m27.790758816s) [upload=13216]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:34.254413826 (12h45m33.843758814s) [upload=13234]: Starting upload Aug 7 22:25:34.254496073 (12h45m33.84384106s) [upload=13234]: Done starting upload Aug 7 22:25:34.254567009 (12h45m33.843911996s) [upload=13234]: Uploading segments... Aug 7 22:25:34.254604511 (12h45m33.843949508s) [upload=13234]: Got next segment. Inline: true Aug 7 22:25:34.254609231 (12h45m33.843954218s) [upload=13234]: Waiting for error group managing segments... Aug 7 22:25:34.254647023 (12h45m33.84399201s) [upload=13234]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:34.254651561 (12h45m33.843996558s) [upload=13234]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:34.254654356 (12h45m33.843999343s) [upload=13234]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:33.597318767 (12h46m33.186663754s) [upload=13234]: Done uploading segments... Aug 7 22:26:33.597351351 (12h46m33.186696338s) [upload=13234]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:26:34.495602477 (12h46m34.084947454s) [upload=13247]: Starting upload Aug 7 22:26:34.495652543 (12h46m34.08499753s) [upload=13247]: Done starting upload Aug 7 22:26:34.495718250 (12h46m34.085063237s) [upload=13247]: Uploading segments... Aug 7 22:26:34.495750731 (12h46m34.085095719s) [upload=13247]: Got next segment. Inline: true Aug 7 22:26:34.495754388 (12h46m34.085099375s) [upload=13247]: Waiting for error group managing segments... Aug 7 22:26:34.495787031 (12h46m34.085132018s) [upload=13247]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:34.495791108 (12h46m34.085136095s) [upload=13247]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:34.495793463 (12h46m34.08513844s) [upload=13247]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:33.854971690 (12h47m33.444316667s) [upload=13247]: Done uploading segments... Aug 7 22:27:33.855000124 (12h47m33.444345111s) [upload=13247]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:47.578421960 (12h47m47.167766947s) [upload=13264]: Starting upload Aug 7 22:27:47.578474811 (12h47m47.167819798s) [upload=13264]: Done starting upload Aug 7 22:27:47.578529386 (12h47m47.167874363s) [upload=13264]: Uploading segments... Aug 7 22:27:47.578558632 (12h47m47.167903619s) [upload=13264]: Got next segment. Inline: true Aug 7 22:27:47.578561968 (12h47m47.167906945s) [upload=13264]: Waiting for error group managing segments... Aug 7 22:27:47.578595061 (12h47m47.167940048s) [upload=13264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:47.578599049 (12h47m47.167944036s) [upload=13264]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:47.578601603 (12h47m47.16794659s) [upload=13264]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:47.846848491 (12h47m47.436193478s) [upload=13264]: Done uploading segments... Aug 7 22:27:47.846867718 (12h47m47.436212695s) [upload=13264]: Upload finished. err: <nil> Aug 7 22:28:34.302216720 (12h48m33.891561707s) [upload=13274]: Starting upload Aug 7 22:28:34.302270392 (12h48m33.891615379s) [upload=13274]: Done starting upload Aug 7 22:28:34.302346498 (12h48m33.891691485s) [upload=13274]: Uploading segments... Aug 7 22:28:34.302375654 (12h48m33.891720641s) [upload=13274]: Got next segment. Inline: true Aug 7 22:28:34.302379070 (12h48m33.891724057s) [upload=13274]: Waiting for error group managing segments... Aug 7 22:28:34.302409759 (12h48m33.891754746s) [upload=13274]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:34.302413246 (12h48m33.891758234s) [upload=13274]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:34.302415511 (12h48m33.891760488s) [upload=13274]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:29.866705738 (12h49m29.456050725s) [upload=13274]: Done uploading segments... Aug 7 22:29:29.866723503 (12h49m29.45606849s) [upload=13274]: Upload finished. err: <nil> Duration: 1.880439351s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 656.740372ms target: https://us1.storj.io/login FAIL (2025-08-07 22:43:49 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 656.78102ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [............X..............X.........X....X.................] duration: 8.699484905s FAIL (2025-08-07 22:43:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== ding piece... Aug 7 22:30:27.106182415 (12h50m26.695527402s) [noise=true , address=64.49.13.137:4494 , piece_id=LFDJNPET6CVU2GRXM2KJ3LGNJPTGJ5EM4PWPHF3XVYMPHS366DZA, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.106447803 (12h50m26.69579278s) [noise=true , address=104.36.82.61:4404 , piece_id=ZLDF3FSQTOGUD4OMZYNQZZ6MUTVUF6NICTZBHRPBUM5OR3SDYVIA, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.106779869 (12h50m26.696124856s) [noise=true , address=104.36.82.52:4114 , piece_id=C5IZX5N3O4JEE2RDMVYZGSOWXVKQSGBJD67NOUNFT4LHMRNZL37A, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.107212025 (12h50m26.696557012s) [noise=true , address=104.36.82.9:4033 , piece_id=WDDQWPVVTQKSC4JUHUXZ27G6GJXGGBCXBBSQMVEX7ONSCLEFFTAA, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.108229842 (12h50m26.697574829s) [noise=true , address=104.36.82.37:4684 , piece_id=3VZGRTMMP3FQSDOTGNTVLJ43IHMA5FSUBOZ4P2SKWCYG5COVF7LQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.108480002 (12h50m26.697824989s) [noise=true , address=104.36.82.44:4884 , piece_id=HRBXMCCRYU4FBS73Q2IB5BK5BUUHAALRNODQ5XERZICLQ6PUVOBQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.109071673 (12h50m26.69841666s) [noise=true , address=45.91.116.13:4160 , piece_id=UGSRSWK3RVAIVHVFWBHMUGHTZLZNCJN3DSSJTBF3EP3DY4UHMG5A, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.109303346 (12h50m26.698648333s) [noise=true , address=104.36.82.46:4504 , piece_id=W7DK423UD6SXLWL3LVYWLGTCFNUDATLWKEMQIPYXX4WZ555GD34A, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.109532515 (12h50m26.698877492s) [noise=true , address=79.127.201.215:4356 , piece_id=FUQGELP6V2FJZ6PVPVKMYKWEAY25P7OAYA5ORTS26FL4FMGAF7XQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.109694265 (12h50m26.699039252s) [noise=true , address=79.127.205.235:4659 , piece_id=PDPXLV3GK4XVZTDBS2GC564GUEVII3XLOI54ALEUL2YWMBDFUNNQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.109942329 (12h50m26.699287316s) [noise=true , address=79.127.205.234:4588 , piece_id=SCZQTO3IY6647Q6MRG3J2AMSWJAWIYKPTX2S2YQHIENXEFZ3C53A, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.110167320 (12h50m26.699512307s) [noise=true , address=104.36.82.34:4576 , piece_id=G3A3X2KHUXIYXEF4FODXHPISXNSWNTLTP5RIXXPIWGMDWWEYBGUQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.110377551 (12h50m26.699722538s) [noise=true , address=104.36.82.16:4325 , piece_id=M4UMVR3RXMLSL3BI55AHRHPL46AAO6MU7V3V3GBZ7HQCR7WRZWSQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.110578186 (12h50m26.699923173s) [noise=true , address=104.36.82.35:4583 , piece_id=L4R2PO7KNIIOIN7BDCYLRCO2MRR7XUERKE4JA5LZ4AW6K37ZKBZQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:28.206727754 (12h50m27.796072741s) [upload=13303]: Next returned nil segment Aug 7 22:30:28.206744677 (12h50m27.796089664s) [upload=13303]: Waiting for error group managing segments... Aug 7 22:30:28.490025109 (12h50m28.079370096s) [noise=true , address=79.127.205.235:4659 , piece_id=PDPXLV3GK4XVZTDBS2GC564GUEVII3XLOI54ALEUL2YWMBDFUNNQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:28.506048426 (12h50m28.095393403s) [noise=true , address=79.127.205.234:4588 , piece_id=SCZQTO3IY6647Q6MRG3J2AMSWJAWIYKPTX2S2YQHIENXEFZ3C53A, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:28.547644951 (12h50m28.136989938s) [noise=true , address=64.49.13.169:4709 , piece_id=AWEF3LXORD7TDQACIKTDS4NM5XHF676LYFTKKULKNCQ6HA6ABNLQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:28.617351799 (12h50m28.206696786s) [noise=true , address=64.49.13.137:4494 , piece_id=LFDJNPET6CVU2GRXM2KJ3LGNJPTGJ5EM4PWPHF3XVYMPHS366DZA, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.022989834 (12h50m28.612334841s) [noise=true , address=79.127.201.215:4356 , piece_id=FUQGELP6V2FJZ6PVPVKMYKWEAY25P7OAYA5ORTS26FL4FMGAF7XQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.123377683 (12h50m28.71272267s) [noise=true , address=104.36.82.62:4435 , piece_id=A6FYWW6YDPCOCWN3NYSVWHP6G3HW64V5PRS5PEMBFPVHILQOJSYA, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.128754752 (12h50m28.718099729s) [noise=true , address=104.36.82.58:4266 , piece_id=IHNIYP6QQ5HRNGMAHY3NNXQMJM6TOGP4JM5INFUQONEKONUD3BEA, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.473373880 (12h50m29.062718857s) [noise=true , address=104.36.82.52:4114 , piece_id=C5IZX5N3O4JEE2RDMVYZGSOWXVKQSGBJD67NOUNFT4LHMRNZL37A, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.539252935 (12h50m29.128597922s) [noise=true , address=104.36.82.34:4576 , piece_id=G3A3X2KHUXIYXEF4FODXHPISXNSWNTLTP5RIXXPIWGMDWWEYBGUQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.597507541 (12h50m29.186852528s) [noise=true , address=104.36.82.35:4583 , piece_id=L4R2PO7KNIIOIN7BDCYLRCO2MRR7XUERKE4JA5LZ4AW6K37ZKBZQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.641259711 (12h50m29.230604718s) [noise=true , address=45.91.116.13:4160 , piece_id=UGSRSWK3RVAIVHVFWBHMUGHTZLZNCJN3DSSJTBF3EP3DY4UHMG5A, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:30.117032753 (12h50m29.70637773s) [noise=true , address=104.36.82.46:4504 , piece_id=W7DK423UD6SXLWL3LVYWLGTCFNUDATLWKEMQIPYXX4WZ555GD34A, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:30.995539243 (12h50m30.58488423s) [noise=true , address=104.36.82.37:4684 , piece_id=3VZGRTMMP3FQSDOTGNTVLJ43IHMA5FSUBOZ4P2SKWCYG5COVF7LQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:31.436214934 (12h50m31.025559921s) [noise=true , address=104.36.82.61:4404 , piece_id=ZLDF3FSQTOGUD4OMZYNQZZ6MUTVUF6NICTZBHRPBUM5OR3SDYVIA, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:31.436243469 (12h50m31.025588456s) [seg_pos={0 0}, upload=13303]: Segment reached optimal threshold of 14 pieces. Aug 7 22:30:31.436426830 (12h50m31.025771817s) [noise=true , address=104.36.82.16:4325 , piece_id=M4UMVR3RXMLSL3BI55AHRHPL46AAO6MU7V3V3GBZ7HQCR7WRZWSQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aTYNxPbXJGM29Xqovc5VK4cun64wr2DoduaVCvKDj38xCLGKJ): protocol: expected piece hash; context canceled; EOF Aug 7 22:30:31.436556607 (12h50m31.025901594s) [noise=true , address=104.36.82.9:4033 , piece_id=WDDQWPVVTQKSC4JUHUXZ27G6GJXGGBCXBBSQMVEX7ONSCLEFFTAA, seg_pos={0 0}, upload=13303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C): protocol: write tcp 188.166.225.170:49338->104.36.82.9:4033: use of closed network connection; write tcp 188.166.225.170:49338->104.36.82.9:4033: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49338->104.36.82.9:4033: use of closed network connection Aug 7 22:30:31.436606383 (12h50m31.02595137s) [noise=true , address=104.36.82.44:4884 , piece_id=HRBXMCCRYU4FBS73Q2IB5BK5BUUHAALRNODQ5XERZICLQ6PUVOBQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qGXdnm83P4eMnCtZdRt2x5yAWP7pQafWwoPG72daZJVR9pyKG): protocol: write tcp 188.166.225.170:45006->104.36.82.44:4884: use of closed network connection; write tcp 188.166.225.170:45006->104.36.82.44:4884: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45006->104.36.82.44:4884: use of closed network connection Aug 7 22:30:31.436613496 (12h50m31.025958483s) [noise=true , address=104.36.82.5:4853 , piece_id=FZOYMSE2HPKJUXG7RXFQH77H7GTC34UFQVKAXJBKQ4YYI7ICAKMA, seg_pos={0 0}, upload=13303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u): protocol: write tcp 188.166.225.170:54220->104.36.82.5:4853: use of closed network connection; write tcp 188.166.225.170:54220->104.36.82.5:4853: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54220->104.36.82.5:4853: use of closed network connection Aug 7 22:30:31.436624858 (12h50m31.025969845s) [seg_pos={0 0}, upload=13303]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:30:31.436720120 (12h50m31.026065107s) [upload=13303]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:30:31.436724318 (12h50m31.026069295s) [upload=13303]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:30:31.748725210 (12h50m31.338070187s) [upload=13303]: Done uploading segments... Aug 7 22:30:31.748742002 (12h50m31.338086989s) [upload=13303]: Upload finished. err: <nil> Aug 7 22:31:18.785686607 (12h51m18.375031594s) [upload=13318]: Starting upload Aug 7 22:31:18.785729449 (12h51m18.375074437s) [upload=13318]: Done starting upload Aug 7 22:31:18.785814212 (12h51m18.375159189s) [upload=13318]: Uploading segments... Aug 7 22:31:18.785960702 (12h51m18.375305679s) [upload=13318]: Got next segment. Inline: false Aug 7 22:31:18.785988354 (12h51m18.375333341s) [upload=13318]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:31:18.785993003 (12h51m18.37533798s) [upload=13318]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:31:19.001950809 (12h51m18.591295786s) [upload=13318]: Next segment err: metaclient: Too Many Requests Aug 7 22:31:19.001977260 (12h51m18.591322247s) [upload=13318]: Done uploading segments... Aug 7 22:31:19.001992078 (12h51m18.591337055s) [upload=13318]: Upload finished. err: metaclient: Too Many Requests Duration: 8.69951315s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 24 out of 60 is failed (threshold=3) [................XXXXXXXXXXXXXXXXXXX.XXX.X.X.................] duration: 2.387429937s FAIL (2025-08-07 22:43:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.387446168s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 239.281811ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 22:43:39 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 239.306117ms edge gateway vhost bgp us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [......................XXXXXXXXXXXXXXXXX.X...................] duration: 5.737689587s FAIL (2025-08-07 22:43:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.737708645s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 19 out of 60 is failed (threshold=3) [................XXXXXXXXXXXXXXXX.XX..X......................] duration: 1.924597563s FAIL (2025-08-07 22:43:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7 22:22:45.818723517 (12h42m45.408068504s) [upload=13188]: Done starting upload Aug 7 22:22:45.818774856 (12h42m45.408119843s) [upload=13188]: Uploading segments... Aug 7 22:22:45.818807999 (12h42m45.408152996s) [upload=13188]: Got next segment. Inline: true Aug 7 22:22:45.818812908 (12h42m45.408157895s) [upload=13188]: Waiting for error group managing segments... Aug 7 22:22:45.818870779 (12h42m45.408215776s) [upload=13188]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:45.818879214 (12h42m45.408224202s) [upload=13188]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:45.818883494 (12h42m45.408228481s) [upload=13188]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:15.175828623 (12h43m14.76517361s) [upload=13188]: Done uploading segments... Aug 7 22:23:15.175844935 (12h43m14.765189912s) [upload=13188]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:16.350710694 (12h43m15.940055691s) [upload=13195]: Starting upload Aug 7 22:23:16.350768756 (12h43m15.940113733s) [upload=13195]: Done starting upload Aug 7 22:23:16.350814804 (12h43m15.940159791s) [upload=13195]: Uploading segments... Aug 7 22:23:16.350849079 (12h43m15.940194056s) [upload=13195]: Got next segment. Inline: true Aug 7 22:23:16.350852777 (12h43m15.940197764s) [upload=13195]: Waiting for error group managing segments... Aug 7 22:23:16.350884928 (12h43m15.940229915s) [upload=13195]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:16.350888735 (12h43m15.940233722s) [upload=13195]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:16.350891020 (12h43m15.940236007s) [upload=13195]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:45.695642688 (12h43m45.284987675s) [upload=13195]: Done uploading segments... Aug 7 22:23:45.695657867 (12h43m45.285002854s) [upload=13195]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:08.254775832 (12h44m7.844120819s) [upload=13210]: Starting upload Aug 7 22:24:08.254857087 (12h44m7.844202074s) [upload=13210]: Done starting upload Aug 7 22:24:08.254936559 (12h44m7.844281546s) [upload=13210]: Uploading segments... Aug 7 22:24:08.255031861 (12h44m7.844376858s) [upload=13210]: Got next segment. Inline: true Aug 7 22:24:08.255037032 (12h44m7.844382019s) [upload=13210]: Waiting for error group managing segments... Aug 7 22:24:08.255078220 (12h44m7.844423207s) [upload=13210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:08.255093450 (12h44m7.844438437s) [upload=13210]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:08.255096435 (12h44m7.844441422s) [upload=13210]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:37.608974587 (12h44m37.198319574s) [upload=13210]: Done uploading segments... Aug 7 22:24:37.609189158 (12h44m37.198534195s) [upload=13210]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:19.078840222 (12h45m18.668185229s) [upload=13227]: Starting upload Aug 7 22:25:19.078945333 (12h45m18.66829035s) [upload=13227]: Done starting upload Aug 7 22:25:19.079016760 (12h45m18.668361767s) [upload=13227]: Uploading segments... Aug 7 22:25:19.079058389 (12h45m18.668403377s) [upload=13227]: Got next segment. Inline: true Aug 7 22:25:19.079064030 (12h45m18.668409007s) [upload=13227]: Waiting for error group managing segments... Aug 7 22:25:19.079115388 (12h45m18.668460375s) [upload=13227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:19.079121570 (12h45m18.668466557s) [upload=13227]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:19.079126009 (12h45m18.668471077s) [upload=13227]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:48.426015265 (12h45m48.015360252s) [upload=13227]: Done uploading segments... Aug 7 22:25:48.426036394 (12h45m48.015381381s) [upload=13227]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:26:18.229360345 (12h46m17.818705332s) [upload=13243]: Starting upload Aug 7 22:26:18.229421673 (12h46m17.81876665s) [upload=13243]: Done starting upload Aug 7 22:26:18.229465397 (12h46m17.818810384s) [upload=13243]: Uploading segments... Aug 7 22:26:18.229495053 (12h46m17.81884004s) [upload=13243]: Got next segment. Inline: true Aug 7 22:26:18.229498680 (12h46m17.818843667s) [upload=13243]: Waiting for error group managing segments... Aug 7 22:26:18.229531593 (12h46m17.81887658s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:18.229535170 (12h46m17.818880157s) [upload=13243]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:18.229537614 (12h46m17.818882601s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:18.493923010 (12h46m18.083268007s) [upload=13243]: Done uploading segments... Aug 7 22:26:18.493948349 (12h46m18.083293346s) [upload=13243]: Upload finished. err: <nil> Aug 7 22:26:34.238583280 (12h46m33.827928267s) [upload=13246]: Starting upload Aug 7 22:26:34.238671890 (12h46m33.828016897s) [upload=13246]: Done starting upload Aug 7 22:26:34.238753585 (12h46m33.828098573s) [upload=13246]: Uploading segments... Aug 7 22:26:34.238795296 (12h46m33.828140283s) [upload=13246]: Got next segment. Inline: true Aug 7 22:26:34.238800546 (12h46m33.828145533s) [upload=13246]: Waiting for error group managing segments... Aug 7 22:26:34.238842697 (12h46m33.828187684s) [upload=13246]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:34.238847656 (12h46m33.828192633s) [upload=13246]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:34.238851193 (12h46m33.82819618s) [upload=13246]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:03.590020724 (12h47m3.179365711s) [upload=13246]: Done uploading segments... Aug 7 22:27:03.590046854 (12h47m3.179391861s) [upload=13246]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:08.455168905 (12h47m8.044513892s) [upload=13253]: Starting upload Aug 7 22:27:08.455224000 (12h47m8.044568977s) [upload=13253]: Done starting upload Aug 7 22:27:08.455266011 (12h47m8.044610998s) [upload=13253]: Uploading segments... Aug 7 22:27:08.455292903 (12h47m8.04463789s) [upload=13253]: Got next segment. Inline: true Aug 7 22:27:08.455296630 (12h47m8.044641607s) [upload=13253]: Waiting for error group managing segments... Aug 7 22:27:08.455327689 (12h47m8.044672676s) [upload=13253]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:08.455331626 (12h47m8.044676613s) [upload=13253]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:08.455334372 (12h47m8.044679349s) [upload=13253]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:37.813944606 (12h47m37.403289643s) [upload=13253]: Done uploading segments... Aug 7 22:27:37.813971608 (12h47m37.403316615s) [upload=13253]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:28:03.718747816 (12h48m3.308092803s) [upload=13266]: Starting upload Aug 7 22:28:03.718803663 (12h48m3.30814865s) [upload=13266]: Done starting upload Aug 7 22:28:03.718851525 (12h48m3.308196512s) [upload=13266]: Uploading segments... Aug 7 22:28:03.718879849 (12h48m3.308224836s) [upload=13266]: Got next segment. Inline: true Aug 7 22:28:03.718883686 (12h48m3.308228673s) [upload=13266]: Waiting for error group managing segments... Aug 7 22:28:03.718918023 (12h48m3.30826301s) [upload=13266]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:03.718922461 (12h48m3.308267448s) [upload=13266]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:03.718924966 (12h48m3.308269953s) [upload=13266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:04.001556505 (12h48m3.590901502s) [upload=13266]: Done uploading segments... Aug 7 22:28:04.001574169 (12h48m3.590919156s) [upload=13266]: Upload finished. err: <nil> Aug 7 22:28:53.653874975 (12h48m53.243219952s) [upload=13280]: Starting upload Aug 7 22:28:53.653923989 (12h48m53.243268976s) [upload=13280]: Done starting upload Aug 7 22:28:53.653970338 (12h48m53.243315315s) [upload=13280]: Uploading segments... Aug 7 22:28:53.654001658 (12h48m53.243346645s) [upload=13280]: Got next segment. Inline: true Aug 7 22:28:53.654005195 (12h48m53.243350182s) [upload=13280]: Waiting for error group managing segments... Aug 7 22:28:53.654037516 (12h48m53.243382503s) [upload=13280]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:53.654041253 (12h48m53.24338623s) [upload=13280]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:53.654043598 (12h48m53.243388575s) [upload=13280]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:53.931677747 (12h48m53.521022724s) [upload=13280]: Done uploading segments... Aug 7 22:28:53.931693927 (12h48m53.521038914s) [upload=13280]: Upload finished. err: <nil> Aug 7 22:29:13.069018260 (12h49m12.658363247s) [upload=13284]: Starting upload Aug 7 22:29:13.069075108 (12h49m12.658420096s) [upload=13284]: Done starting upload Aug 7 22:29:13.069124593 (12h49m12.65846958s) [upload=13284]: Uploading segments... Aug 7 22:29:13.069159800 (12h49m12.658504787s) [upload=13284]: Got next segment. Inline: true Aug 7 22:29:13.069163377 (12h49m12.658508364s) [upload=13284]: Waiting for error group managing segments... Aug 7 22:29:13.069195007 (12h49m12.658539995s) [upload=13284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:29:13.069198645 (12h49m12.658543632s) [upload=13284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:29:13.069201010 (12h49m12.658545997s) [upload=13284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:42.426836268 (12h49m42.016181256s) [upload=13284]: Done uploading segments... Aug 7 22:29:42.426899691 (12h49m42.016244678s) [upload=13284]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.924626629s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 11 out of 60 is failed (threshold=3) [..............................XXXXXXXXX..X.X................] duration: 6.310207301s FAIL (2025-08-07 22:43:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=13236]: Begin upload segment... Aug 7 22:25:52.992307080 (12h45m52.581652067s) [seg_pos={0 0}, upload=13236]: Done begin upload segment. Aug 7 22:25:52.992331798 (12h45m52.581676775s) [noise=true , address=104.36.82.13:4213 , piece_id=5W5FD4WMMSZ7BKHFQI4GYOODXZ2P2MMAIQNK6CD6XILH22C53JGQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992683770 (12h45m52.582028757s) [noise=true , address=104.36.82.9:4024 , piece_id=VBFOHRWDKWRY2774KJVK3JXJSMG4CPVDBBECCSZVJHR4274KLRMA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992785475 (12h45m52.582130462s) [noise=true , address=104.36.82.22:4527 , piece_id=JBNJOSETDAB6VG7PGVXIBUP6GPF2FBPJURBZNBR3BJOGMRK3EIQA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992942396 (12h45m52.582287373s) [noise=true , address=45.91.116.11:4124 , piece_id=TLATR2WNDVRR3WWTT3FZARUJ2QWWQFZBDYXLZ7PUPFOAEGJI4BTQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993132439 (12h45m52.582477457s) [noise=true , address=104.36.82.5:4855 , piece_id=6EW7GNJ6KVWKNFHGFZVPEYPD4TDIVVZUXRW6AGJ3IGVANEAHYDUA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993337862 (12h45m52.582682849s) [noise=true , address=64.49.13.170:4717 , piece_id=52FN65VRQ24LBOGHJ22EQQM3MNI3EB2UXB57P5LD5GPLSABNS22Q, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993511144 (12h45m52.582856131s) [noise=true , address=104.36.82.60:4361 , piece_id=EPWXHNXPVAUX75FUAXY57QZDKRXIEDHRXFL6PEBQGXY2A3Q4KNHA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993715204 (12h45m52.583060191s) [noise=true , address=104.36.82.58:4281 , piece_id=5UH27M25HE2LEZJ53H7QBKU6I6N2JE6F4PZTLOE6XP3GEDCZLEEQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993820806 (12h45m52.583165784s) [noise=true , address=45.91.116.9:4284 , piece_id=LT3RP2EBVHIFIMAIGM3DW6QMPNLKZCK5FD4GQEQI6BFWYD56ZUBQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994068250 (12h45m52.583413237s) [noise=true , address=79.127.205.233:4633 , piece_id=LBO5XLAXBRDURMDMN6MZ4HLVSG4TTUZPWCFTOTB6QMWAS5IIRLBA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994092416 (12h45m52.583437393s) [noise=true , address=104.36.82.60:4346 , piece_id=3NU3QBO5DAW5ZB7NTAHWP4C7RE4BYDPHZ77JOUQT3R3CBTOMONLA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994295394 (12h45m52.583640371s) [noise=true , address=45.91.116.7:4037 , piece_id=2NDGWBVYO3D2RYJ57QSXT632TT24SMRNWA3YZ2YW2BN6TBBLMRQA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994307056 (12h45m52.583652033s) [noise=true , address=104.36.82.46:4505 , piece_id=A7XPGAYUTNB26A6DJQNQ4NZ4FG27FUBQNEH5AQ7IFLYR3DLGG6SQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994547126 (12h45m52.583892103s) [noise=true , address=104.36.82.61:4389 , piece_id=WBB2Y7YYCEML23D3CKJHGIEOUELRTVG3NXVGI4JDD7XPNQ4STENQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994782436 (12h45m52.584127423s) [noise=true , address=64.49.13.154:4934 , piece_id=LDQTK4SFJPTL7NAY5XXKBIFCVLNRSLHLACTJ65MWCHFSBV4EKJYA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998117076 (12h45m52.587462053s) [noise=true , address=104.36.82.56:4252 , piece_id=XQWAMPOYTVGKYEHIEEEIDDST5BOIN2XCT7B3SFUR4U7PQBROEELQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998596283 (12h45m52.58794127s) [noise=true , address=104.36.82.15:4261 , piece_id=C3RUC2F5LVRKH4MYTXDYOAP5KJGRZ6TAX57XULVBAKOS7ZK4QONQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998917628 (12h45m52.588262615s) [noise=true , address=104.36.82.23:4473 , piece_id=N54OVKSO4KKX4NPORULBLAPVE5UT5RU6BBIEJCCMKG5553GVY6JA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:54.121532197 (12h45m53.710877184s) [upload=13236]: Next returned nil segment Aug 7 22:25:54.121549150 (12h45m53.710894137s) [upload=13236]: Waiting for error group managing segments... Aug 7 22:25:54.447812563 (12h45m54.03715755s) [noise=true , address=79.127.205.233:4633 , piece_id=LBO5XLAXBRDURMDMN6MZ4HLVSG4TTUZPWCFTOTB6QMWAS5IIRLBA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:54.450658769 (12h45m54.040003756s) [noise=true , address=64.49.13.170:4717 , piece_id=52FN65VRQ24LBOGHJ22EQQM3MNI3EB2UXB57P5LD5GPLSABNS22Q, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:54.505775361 (12h45m54.095120349s) [noise=true , address=64.49.13.154:4934 , piece_id=LDQTK4SFJPTL7NAY5XXKBIFCVLNRSLHLACTJ65MWCHFSBV4EKJYA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.074032067 (12h45m54.663377054s) [noise=true , address=104.36.82.60:4346 , piece_id=3NU3QBO5DAW5ZB7NTAHWP4C7RE4BYDPHZ77JOUQT3R3CBTOMONLA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.160612428 (12h45m54.749957415s) [noise=true , address=104.36.82.61:4389 , piece_id=WBB2Y7YYCEML23D3CKJHGIEOUELRTVG3NXVGI4JDD7XPNQ4STENQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.161069252 (12h45m54.750414239s) [noise=true , address=104.36.82.22:4527 , piece_id=JBNJOSETDAB6VG7PGVXIBUP6GPF2FBPJURBZNBR3BJOGMRK3EIQA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.172460447 (12h45m54.761805434s) [noise=true , address=104.36.82.60:4361 , piece_id=EPWXHNXPVAUX75FUAXY57QZDKRXIEDHRXFL6PEBQGXY2A3Q4KNHA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.217919931 (12h45m54.807264908s) [noise=true , address=104.36.82.56:4252 , piece_id=XQWAMPOYTVGKYEHIEEEIDDST5BOIN2XCT7B3SFUR4U7PQBROEELQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.226637230 (12h45m54.815982217s) [noise=true , address=104.36.82.58:4281 , piece_id=5UH27M25HE2LEZJ53H7QBKU6I6N2JE6F4PZTLOE6XP3GEDCZLEEQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.271492068 (12h45m54.860837055s) [noise=true , address=45.91.116.11:4124 , piece_id=TLATR2WNDVRR3WWTT3FZARUJ2QWWQFZBDYXLZ7PUPFOAEGJI4BTQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.281497411 (12h45m54.870842388s) [noise=true , address=45.91.116.9:4284 , piece_id=LT3RP2EBVHIFIMAIGM3DW6QMPNLKZCK5FD4GQEQI6BFWYD56ZUBQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.402216807 (12h45m54.991561794s) [noise=true , address=104.36.82.9:4024 , piece_id=VBFOHRWDKWRY2774KJVK3JXJSMG4CPVDBBECCSZVJHR4274KLRMA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.632857570 (12h45m55.222202558s) [noise=true , address=104.36.82.5:4855 , piece_id=6EW7GNJ6KVWKNFHGFZVPEYPD4TDIVVZUXRW6AGJ3IGVANEAHYDUA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:56.838313689 (12h45m56.427658676s) [noise=true , address=104.36.82.13:4213 , piece_id=5W5FD4WMMSZ7BKHFQI4GYOODXZ2P2MMAIQNK6CD6XILH22C53JGQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:56.838338066 (12h45m56.427683053s) [seg_pos={0 0}, upload=13236]: Segment reached optimal threshold of 14 pieces. Aug 7 22:25:56.838585389 (12h45m56.427930366s) [noise=true , address=45.91.116.7:4037 , piece_id=2NDGWBVYO3D2RYJ57QSXT632TT24SMRNWA3YZ2YW2BN6TBBLMRQA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; write tcp 188.166.225.170:55570->45.91.116.7:4037: use of closed network connection; write tcp 188.166.225.170:55570->45.91.116.7:4037: use of closed network connection; EOF Aug 7 22:25:56.838630465 (12h45m56.427975452s) [noise=true , address=104.36.82.46:4505 , piece_id=A7XPGAYUTNB26A6DJQNQ4NZ4FG27FUBQNEH5AQ7IFLYR3DLGG6SQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf): protocol: write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection; write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection Aug 7 22:25:56.838682564 (12h45m56.428027551s) [noise=true , address=104.36.82.23:4473 , piece_id=N54OVKSO4KKX4NPORULBLAPVE5UT5RU6BBIEJCCMKG5553GVY6JA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC): protocol: write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection; write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection Aug 7 22:25:56.838697754 (12h45m56.428042731s) [noise=true , address=104.36.82.15:4261 , piece_id=C3RUC2F5LVRKH4MYTXDYOAP5KJGRZ6TAX57XULVBAKOS7ZK4QONQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i1jcJdEP7Ch9uaGcjDaVxpodx55dJEHUKFbcr1GeKQ9waWjYz): protocol: expected piece hash; context canceled; EOF Aug 7 22:25:56.838720558 (12h45m56.428065535s) [seg_pos={0 0}, upload=13236]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:25:56.838823714 (12h45m56.428168701s) [upload=13236]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:25:56.838828013 (12h45m56.428173s) [upload=13236]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:30:17.177125535 (12h50m16.766470522s) [upload=13236]: Done uploading segments... Aug 7 22:30:17.177153268 (12h50m16.766498255s) [upload=13236]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.310239282s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [....................XXXXXXXXXXXXXXXXXXX...X.................] duration: 5.967204163s FAIL (2025-08-07 22:43:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.967229091s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.......................XXXXXXXXXXXXXXXXX....................] duration: 6.646756882s FAIL (2025-08-07 22:43:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :32.805219872 (12h43m32.394564859s) [upload=13200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:32.805223589 (12h43m32.394568576s) [upload=13200]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:24:02.160684947 (12h44m1.750029944s) [upload=13200]: Next segment err: context deadline exceeded Aug 7 22:24:02.160927551 (12h44m1.750272538s) [upload=13200]: Done uploading segments... Aug 7 22:24:02.160942019 (12h44m1.750287007s) [upload=13200]: Upload finished. err: context deadline exceeded Aug 7 22:24:12.500861314 (12h44m12.090206301s) [upload=13211]: Starting upload Aug 7 22:24:12.500912582 (12h44m12.090257569s) [upload=13211]: Done starting upload Aug 7 22:24:12.500973068 (12h44m12.090318045s) [upload=13211]: Uploading segments... Aug 7 22:24:12.501078069 (12h44m12.090423056s) [upload=13211]: Got next segment. Inline: false Aug 7 22:24:12.501106834 (12h44m12.090451821s) [upload=13211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:12.501110571 (12h44m12.090455558s) [upload=13211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:24:25.371267525 (12h44m24.960612512s) [seg_pos={0 0}, upload=13211]: Begin upload segment... Aug 7 22:24:25.371330936 (12h44m24.960675953s) [seg_pos={0 0}, upload=13211]: Done begin upload segment. Aug 7 22:24:25.371383426 (12h44m24.960728413s) [noise=true , address=77.90.31.14:4661 , piece_id=73RH6546ZSA46KNTPMI3YPHM65Y3DJG5PCAAPGYJJFXDQ6KM5MRA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.371396932 (12h44m24.96074194s) [noise=true , address=77.90.31.19:4841 , piece_id=GPGSM3FHS76WAHMKYCXKIU57NEENBDZFAMIBSFP6JWV5K33BDCOQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.371863044 (12h44m24.961208021s) [noise=true , address=45.91.116.9:4261 , piece_id=F3JBOLQO76A5WQKYNQAKPAWVWZO5U5EYX7ZZIKQMOAUZUQCRZ7GA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372024263 (12h44m24.96136925s) [noise=true , address=45.91.116.12:4225 , piece_id=47OO6QUMLHWH54IAXRLTQEN56Z5ITGXCGKSTQFGSHTF32X3Z7HSQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372219627 (12h44m24.961564614s) [noise=true , address=104.36.82.2:4729 , piece_id=JYG3SHRCFAE37242ADYQA2CXJRRGO5APTOBZ5725GMZ2FRMPYLKA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372406654 (12h44m24.961751641s) [noise=true , address=45.91.116.7:4009 , piece_id=N25P6VDHR6RAW44ZNMAPNYJUX4AUV2UI3ROEX67KTWLK7RRB4K7Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372852217 (12h44m24.962197204s) [noise=true , address=45.91.116.6:4973 , piece_id=WLZJLHSER36K377KZCHHCLWNANIVKGHVKF2I3FBC5UDDLSIUCHFQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373233576 (12h44m24.962578563s) [noise=true , address=45.91.116.4:4937 , piece_id=ALO2UUBMXABZGWARBYTKGYY5DHLJEGULYGIZ7E4JQCYNFGP4JG7Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373581912 (12h44m24.9629269s) [noise=true , address=45.91.116.5:4299 , piece_id=LMJE5NXY2LTDFI3BZY2YGMAXFURMI5NOSZWW2ABZK4NFKQK5JROQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373901074 (12h44m24.963246061s) [noise=true , address=104.36.82.11:4113 , piece_id=7UYIPJZWWCEIMHEOPGC3DKXXTULAHTNF5QMPJJYIHXTKPSDOMM5A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374042614 (12h44m24.963387601s) [noise=true , address=77.90.31.17:4772 , piece_id=UEGCYD3SF4UF3CHRT5GEAZ6B77VKMG7S7PMSSRLZT6IP4ZKOTHMA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374483207 (12h44m24.963828194s) [noise=true , address=104.36.82.7:4912 , piece_id=H6DSVHM5ZX2GWEPHGRDH2WEDRHCLEYLZ62ISEHBLEITT4NGNDT7A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374810343 (12h44m24.96415533s) [noise=true , address=45.91.116.4:4939 , piece_id=S4PTGHW7FTLW5Q7W2VQZHLAI4YQZ25U35J7LUYSS5EXK552GUE2A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375117380 (12h44m24.964462368s) [noise=true , address=77.90.31.11:4555 , piece_id=CTQZTDFEGF756WLYI6U6NGMGBLFOKXZX3QWTQE7KBAITW43WVIJA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375526834 (12h44m24.964871821s) [noise=true , address=45.91.116.10:4082 , piece_id=4QZEPLP5I47FAOHIYQWJEBODVUEWTM25IQ652LDWLFZERCPTEXHA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375955474 (12h44m24.965300461s) [noise=true , address=104.36.82.10:4078 , piece_id=QKGPABRCDU26EDHDNRMCHMUQ3XTOQSN47WG5RKF6S4LD7OG2HY6Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:27.072833335 (12h44m26.662178332s) [upload=13211]: Next returned nil segment Aug 7 22:24:27.072902037 (12h44m26.662247034s) [upload=13211]: Waiting for error group managing segments... Aug 7 22:24:27.436037535 (12h44m27.025382522s) [noise=true , address=104.36.82.7:4912 , piece_id=H6DSVHM5ZX2GWEPHGRDH2WEDRHCLEYLZ62ISEHBLEITT4NGNDT7A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.446916249 (12h44m27.036261236s) [noise=true , address=104.36.82.2:4729 , piece_id=JYG3SHRCFAE37242ADYQA2CXJRRGO5APTOBZ5725GMZ2FRMPYLKA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.566730713 (12h44m27.15607569s) [noise=true , address=45.91.116.9:4261 , piece_id=F3JBOLQO76A5WQKYNQAKPAWVWZO5U5EYX7ZZIKQMOAUZUQCRZ7GA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.594802733 (12h44m27.18414772s) [noise=true , address=45.91.116.4:4937 , piece_id=ALO2UUBMXABZGWARBYTKGYY5DHLJEGULYGIZ7E4JQCYNFGP4JG7Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.613347363 (12h44m27.20269235s) [noise=true , address=77.90.31.14:4661 , piece_id=73RH6546ZSA46KNTPMI3YPHM65Y3DJG5PCAAPGYJJFXDQ6KM5MRA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.623084653 (12h44m27.21242964s) [noise=true , address=45.91.116.10:4082 , piece_id=4QZEPLP5I47FAOHIYQWJEBODVUEWTM25IQ652LDWLFZERCPTEXHA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.661186031 (12h44m27.250531018s) [noise=true , address=77.90.31.19:4841 , piece_id=GPGSM3FHS76WAHMKYCXKIU57NEENBDZFAMIBSFP6JWV5K33BDCOQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.698198064 (12h44m27.287543051s) [noise=true , address=45.91.116.6:4973 , piece_id=WLZJLHSER36K377KZCHHCLWNANIVKGHVKF2I3FBC5UDDLSIUCHFQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.724830297 (12h44m27.314175285s) [noise=true , address=45.91.116.4:4939 , piece_id=S4PTGHW7FTLW5Q7W2VQZHLAI4YQZ25U35J7LUYSS5EXK552GUE2A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.734317339 (12h44m27.323662326s) [noise=true , address=45.91.116.5:4299 , piece_id=LMJE5NXY2LTDFI3BZY2YGMAXFURMI5NOSZWW2ABZK4NFKQK5JROQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.908078026 (12h44m27.497423013s) [noise=true , address=77.90.31.17:4772 , piece_id=UEGCYD3SF4UF3CHRT5GEAZ6B77VKMG7S7PMSSRLZT6IP4ZKOTHMA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.985050925 (12h44m27.574395912s) [noise=true , address=45.91.116.7:4009 , piece_id=N25P6VDHR6RAW44ZNMAPNYJUX4AUV2UI3ROEX67KTWLK7RRB4K7Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:28.048748935 (12h44m27.638093922s) [noise=true , address=77.90.31.11:4555 , piece_id=CTQZTDFEGF756WLYI6U6NGMGBLFOKXZX3QWTQE7KBAITW43WVIJA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:28.048775115 (12h44m27.638120102s) [seg_pos={0 0}, upload=13211]: Segment reached optimal threshold of 13 pieces. Aug 7 22:24:28.048900375 (12h44m27.638245362s) [noise=true , address=45.91.116.12:4225 , piece_id=47OO6QUMLHWH54IAXRLTQEN56Z5ITGXCGKSTQFGSHTF32X3Z7HSQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Aug 7 22:24:28.049006828 (12h44m27.638351815s) [noise=true , address=104.36.82.11:4113 , piece_id=7UYIPJZWWCEIMHEOPGC3DKXXTULAHTNF5QMPJJYIHXTKPSDOMM5A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection; write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection Aug 7 22:24:28.049050902 (12h44m27.638395889s) [noise=true , address=104.36.82.10:4078 , piece_id=QKGPABRCDU26EDHDNRMCHMUQ3XTOQSN47WG5RKF6S4LD7OG2HY6Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection; write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection Aug 7 22:24:28.049061182 (12h44m27.638406169s) [seg_pos={0 0}, upload=13211]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 22:24:28.049126247 (12h44m27.638471224s) [upload=13211]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:24:28.049130154 (12h44m27.638475141s) [upload=13211]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:23.601003390 (12h47m23.190348377s) [upload=13211]: Done uploading segments... Aug 7 22:27:23.601027807 (12h47m23.190372794s) [upload=13211]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.646786187s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [...........................X..X.............................] duration: 801.85755ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 22:43:23 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 801.882037ms edge gateway vhost us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 656.500353ms target: https://us1.storj.io/login FAIL (2025-08-07 22:43:18 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 656.520912ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 17 out of 60 is failed (threshold=3) [.....................XXXXXXXXXXXXXXXX.X.....................] duration: 1.915357015s FAIL (2025-08-07 22:43:11 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== starting upload Aug 7 22:19:13.035284043 (12h39m12.62462903s) [upload=13134]: Uploading segments... Aug 7 22:19:13.035347064 (12h39m12.624692051s) [upload=13134]: Got next segment. Inline: true Aug 7 22:19:13.035351633 (12h39m12.62469662s) [upload=13134]: Waiting for error group managing segments... Aug 7 22:19:13.035393122 (12h39m12.624738099s) [upload=13134]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:19:13.035397750 (12h39m12.624742739s) [upload=13134]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:19:13.035400156 (12h39m12.624745143s) [upload=13134]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:20:12.392096704 (12h40m11.981441692s) [upload=13134]: Done uploading segments... Aug 7 22:20:12.392135208 (12h40m11.981480225s) [upload=13134]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:20:13.302960454 (12h40m12.892305441s) [upload=13150]: Starting upload Aug 7 22:20:13.303052761 (12h40m12.892397748s) [upload=13150]: Done starting upload Aug 7 22:20:13.303103337 (12h40m12.892448325s) [upload=13150]: Uploading segments... Aug 7 22:20:13.303136621 (12h40m12.892481608s) [upload=13150]: Got next segment. Inline: true Aug 7 22:20:13.303140047 (12h40m12.892485034s) [upload=13150]: Waiting for error group managing segments... Aug 7 22:20:13.303175805 (12h40m12.892520793s) [upload=13150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:20:13.303181377 (12h40m12.892526364s) [upload=13150]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:20:13.303183922 (12h40m12.892528909s) [upload=13150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:21:12.655202481 (12h41m12.244547468s) [upload=13150]: Done uploading segments... Aug 7 22:21:12.655262385 (12h41m12.244607372s) [upload=13150]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:21:28.179922232 (12h41m27.769267219s) [upload=13167]: Starting upload Aug 7 22:21:28.180014519 (12h41m27.769359506s) [upload=13167]: Done starting upload Aug 7 22:21:28.180090304 (12h41m27.769435291s) [upload=13167]: Uploading segments... Aug 7 22:21:28.180123167 (12h41m27.769468154s) [upload=13167]: Got next segment. Inline: true Aug 7 22:21:28.180126524 (12h41m27.769471511s) [upload=13167]: Waiting for error group managing segments... Aug 7 22:21:28.180185667 (12h41m27.769530654s) [upload=13167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:21:28.180189734 (12h41m27.769534721s) [upload=13167]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:21:28.180192199 (12h41m27.769537186s) [upload=13167]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:22:27.535085641 (12h42m27.124430628s) [upload=13167]: Done uploading segments... Aug 7 22:22:27.535157829 (12h42m27.124502816s) [upload=13167]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:22:28.402559244 (12h42m27.991904221s) [upload=13183]: Starting upload Aug 7 22:22:28.402660068 (12h42m27.992005055s) [upload=13183]: Done starting upload Aug 7 22:22:28.402728308 (12h42m27.992073335s) [upload=13183]: Uploading segments... Aug 7 22:22:28.402775298 (12h42m27.992120335s) [upload=13183]: Got next segment. Inline: true Aug 7 22:22:28.402781640 (12h42m27.992126627s) [upload=13183]: Waiting for error group managing segments... Aug 7 22:22:28.402833088 (12h42m27.992178146s) [upload=13183]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:28.402840001 (12h42m27.992184988s) [upload=13183]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:28.402845021 (12h42m27.992190008s) [upload=13183]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:27.752834209 (12h43m27.342179246s) [upload=13183]: Done uploading segments... Aug 7 22:23:27.752884696 (12h43m27.342229693s) [upload=13183]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:28.630476036 (12h43m28.219821014s) [upload=13198]: Starting upload Aug 7 22:23:28.630528016 (12h43m28.219873003s) [upload=13198]: Done starting upload Aug 7 22:23:28.630599864 (12h43m28.219944851s) [upload=13198]: Uploading segments... Aug 7 22:23:28.630637285 (12h43m28.219982272s) [upload=13198]: Got next segment. Inline: true Aug 7 22:23:28.630640952 (12h43m28.219985939s) [upload=13198]: Waiting for error group managing segments... Aug 7 22:23:28.630673745 (12h43m28.220018732s) [upload=13198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:28.630677352 (12h43m28.220022339s) [upload=13198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:28.630679597 (12h43m28.220024584s) [upload=13198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:27.987023038 (12h44m27.576368025s) [upload=13198]: Done uploading segments... Aug 7 22:24:27.987035291 (12h44m27.576380268s) [upload=13198]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:28.848154918 (12h44m28.437499895s) [upload=13216]: Starting upload Aug 7 22:24:28.848218670 (12h44m28.437563657s) [upload=13216]: Done starting upload Aug 7 22:24:28.848264347 (12h44m28.437609324s) [upload=13216]: Uploading segments... Aug 7 22:24:28.848331396 (12h44m28.437676373s) [upload=13216]: Got next segment. Inline: true Aug 7 22:24:28.848335494 (12h44m28.437680481s) [upload=13216]: Waiting for error group managing segments... Aug 7 22:24:28.848369038 (12h44m28.437714025s) [upload=13216]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:28.848373045 (12h44m28.437718032s) [upload=13216]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:28.848375189 (12h44m28.437720176s) [upload=13216]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:28.201395574 (12h45m27.790740561s) [upload=13216]: Done uploading segments... Aug 7 22:25:28.201413839 (12h45m27.790758816s) [upload=13216]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:34.254413826 (12h45m33.843758814s) [upload=13234]: Starting upload Aug 7 22:25:34.254496073 (12h45m33.84384106s) [upload=13234]: Done starting upload Aug 7 22:25:34.254567009 (12h45m33.843911996s) [upload=13234]: Uploading segments... Aug 7 22:25:34.254604511 (12h45m33.843949508s) [upload=13234]: Got next segment. Inline: true Aug 7 22:25:34.254609231 (12h45m33.843954218s) [upload=13234]: Waiting for error group managing segments... Aug 7 22:25:34.254647023 (12h45m33.84399201s) [upload=13234]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:34.254651561 (12h45m33.843996558s) [upload=13234]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:34.254654356 (12h45m33.843999343s) [upload=13234]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:33.597318767 (12h46m33.186663754s) [upload=13234]: Done uploading segments... Aug 7 22:26:33.597351351 (12h46m33.186696338s) [upload=13234]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:26:34.495602477 (12h46m34.084947454s) [upload=13247]: Starting upload Aug 7 22:26:34.495652543 (12h46m34.08499753s) [upload=13247]: Done starting upload Aug 7 22:26:34.495718250 (12h46m34.085063237s) [upload=13247]: Uploading segments... Aug 7 22:26:34.495750731 (12h46m34.085095719s) [upload=13247]: Got next segment. Inline: true Aug 7 22:26:34.495754388 (12h46m34.085099375s) [upload=13247]: Waiting for error group managing segments... Aug 7 22:26:34.495787031 (12h46m34.085132018s) [upload=13247]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:34.495791108 (12h46m34.085136095s) [upload=13247]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:34.495793463 (12h46m34.08513844s) [upload=13247]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:33.854971690 (12h47m33.444316667s) [upload=13247]: Done uploading segments... Aug 7 22:27:33.855000124 (12h47m33.444345111s) [upload=13247]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:47.578421960 (12h47m47.167766947s) [upload=13264]: Starting upload Aug 7 22:27:47.578474811 (12h47m47.167819798s) [upload=13264]: Done starting upload Aug 7 22:27:47.578529386 (12h47m47.167874363s) [upload=13264]: Uploading segments... Aug 7 22:27:47.578558632 (12h47m47.167903619s) [upload=13264]: Got next segment. Inline: true Aug 7 22:27:47.578561968 (12h47m47.167906945s) [upload=13264]: Waiting for error group managing segments... Aug 7 22:27:47.578595061 (12h47m47.167940048s) [upload=13264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:47.578599049 (12h47m47.167944036s) [upload=13264]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:47.578601603 (12h47m47.16794659s) [upload=13264]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:47.846848491 (12h47m47.436193478s) [upload=13264]: Done uploading segments... Aug 7 22:27:47.846867718 (12h47m47.436212695s) [upload=13264]: Upload finished. err: <nil> Aug 7 22:28:34.302216720 (12h48m33.891561707s) [upload=13274]: Starting upload Aug 7 22:28:34.302270392 (12h48m33.891615379s) [upload=13274]: Done starting upload Aug 7 22:28:34.302346498 (12h48m33.891691485s) [upload=13274]: Uploading segments... Aug 7 22:28:34.302375654 (12h48m33.891720641s) [upload=13274]: Got next segment. Inline: true Aug 7 22:28:34.302379070 (12h48m33.891724057s) [upload=13274]: Waiting for error group managing segments... Aug 7 22:28:34.302409759 (12h48m33.891754746s) [upload=13274]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:34.302413246 (12h48m33.891758234s) [upload=13274]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:34.302415511 (12h48m33.891760488s) [upload=13274]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:29.866705738 (12h49m29.456050725s) [upload=13274]: Done uploading segments... Aug 7 22:29:29.866723503 (12h49m29.45606849s) [upload=13274]: Upload finished. err: <nil> Duration: 1.915389478s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [........................XXXXXXXXXXXXXXXXX...................] duration: 6.292438562s FAIL (2025-08-07 22:43:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :32.805219872 (12h43m32.394564859s) [upload=13200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:32.805223589 (12h43m32.394568576s) [upload=13200]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:24:02.160684947 (12h44m1.750029944s) [upload=13200]: Next segment err: context deadline exceeded Aug 7 22:24:02.160927551 (12h44m1.750272538s) [upload=13200]: Done uploading segments... Aug 7 22:24:02.160942019 (12h44m1.750287007s) [upload=13200]: Upload finished. err: context deadline exceeded Aug 7 22:24:12.500861314 (12h44m12.090206301s) [upload=13211]: Starting upload Aug 7 22:24:12.500912582 (12h44m12.090257569s) [upload=13211]: Done starting upload Aug 7 22:24:12.500973068 (12h44m12.090318045s) [upload=13211]: Uploading segments... Aug 7 22:24:12.501078069 (12h44m12.090423056s) [upload=13211]: Got next segment. Inline: false Aug 7 22:24:12.501106834 (12h44m12.090451821s) [upload=13211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:12.501110571 (12h44m12.090455558s) [upload=13211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:24:25.371267525 (12h44m24.960612512s) [seg_pos={0 0}, upload=13211]: Begin upload segment... Aug 7 22:24:25.371330936 (12h44m24.960675953s) [seg_pos={0 0}, upload=13211]: Done begin upload segment. Aug 7 22:24:25.371383426 (12h44m24.960728413s) [noise=true , address=77.90.31.14:4661 , piece_id=73RH6546ZSA46KNTPMI3YPHM65Y3DJG5PCAAPGYJJFXDQ6KM5MRA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.371396932 (12h44m24.96074194s) [noise=true , address=77.90.31.19:4841 , piece_id=GPGSM3FHS76WAHMKYCXKIU57NEENBDZFAMIBSFP6JWV5K33BDCOQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.371863044 (12h44m24.961208021s) [noise=true , address=45.91.116.9:4261 , piece_id=F3JBOLQO76A5WQKYNQAKPAWVWZO5U5EYX7ZZIKQMOAUZUQCRZ7GA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372024263 (12h44m24.96136925s) [noise=true , address=45.91.116.12:4225 , piece_id=47OO6QUMLHWH54IAXRLTQEN56Z5ITGXCGKSTQFGSHTF32X3Z7HSQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372219627 (12h44m24.961564614s) [noise=true , address=104.36.82.2:4729 , piece_id=JYG3SHRCFAE37242ADYQA2CXJRRGO5APTOBZ5725GMZ2FRMPYLKA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372406654 (12h44m24.961751641s) [noise=true , address=45.91.116.7:4009 , piece_id=N25P6VDHR6RAW44ZNMAPNYJUX4AUV2UI3ROEX67KTWLK7RRB4K7Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372852217 (12h44m24.962197204s) [noise=true , address=45.91.116.6:4973 , piece_id=WLZJLHSER36K377KZCHHCLWNANIVKGHVKF2I3FBC5UDDLSIUCHFQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373233576 (12h44m24.962578563s) [noise=true , address=45.91.116.4:4937 , piece_id=ALO2UUBMXABZGWARBYTKGYY5DHLJEGULYGIZ7E4JQCYNFGP4JG7Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373581912 (12h44m24.9629269s) [noise=true , address=45.91.116.5:4299 , piece_id=LMJE5NXY2LTDFI3BZY2YGMAXFURMI5NOSZWW2ABZK4NFKQK5JROQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373901074 (12h44m24.963246061s) [noise=true , address=104.36.82.11:4113 , piece_id=7UYIPJZWWCEIMHEOPGC3DKXXTULAHTNF5QMPJJYIHXTKPSDOMM5A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374042614 (12h44m24.963387601s) [noise=true , address=77.90.31.17:4772 , piece_id=UEGCYD3SF4UF3CHRT5GEAZ6B77VKMG7S7PMSSRLZT6IP4ZKOTHMA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374483207 (12h44m24.963828194s) [noise=true , address=104.36.82.7:4912 , piece_id=H6DSVHM5ZX2GWEPHGRDH2WEDRHCLEYLZ62ISEHBLEITT4NGNDT7A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374810343 (12h44m24.96415533s) [noise=true , address=45.91.116.4:4939 , piece_id=S4PTGHW7FTLW5Q7W2VQZHLAI4YQZ25U35J7LUYSS5EXK552GUE2A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375117380 (12h44m24.964462368s) [noise=true , address=77.90.31.11:4555 , piece_id=CTQZTDFEGF756WLYI6U6NGMGBLFOKXZX3QWTQE7KBAITW43WVIJA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375526834 (12h44m24.964871821s) [noise=true , address=45.91.116.10:4082 , piece_id=4QZEPLP5I47FAOHIYQWJEBODVUEWTM25IQ652LDWLFZERCPTEXHA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375955474 (12h44m24.965300461s) [noise=true , address=104.36.82.10:4078 , piece_id=QKGPABRCDU26EDHDNRMCHMUQ3XTOQSN47WG5RKF6S4LD7OG2HY6Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:27.072833335 (12h44m26.662178332s) [upload=13211]: Next returned nil segment Aug 7 22:24:27.072902037 (12h44m26.662247034s) [upload=13211]: Waiting for error group managing segments... Aug 7 22:24:27.436037535 (12h44m27.025382522s) [noise=true , address=104.36.82.7:4912 , piece_id=H6DSVHM5ZX2GWEPHGRDH2WEDRHCLEYLZ62ISEHBLEITT4NGNDT7A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.446916249 (12h44m27.036261236s) [noise=true , address=104.36.82.2:4729 , piece_id=JYG3SHRCFAE37242ADYQA2CXJRRGO5APTOBZ5725GMZ2FRMPYLKA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.566730713 (12h44m27.15607569s) [noise=true , address=45.91.116.9:4261 , piece_id=F3JBOLQO76A5WQKYNQAKPAWVWZO5U5EYX7ZZIKQMOAUZUQCRZ7GA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.594802733 (12h44m27.18414772s) [noise=true , address=45.91.116.4:4937 , piece_id=ALO2UUBMXABZGWARBYTKGYY5DHLJEGULYGIZ7E4JQCYNFGP4JG7Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.613347363 (12h44m27.20269235s) [noise=true , address=77.90.31.14:4661 , piece_id=73RH6546ZSA46KNTPMI3YPHM65Y3DJG5PCAAPGYJJFXDQ6KM5MRA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.623084653 (12h44m27.21242964s) [noise=true , address=45.91.116.10:4082 , piece_id=4QZEPLP5I47FAOHIYQWJEBODVUEWTM25IQ652LDWLFZERCPTEXHA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.661186031 (12h44m27.250531018s) [noise=true , address=77.90.31.19:4841 , piece_id=GPGSM3FHS76WAHMKYCXKIU57NEENBDZFAMIBSFP6JWV5K33BDCOQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.698198064 (12h44m27.287543051s) [noise=true , address=45.91.116.6:4973 , piece_id=WLZJLHSER36K377KZCHHCLWNANIVKGHVKF2I3FBC5UDDLSIUCHFQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.724830297 (12h44m27.314175285s) [noise=true , address=45.91.116.4:4939 , piece_id=S4PTGHW7FTLW5Q7W2VQZHLAI4YQZ25U35J7LUYSS5EXK552GUE2A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.734317339 (12h44m27.323662326s) [noise=true , address=45.91.116.5:4299 , piece_id=LMJE5NXY2LTDFI3BZY2YGMAXFURMI5NOSZWW2ABZK4NFKQK5JROQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.908078026 (12h44m27.497423013s) [noise=true , address=77.90.31.17:4772 , piece_id=UEGCYD3SF4UF3CHRT5GEAZ6B77VKMG7S7PMSSRLZT6IP4ZKOTHMA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.985050925 (12h44m27.574395912s) [noise=true , address=45.91.116.7:4009 , piece_id=N25P6VDHR6RAW44ZNMAPNYJUX4AUV2UI3ROEX67KTWLK7RRB4K7Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:28.048748935 (12h44m27.638093922s) [noise=true , address=77.90.31.11:4555 , piece_id=CTQZTDFEGF756WLYI6U6NGMGBLFOKXZX3QWTQE7KBAITW43WVIJA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:28.048775115 (12h44m27.638120102s) [seg_pos={0 0}, upload=13211]: Segment reached optimal threshold of 13 pieces. Aug 7 22:24:28.048900375 (12h44m27.638245362s) [noise=true , address=45.91.116.12:4225 , piece_id=47OO6QUMLHWH54IAXRLTQEN56Z5ITGXCGKSTQFGSHTF32X3Z7HSQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Aug 7 22:24:28.049006828 (12h44m27.638351815s) [noise=true , address=104.36.82.11:4113 , piece_id=7UYIPJZWWCEIMHEOPGC3DKXXTULAHTNF5QMPJJYIHXTKPSDOMM5A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection; write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection Aug 7 22:24:28.049050902 (12h44m27.638395889s) [noise=true , address=104.36.82.10:4078 , piece_id=QKGPABRCDU26EDHDNRMCHMUQ3XTOQSN47WG5RKF6S4LD7OG2HY6Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection; write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection Aug 7 22:24:28.049061182 (12h44m27.638406169s) [seg_pos={0 0}, upload=13211]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 22:24:28.049126247 (12h44m27.638471224s) [upload=13211]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:24:28.049130154 (12h44m27.638475141s) [upload=13211]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:23.601003390 (12h47m23.190348377s) [upload=13211]: Done uploading segments... Aug 7 22:27:23.601027807 (12h47m23.190372794s) [upload=13211]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.292461878s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..XX..X..............................XXX....................] duration: 5.8000227s FAIL (2025-08-07 22:42:59 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.800049191s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [.......................XXXXXXXXXXXXXXXXX.X..................] duration: 5.618447802s FAIL (2025-08-07 22:42:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.618466358s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 21 out of 60 is failed (threshold=3) [.................XXXXXXXXXXXXXXXXXXX.X..X...................] duration: 1.905653042s FAIL (2025-08-07 22:42:51 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.905674443s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............X..............X.........X....X................] duration: 6.606195226s FAIL (2025-08-07 22:42:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== ding piece... Aug 7 22:30:27.106182415 (12h50m26.695527402s) [noise=true , address=64.49.13.137:4494 , piece_id=LFDJNPET6CVU2GRXM2KJ3LGNJPTGJ5EM4PWPHF3XVYMPHS366DZA, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.106447803 (12h50m26.69579278s) [noise=true , address=104.36.82.61:4404 , piece_id=ZLDF3FSQTOGUD4OMZYNQZZ6MUTVUF6NICTZBHRPBUM5OR3SDYVIA, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.106779869 (12h50m26.696124856s) [noise=true , address=104.36.82.52:4114 , piece_id=C5IZX5N3O4JEE2RDMVYZGSOWXVKQSGBJD67NOUNFT4LHMRNZL37A, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.107212025 (12h50m26.696557012s) [noise=true , address=104.36.82.9:4033 , piece_id=WDDQWPVVTQKSC4JUHUXZ27G6GJXGGBCXBBSQMVEX7ONSCLEFFTAA, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.108229842 (12h50m26.697574829s) [noise=true , address=104.36.82.37:4684 , piece_id=3VZGRTMMP3FQSDOTGNTVLJ43IHMA5FSUBOZ4P2SKWCYG5COVF7LQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.108480002 (12h50m26.697824989s) [noise=true , address=104.36.82.44:4884 , piece_id=HRBXMCCRYU4FBS73Q2IB5BK5BUUHAALRNODQ5XERZICLQ6PUVOBQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.109071673 (12h50m26.69841666s) [noise=true , address=45.91.116.13:4160 , piece_id=UGSRSWK3RVAIVHVFWBHMUGHTZLZNCJN3DSSJTBF3EP3DY4UHMG5A, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.109303346 (12h50m26.698648333s) [noise=true , address=104.36.82.46:4504 , piece_id=W7DK423UD6SXLWL3LVYWLGTCFNUDATLWKEMQIPYXX4WZ555GD34A, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.109532515 (12h50m26.698877492s) [noise=true , address=79.127.201.215:4356 , piece_id=FUQGELP6V2FJZ6PVPVKMYKWEAY25P7OAYA5ORTS26FL4FMGAF7XQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.109694265 (12h50m26.699039252s) [noise=true , address=79.127.205.235:4659 , piece_id=PDPXLV3GK4XVZTDBS2GC564GUEVII3XLOI54ALEUL2YWMBDFUNNQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.109942329 (12h50m26.699287316s) [noise=true , address=79.127.205.234:4588 , piece_id=SCZQTO3IY6647Q6MRG3J2AMSWJAWIYKPTX2S2YQHIENXEFZ3C53A, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.110167320 (12h50m26.699512307s) [noise=true , address=104.36.82.34:4576 , piece_id=G3A3X2KHUXIYXEF4FODXHPISXNSWNTLTP5RIXXPIWGMDWWEYBGUQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.110377551 (12h50m26.699722538s) [noise=true , address=104.36.82.16:4325 , piece_id=M4UMVR3RXMLSL3BI55AHRHPL46AAO6MU7V3V3GBZ7HQCR7WRZWSQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:27.110578186 (12h50m26.699923173s) [noise=true , address=104.36.82.35:4583 , piece_id=L4R2PO7KNIIOIN7BDCYLRCO2MRR7XUERKE4JA5LZ4AW6K37ZKBZQ, seg_pos={0 0}, upload=13303]: Uploading piece... Aug 7 22:30:28.206727754 (12h50m27.796072741s) [upload=13303]: Next returned nil segment Aug 7 22:30:28.206744677 (12h50m27.796089664s) [upload=13303]: Waiting for error group managing segments... Aug 7 22:30:28.490025109 (12h50m28.079370096s) [noise=true , address=79.127.205.235:4659 , piece_id=PDPXLV3GK4XVZTDBS2GC564GUEVII3XLOI54ALEUL2YWMBDFUNNQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:28.506048426 (12h50m28.095393403s) [noise=true , address=79.127.205.234:4588 , piece_id=SCZQTO3IY6647Q6MRG3J2AMSWJAWIYKPTX2S2YQHIENXEFZ3C53A, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:28.547644951 (12h50m28.136989938s) [noise=true , address=64.49.13.169:4709 , piece_id=AWEF3LXORD7TDQACIKTDS4NM5XHF676LYFTKKULKNCQ6HA6ABNLQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:28.617351799 (12h50m28.206696786s) [noise=true , address=64.49.13.137:4494 , piece_id=LFDJNPET6CVU2GRXM2KJ3LGNJPTGJ5EM4PWPHF3XVYMPHS366DZA, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.022989834 (12h50m28.612334841s) [noise=true , address=79.127.201.215:4356 , piece_id=FUQGELP6V2FJZ6PVPVKMYKWEAY25P7OAYA5ORTS26FL4FMGAF7XQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.123377683 (12h50m28.71272267s) [noise=true , address=104.36.82.62:4435 , piece_id=A6FYWW6YDPCOCWN3NYSVWHP6G3HW64V5PRS5PEMBFPVHILQOJSYA, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.128754752 (12h50m28.718099729s) [noise=true , address=104.36.82.58:4266 , piece_id=IHNIYP6QQ5HRNGMAHY3NNXQMJM6TOGP4JM5INFUQONEKONUD3BEA, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.473373880 (12h50m29.062718857s) [noise=true , address=104.36.82.52:4114 , piece_id=C5IZX5N3O4JEE2RDMVYZGSOWXVKQSGBJD67NOUNFT4LHMRNZL37A, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.539252935 (12h50m29.128597922s) [noise=true , address=104.36.82.34:4576 , piece_id=G3A3X2KHUXIYXEF4FODXHPISXNSWNTLTP5RIXXPIWGMDWWEYBGUQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.597507541 (12h50m29.186852528s) [noise=true , address=104.36.82.35:4583 , piece_id=L4R2PO7KNIIOIN7BDCYLRCO2MRR7XUERKE4JA5LZ4AW6K37ZKBZQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:29.641259711 (12h50m29.230604718s) [noise=true , address=45.91.116.13:4160 , piece_id=UGSRSWK3RVAIVHVFWBHMUGHTZLZNCJN3DSSJTBF3EP3DY4UHMG5A, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:30.117032753 (12h50m29.70637773s) [noise=true , address=104.36.82.46:4504 , piece_id=W7DK423UD6SXLWL3LVYWLGTCFNUDATLWKEMQIPYXX4WZ555GD34A, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:30.995539243 (12h50m30.58488423s) [noise=true , address=104.36.82.37:4684 , piece_id=3VZGRTMMP3FQSDOTGNTVLJ43IHMA5FSUBOZ4P2SKWCYG5COVF7LQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:31.436214934 (12h50m31.025559921s) [noise=true , address=104.36.82.61:4404 , piece_id=ZLDF3FSQTOGUD4OMZYNQZZ6MUTVUF6NICTZBHRPBUM5OR3SDYVIA, seg_pos={0 0}, upload=13303]: Done uploading piece. err: <nil> Aug 7 22:30:31.436243469 (12h50m31.025588456s) [seg_pos={0 0}, upload=13303]: Segment reached optimal threshold of 14 pieces. Aug 7 22:30:31.436426830 (12h50m31.025771817s) [noise=true , address=104.36.82.16:4325 , piece_id=M4UMVR3RXMLSL3BI55AHRHPL46AAO6MU7V3V3GBZ7HQCR7WRZWSQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aTYNxPbXJGM29Xqovc5VK4cun64wr2DoduaVCvKDj38xCLGKJ): protocol: expected piece hash; context canceled; EOF Aug 7 22:30:31.436556607 (12h50m31.025901594s) [noise=true , address=104.36.82.9:4033 , piece_id=WDDQWPVVTQKSC4JUHUXZ27G6GJXGGBCXBBSQMVEX7ONSCLEFFTAA, seg_pos={0 0}, upload=13303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C): protocol: write tcp 188.166.225.170:49338->104.36.82.9:4033: use of closed network connection; write tcp 188.166.225.170:49338->104.36.82.9:4033: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49338->104.36.82.9:4033: use of closed network connection Aug 7 22:30:31.436606383 (12h50m31.02595137s) [noise=true , address=104.36.82.44:4884 , piece_id=HRBXMCCRYU4FBS73Q2IB5BK5BUUHAALRNODQ5XERZICLQ6PUVOBQ, seg_pos={0 0}, upload=13303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qGXdnm83P4eMnCtZdRt2x5yAWP7pQafWwoPG72daZJVR9pyKG): protocol: write tcp 188.166.225.170:45006->104.36.82.44:4884: use of closed network connection; write tcp 188.166.225.170:45006->104.36.82.44:4884: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45006->104.36.82.44:4884: use of closed network connection Aug 7 22:30:31.436613496 (12h50m31.025958483s) [noise=true , address=104.36.82.5:4853 , piece_id=FZOYMSE2HPKJUXG7RXFQH77H7GTC34UFQVKAXJBKQ4YYI7ICAKMA, seg_pos={0 0}, upload=13303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u): protocol: write tcp 188.166.225.170:54220->104.36.82.5:4853: use of closed network connection; write tcp 188.166.225.170:54220->104.36.82.5:4853: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54220->104.36.82.5:4853: use of closed network connection Aug 7 22:30:31.436624858 (12h50m31.025969845s) [seg_pos={0 0}, upload=13303]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:30:31.436720120 (12h50m31.026065107s) [upload=13303]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:30:31.436724318 (12h50m31.026069295s) [upload=13303]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:30:31.748725210 (12h50m31.338070187s) [upload=13303]: Done uploading segments... Aug 7 22:30:31.748742002 (12h50m31.338086989s) [upload=13303]: Upload finished. err: <nil> Aug 7 22:31:18.785686607 (12h51m18.375031594s) [upload=13318]: Starting upload Aug 7 22:31:18.785729449 (12h51m18.375074437s) [upload=13318]: Done starting upload Aug 7 22:31:18.785814212 (12h51m18.375159189s) [upload=13318]: Uploading segments... Aug 7 22:31:18.785960702 (12h51m18.375305679s) [upload=13318]: Got next segment. Inline: false Aug 7 22:31:18.785988354 (12h51m18.375333341s) [upload=13318]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:31:18.785993003 (12h51m18.37533798s) [upload=13318]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:31:19.001950809 (12h51m18.591295786s) [upload=13318]: Next segment err: metaclient: Too Many Requests Aug 7 22:31:19.001977260 (12h51m18.591322247s) [upload=13318]: Done uploading segments... Aug 7 22:31:19.001992078 (12h51m18.591337055s) [upload=13318]: Upload finished. err: metaclient: Too Many Requests Duration: 6.606232398s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 231.163959ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 22:42:45 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 231.219465ms edge gateway vhost bgp us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 24 out of 60 is failed (threshold=3) [.................XXXXXXXXXXXXXXXXXXX.XXX.X.X................] duration: 2.340288784s FAIL (2025-08-07 22:42:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.34030813s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X..XX..X..............................XXX...................] duration: 5.924714951s FAIL (2025-08-07 22:42:33 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.924739809s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.........................XXXXXXXXXXXXXXXXX..................] duration: 6.499752414s FAIL (2025-08-07 22:42:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :32.805219872 (12h43m32.394564859s) [upload=13200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:32.805223589 (12h43m32.394568576s) [upload=13200]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:24:02.160684947 (12h44m1.750029944s) [upload=13200]: Next segment err: context deadline exceeded Aug 7 22:24:02.160927551 (12h44m1.750272538s) [upload=13200]: Done uploading segments... Aug 7 22:24:02.160942019 (12h44m1.750287007s) [upload=13200]: Upload finished. err: context deadline exceeded Aug 7 22:24:12.500861314 (12h44m12.090206301s) [upload=13211]: Starting upload Aug 7 22:24:12.500912582 (12h44m12.090257569s) [upload=13211]: Done starting upload Aug 7 22:24:12.500973068 (12h44m12.090318045s) [upload=13211]: Uploading segments... Aug 7 22:24:12.501078069 (12h44m12.090423056s) [upload=13211]: Got next segment. Inline: false Aug 7 22:24:12.501106834 (12h44m12.090451821s) [upload=13211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:12.501110571 (12h44m12.090455558s) [upload=13211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:24:25.371267525 (12h44m24.960612512s) [seg_pos={0 0}, upload=13211]: Begin upload segment... Aug 7 22:24:25.371330936 (12h44m24.960675953s) [seg_pos={0 0}, upload=13211]: Done begin upload segment. Aug 7 22:24:25.371383426 (12h44m24.960728413s) [noise=true , address=77.90.31.14:4661 , piece_id=73RH6546ZSA46KNTPMI3YPHM65Y3DJG5PCAAPGYJJFXDQ6KM5MRA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.371396932 (12h44m24.96074194s) [noise=true , address=77.90.31.19:4841 , piece_id=GPGSM3FHS76WAHMKYCXKIU57NEENBDZFAMIBSFP6JWV5K33BDCOQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.371863044 (12h44m24.961208021s) [noise=true , address=45.91.116.9:4261 , piece_id=F3JBOLQO76A5WQKYNQAKPAWVWZO5U5EYX7ZZIKQMOAUZUQCRZ7GA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372024263 (12h44m24.96136925s) [noise=true , address=45.91.116.12:4225 , piece_id=47OO6QUMLHWH54IAXRLTQEN56Z5ITGXCGKSTQFGSHTF32X3Z7HSQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372219627 (12h44m24.961564614s) [noise=true , address=104.36.82.2:4729 , piece_id=JYG3SHRCFAE37242ADYQA2CXJRRGO5APTOBZ5725GMZ2FRMPYLKA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372406654 (12h44m24.961751641s) [noise=true , address=45.91.116.7:4009 , piece_id=N25P6VDHR6RAW44ZNMAPNYJUX4AUV2UI3ROEX67KTWLK7RRB4K7Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.372852217 (12h44m24.962197204s) [noise=true , address=45.91.116.6:4973 , piece_id=WLZJLHSER36K377KZCHHCLWNANIVKGHVKF2I3FBC5UDDLSIUCHFQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373233576 (12h44m24.962578563s) [noise=true , address=45.91.116.4:4937 , piece_id=ALO2UUBMXABZGWARBYTKGYY5DHLJEGULYGIZ7E4JQCYNFGP4JG7Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373581912 (12h44m24.9629269s) [noise=true , address=45.91.116.5:4299 , piece_id=LMJE5NXY2LTDFI3BZY2YGMAXFURMI5NOSZWW2ABZK4NFKQK5JROQ, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.373901074 (12h44m24.963246061s) [noise=true , address=104.36.82.11:4113 , piece_id=7UYIPJZWWCEIMHEOPGC3DKXXTULAHTNF5QMPJJYIHXTKPSDOMM5A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374042614 (12h44m24.963387601s) [noise=true , address=77.90.31.17:4772 , piece_id=UEGCYD3SF4UF3CHRT5GEAZ6B77VKMG7S7PMSSRLZT6IP4ZKOTHMA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374483207 (12h44m24.963828194s) [noise=true , address=104.36.82.7:4912 , piece_id=H6DSVHM5ZX2GWEPHGRDH2WEDRHCLEYLZ62ISEHBLEITT4NGNDT7A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.374810343 (12h44m24.96415533s) [noise=true , address=45.91.116.4:4939 , piece_id=S4PTGHW7FTLW5Q7W2VQZHLAI4YQZ25U35J7LUYSS5EXK552GUE2A, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375117380 (12h44m24.964462368s) [noise=true , address=77.90.31.11:4555 , piece_id=CTQZTDFEGF756WLYI6U6NGMGBLFOKXZX3QWTQE7KBAITW43WVIJA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375526834 (12h44m24.964871821s) [noise=true , address=45.91.116.10:4082 , piece_id=4QZEPLP5I47FAOHIYQWJEBODVUEWTM25IQ652LDWLFZERCPTEXHA, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:25.375955474 (12h44m24.965300461s) [noise=true , address=104.36.82.10:4078 , piece_id=QKGPABRCDU26EDHDNRMCHMUQ3XTOQSN47WG5RKF6S4LD7OG2HY6Q, seg_pos={0 0}, upload=13211]: Uploading piece... Aug 7 22:24:27.072833335 (12h44m26.662178332s) [upload=13211]: Next returned nil segment Aug 7 22:24:27.072902037 (12h44m26.662247034s) [upload=13211]: Waiting for error group managing segments... Aug 7 22:24:27.436037535 (12h44m27.025382522s) [noise=true , address=104.36.82.7:4912 , piece_id=H6DSVHM5ZX2GWEPHGRDH2WEDRHCLEYLZ62ISEHBLEITT4NGNDT7A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.446916249 (12h44m27.036261236s) [noise=true , address=104.36.82.2:4729 , piece_id=JYG3SHRCFAE37242ADYQA2CXJRRGO5APTOBZ5725GMZ2FRMPYLKA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.566730713 (12h44m27.15607569s) [noise=true , address=45.91.116.9:4261 , piece_id=F3JBOLQO76A5WQKYNQAKPAWVWZO5U5EYX7ZZIKQMOAUZUQCRZ7GA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.594802733 (12h44m27.18414772s) [noise=true , address=45.91.116.4:4937 , piece_id=ALO2UUBMXABZGWARBYTKGYY5DHLJEGULYGIZ7E4JQCYNFGP4JG7Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.613347363 (12h44m27.20269235s) [noise=true , address=77.90.31.14:4661 , piece_id=73RH6546ZSA46KNTPMI3YPHM65Y3DJG5PCAAPGYJJFXDQ6KM5MRA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.623084653 (12h44m27.21242964s) [noise=true , address=45.91.116.10:4082 , piece_id=4QZEPLP5I47FAOHIYQWJEBODVUEWTM25IQ652LDWLFZERCPTEXHA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.661186031 (12h44m27.250531018s) [noise=true , address=77.90.31.19:4841 , piece_id=GPGSM3FHS76WAHMKYCXKIU57NEENBDZFAMIBSFP6JWV5K33BDCOQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.698198064 (12h44m27.287543051s) [noise=true , address=45.91.116.6:4973 , piece_id=WLZJLHSER36K377KZCHHCLWNANIVKGHVKF2I3FBC5UDDLSIUCHFQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.724830297 (12h44m27.314175285s) [noise=true , address=45.91.116.4:4939 , piece_id=S4PTGHW7FTLW5Q7W2VQZHLAI4YQZ25U35J7LUYSS5EXK552GUE2A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.734317339 (12h44m27.323662326s) [noise=true , address=45.91.116.5:4299 , piece_id=LMJE5NXY2LTDFI3BZY2YGMAXFURMI5NOSZWW2ABZK4NFKQK5JROQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.908078026 (12h44m27.497423013s) [noise=true , address=77.90.31.17:4772 , piece_id=UEGCYD3SF4UF3CHRT5GEAZ6B77VKMG7S7PMSSRLZT6IP4ZKOTHMA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:27.985050925 (12h44m27.574395912s) [noise=true , address=45.91.116.7:4009 , piece_id=N25P6VDHR6RAW44ZNMAPNYJUX4AUV2UI3ROEX67KTWLK7RRB4K7Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:28.048748935 (12h44m27.638093922s) [noise=true , address=77.90.31.11:4555 , piece_id=CTQZTDFEGF756WLYI6U6NGMGBLFOKXZX3QWTQE7KBAITW43WVIJA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil> Aug 7 22:24:28.048775115 (12h44m27.638120102s) [seg_pos={0 0}, upload=13211]: Segment reached optimal threshold of 13 pieces. Aug 7 22:24:28.048900375 (12h44m27.638245362s) [noise=true , address=45.91.116.12:4225 , piece_id=47OO6QUMLHWH54IAXRLTQEN56Z5ITGXCGKSTQFGSHTF32X3Z7HSQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Aug 7 22:24:28.049006828 (12h44m27.638351815s) [noise=true , address=104.36.82.11:4113 , piece_id=7UYIPJZWWCEIMHEOPGC3DKXXTULAHTNF5QMPJJYIHXTKPSDOMM5A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection; write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection Aug 7 22:24:28.049050902 (12h44m27.638395889s) [noise=true , address=104.36.82.10:4078 , piece_id=QKGPABRCDU26EDHDNRMCHMUQ3XTOQSN47WG5RKF6S4LD7OG2HY6Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection; write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection Aug 7 22:24:28.049061182 (12h44m27.638406169s) [seg_pos={0 0}, upload=13211]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 22:24:28.049126247 (12h44m27.638471224s) [upload=13211]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:24:28.049130154 (12h44m27.638475141s) [upload=13211]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:23.601003390 (12h47m23.190348377s) [upload=13211]: Done uploading segments... Aug 7 22:27:23.601027807 (12h47m23.190372794s) [upload=13211]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.499776329s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 11 out of 60 is failed (threshold=3) [...............................XXXXXXXXX..X.X...............] duration: 6.612346286s FAIL (2025-08-07 22:42:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=13236]: Begin upload segment... Aug 7 22:25:52.992307080 (12h45m52.581652067s) [seg_pos={0 0}, upload=13236]: Done begin upload segment. Aug 7 22:25:52.992331798 (12h45m52.581676775s) [noise=true , address=104.36.82.13:4213 , piece_id=5W5FD4WMMSZ7BKHFQI4GYOODXZ2P2MMAIQNK6CD6XILH22C53JGQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992683770 (12h45m52.582028757s) [noise=true , address=104.36.82.9:4024 , piece_id=VBFOHRWDKWRY2774KJVK3JXJSMG4CPVDBBECCSZVJHR4274KLRMA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992785475 (12h45m52.582130462s) [noise=true , address=104.36.82.22:4527 , piece_id=JBNJOSETDAB6VG7PGVXIBUP6GPF2FBPJURBZNBR3BJOGMRK3EIQA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.992942396 (12h45m52.582287373s) [noise=true , address=45.91.116.11:4124 , piece_id=TLATR2WNDVRR3WWTT3FZARUJ2QWWQFZBDYXLZ7PUPFOAEGJI4BTQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993132439 (12h45m52.582477457s) [noise=true , address=104.36.82.5:4855 , piece_id=6EW7GNJ6KVWKNFHGFZVPEYPD4TDIVVZUXRW6AGJ3IGVANEAHYDUA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993337862 (12h45m52.582682849s) [noise=true , address=64.49.13.170:4717 , piece_id=52FN65VRQ24LBOGHJ22EQQM3MNI3EB2UXB57P5LD5GPLSABNS22Q, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993511144 (12h45m52.582856131s) [noise=true , address=104.36.82.60:4361 , piece_id=EPWXHNXPVAUX75FUAXY57QZDKRXIEDHRXFL6PEBQGXY2A3Q4KNHA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993715204 (12h45m52.583060191s) [noise=true , address=104.36.82.58:4281 , piece_id=5UH27M25HE2LEZJ53H7QBKU6I6N2JE6F4PZTLOE6XP3GEDCZLEEQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.993820806 (12h45m52.583165784s) [noise=true , address=45.91.116.9:4284 , piece_id=LT3RP2EBVHIFIMAIGM3DW6QMPNLKZCK5FD4GQEQI6BFWYD56ZUBQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994068250 (12h45m52.583413237s) [noise=true , address=79.127.205.233:4633 , piece_id=LBO5XLAXBRDURMDMN6MZ4HLVSG4TTUZPWCFTOTB6QMWAS5IIRLBA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994092416 (12h45m52.583437393s) [noise=true , address=104.36.82.60:4346 , piece_id=3NU3QBO5DAW5ZB7NTAHWP4C7RE4BYDPHZ77JOUQT3R3CBTOMONLA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994295394 (12h45m52.583640371s) [noise=true , address=45.91.116.7:4037 , piece_id=2NDGWBVYO3D2RYJ57QSXT632TT24SMRNWA3YZ2YW2BN6TBBLMRQA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994307056 (12h45m52.583652033s) [noise=true , address=104.36.82.46:4505 , piece_id=A7XPGAYUTNB26A6DJQNQ4NZ4FG27FUBQNEH5AQ7IFLYR3DLGG6SQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994547126 (12h45m52.583892103s) [noise=true , address=104.36.82.61:4389 , piece_id=WBB2Y7YYCEML23D3CKJHGIEOUELRTVG3NXVGI4JDD7XPNQ4STENQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.994782436 (12h45m52.584127423s) [noise=true , address=64.49.13.154:4934 , piece_id=LDQTK4SFJPTL7NAY5XXKBIFCVLNRSLHLACTJ65MWCHFSBV4EKJYA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998117076 (12h45m52.587462053s) [noise=true , address=104.36.82.56:4252 , piece_id=XQWAMPOYTVGKYEHIEEEIDDST5BOIN2XCT7B3SFUR4U7PQBROEELQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998596283 (12h45m52.58794127s) [noise=true , address=104.36.82.15:4261 , piece_id=C3RUC2F5LVRKH4MYTXDYOAP5KJGRZ6TAX57XULVBAKOS7ZK4QONQ, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:52.998917628 (12h45m52.588262615s) [noise=true , address=104.36.82.23:4473 , piece_id=N54OVKSO4KKX4NPORULBLAPVE5UT5RU6BBIEJCCMKG5553GVY6JA, seg_pos={0 0}, upload=13236]: Uploading piece... Aug 7 22:25:54.121532197 (12h45m53.710877184s) [upload=13236]: Next returned nil segment Aug 7 22:25:54.121549150 (12h45m53.710894137s) [upload=13236]: Waiting for error group managing segments... Aug 7 22:25:54.447812563 (12h45m54.03715755s) [noise=true , address=79.127.205.233:4633 , piece_id=LBO5XLAXBRDURMDMN6MZ4HLVSG4TTUZPWCFTOTB6QMWAS5IIRLBA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:54.450658769 (12h45m54.040003756s) [noise=true , address=64.49.13.170:4717 , piece_id=52FN65VRQ24LBOGHJ22EQQM3MNI3EB2UXB57P5LD5GPLSABNS22Q, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:54.505775361 (12h45m54.095120349s) [noise=true , address=64.49.13.154:4934 , piece_id=LDQTK4SFJPTL7NAY5XXKBIFCVLNRSLHLACTJ65MWCHFSBV4EKJYA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.074032067 (12h45m54.663377054s) [noise=true , address=104.36.82.60:4346 , piece_id=3NU3QBO5DAW5ZB7NTAHWP4C7RE4BYDPHZ77JOUQT3R3CBTOMONLA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.160612428 (12h45m54.749957415s) [noise=true , address=104.36.82.61:4389 , piece_id=WBB2Y7YYCEML23D3CKJHGIEOUELRTVG3NXVGI4JDD7XPNQ4STENQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.161069252 (12h45m54.750414239s) [noise=true , address=104.36.82.22:4527 , piece_id=JBNJOSETDAB6VG7PGVXIBUP6GPF2FBPJURBZNBR3BJOGMRK3EIQA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.172460447 (12h45m54.761805434s) [noise=true , address=104.36.82.60:4361 , piece_id=EPWXHNXPVAUX75FUAXY57QZDKRXIEDHRXFL6PEBQGXY2A3Q4KNHA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.217919931 (12h45m54.807264908s) [noise=true , address=104.36.82.56:4252 , piece_id=XQWAMPOYTVGKYEHIEEEIDDST5BOIN2XCT7B3SFUR4U7PQBROEELQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.226637230 (12h45m54.815982217s) [noise=true , address=104.36.82.58:4281 , piece_id=5UH27M25HE2LEZJ53H7QBKU6I6N2JE6F4PZTLOE6XP3GEDCZLEEQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.271492068 (12h45m54.860837055s) [noise=true , address=45.91.116.11:4124 , piece_id=TLATR2WNDVRR3WWTT3FZARUJ2QWWQFZBDYXLZ7PUPFOAEGJI4BTQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.281497411 (12h45m54.870842388s) [noise=true , address=45.91.116.9:4284 , piece_id=LT3RP2EBVHIFIMAIGM3DW6QMPNLKZCK5FD4GQEQI6BFWYD56ZUBQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.402216807 (12h45m54.991561794s) [noise=true , address=104.36.82.9:4024 , piece_id=VBFOHRWDKWRY2774KJVK3JXJSMG4CPVDBBECCSZVJHR4274KLRMA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:55.632857570 (12h45m55.222202558s) [noise=true , address=104.36.82.5:4855 , piece_id=6EW7GNJ6KVWKNFHGFZVPEYPD4TDIVVZUXRW6AGJ3IGVANEAHYDUA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:56.838313689 (12h45m56.427658676s) [noise=true , address=104.36.82.13:4213 , piece_id=5W5FD4WMMSZ7BKHFQI4GYOODXZ2P2MMAIQNK6CD6XILH22C53JGQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil> Aug 7 22:25:56.838338066 (12h45m56.427683053s) [seg_pos={0 0}, upload=13236]: Segment reached optimal threshold of 14 pieces. Aug 7 22:25:56.838585389 (12h45m56.427930366s) [noise=true , address=45.91.116.7:4037 , piece_id=2NDGWBVYO3D2RYJ57QSXT632TT24SMRNWA3YZ2YW2BN6TBBLMRQA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; write tcp 188.166.225.170:55570->45.91.116.7:4037: use of closed network connection; write tcp 188.166.225.170:55570->45.91.116.7:4037: use of closed network connection; EOF Aug 7 22:25:56.838630465 (12h45m56.427975452s) [noise=true , address=104.36.82.46:4505 , piece_id=A7XPGAYUTNB26A6DJQNQ4NZ4FG27FUBQNEH5AQ7IFLYR3DLGG6SQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf): protocol: write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection; write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection Aug 7 22:25:56.838682564 (12h45m56.428027551s) [noise=true , address=104.36.82.23:4473 , piece_id=N54OVKSO4KKX4NPORULBLAPVE5UT5RU6BBIEJCCMKG5553GVY6JA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC): protocol: write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection; write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection Aug 7 22:25:56.838697754 (12h45m56.428042731s) [noise=true , address=104.36.82.15:4261 , piece_id=C3RUC2F5LVRKH4MYTXDYOAP5KJGRZ6TAX57XULVBAKOS7ZK4QONQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i1jcJdEP7Ch9uaGcjDaVxpodx55dJEHUKFbcr1GeKQ9waWjYz): protocol: expected piece hash; context canceled; EOF Aug 7 22:25:56.838720558 (12h45m56.428065535s) [seg_pos={0 0}, upload=13236]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:25:56.838823714 (12h45m56.428168701s) [upload=13236]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:25:56.838828013 (12h45m56.428173s) [upload=13236]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:30:17.177125535 (12h50m16.766470522s) [upload=13236]: Done uploading segments... Aug 7 22:30:17.177153268 (12h50m16.766498255s) [upload=13236]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.612378629s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 21 out of 60 is failed (threshold=3) [..................XXXXXXXXXXXXXXXXXXX.X..X..................] duration: 1.856991747s FAIL (2025-08-07 22:42:29 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.857018869s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 19 out of 60 is failed (threshold=3) [.................XXXXXXXXXXXXXXXX.XX..X.....................] duration: 1.909022566s FAIL (2025-08-07 22:42:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7 22:22:45.818723517 (12h42m45.408068504s) [upload=13188]: Done starting upload Aug 7 22:22:45.818774856 (12h42m45.408119843s) [upload=13188]: Uploading segments... Aug 7 22:22:45.818807999 (12h42m45.408152996s) [upload=13188]: Got next segment. Inline: true Aug 7 22:22:45.818812908 (12h42m45.408157895s) [upload=13188]: Waiting for error group managing segments... Aug 7 22:22:45.818870779 (12h42m45.408215776s) [upload=13188]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:22:45.818879214 (12h42m45.408224202s) [upload=13188]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:22:45.818883494 (12h42m45.408228481s) [upload=13188]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:15.175828623 (12h43m14.76517361s) [upload=13188]: Done uploading segments... Aug 7 22:23:15.175844935 (12h43m14.765189912s) [upload=13188]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:23:16.350710694 (12h43m15.940055691s) [upload=13195]: Starting upload Aug 7 22:23:16.350768756 (12h43m15.940113733s) [upload=13195]: Done starting upload Aug 7 22:23:16.350814804 (12h43m15.940159791s) [upload=13195]: Uploading segments... Aug 7 22:23:16.350849079 (12h43m15.940194056s) [upload=13195]: Got next segment. Inline: true Aug 7 22:23:16.350852777 (12h43m15.940197764s) [upload=13195]: Waiting for error group managing segments... Aug 7 22:23:16.350884928 (12h43m15.940229915s) [upload=13195]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:23:16.350888735 (12h43m15.940233722s) [upload=13195]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:23:16.350891020 (12h43m15.940236007s) [upload=13195]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:23:45.695642688 (12h43m45.284987675s) [upload=13195]: Done uploading segments... Aug 7 22:23:45.695657867 (12h43m45.285002854s) [upload=13195]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:24:08.254775832 (12h44m7.844120819s) [upload=13210]: Starting upload Aug 7 22:24:08.254857087 (12h44m7.844202074s) [upload=13210]: Done starting upload Aug 7 22:24:08.254936559 (12h44m7.844281546s) [upload=13210]: Uploading segments... Aug 7 22:24:08.255031861 (12h44m7.844376858s) [upload=13210]: Got next segment. Inline: true Aug 7 22:24:08.255037032 (12h44m7.844382019s) [upload=13210]: Waiting for error group managing segments... Aug 7 22:24:08.255078220 (12h44m7.844423207s) [upload=13210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:24:08.255093450 (12h44m7.844438437s) [upload=13210]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:24:08.255096435 (12h44m7.844441422s) [upload=13210]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:24:37.608974587 (12h44m37.198319574s) [upload=13210]: Done uploading segments... Aug 7 22:24:37.609189158 (12h44m37.198534195s) [upload=13210]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:25:19.078840222 (12h45m18.668185229s) [upload=13227]: Starting upload Aug 7 22:25:19.078945333 (12h45m18.66829035s) [upload=13227]: Done starting upload Aug 7 22:25:19.079016760 (12h45m18.668361767s) [upload=13227]: Uploading segments... Aug 7 22:25:19.079058389 (12h45m18.668403377s) [upload=13227]: Got next segment. Inline: true Aug 7 22:25:19.079064030 (12h45m18.668409007s) [upload=13227]: Waiting for error group managing segments... Aug 7 22:25:19.079115388 (12h45m18.668460375s) [upload=13227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:25:19.079121570 (12h45m18.668466557s) [upload=13227]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:25:19.079126009 (12h45m18.668471077s) [upload=13227]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:25:48.426015265 (12h45m48.015360252s) [upload=13227]: Done uploading segments... Aug 7 22:25:48.426036394 (12h45m48.015381381s) [upload=13227]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:26:18.229360345 (12h46m17.818705332s) [upload=13243]: Starting upload Aug 7 22:26:18.229421673 (12h46m17.81876665s) [upload=13243]: Done starting upload Aug 7 22:26:18.229465397 (12h46m17.818810384s) [upload=13243]: Uploading segments... Aug 7 22:26:18.229495053 (12h46m17.81884004s) [upload=13243]: Got next segment. Inline: true Aug 7 22:26:18.229498680 (12h46m17.818843667s) [upload=13243]: Waiting for error group managing segments... Aug 7 22:26:18.229531593 (12h46m17.81887658s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:18.229535170 (12h46m17.818880157s) [upload=13243]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:18.229537614 (12h46m17.818882601s) [upload=13243]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:26:18.493923010 (12h46m18.083268007s) [upload=13243]: Done uploading segments... Aug 7 22:26:18.493948349 (12h46m18.083293346s) [upload=13243]: Upload finished. err: <nil> Aug 7 22:26:34.238583280 (12h46m33.827928267s) [upload=13246]: Starting upload Aug 7 22:26:34.238671890 (12h46m33.828016897s) [upload=13246]: Done starting upload Aug 7 22:26:34.238753585 (12h46m33.828098573s) [upload=13246]: Uploading segments... Aug 7 22:26:34.238795296 (12h46m33.828140283s) [upload=13246]: Got next segment. Inline: true Aug 7 22:26:34.238800546 (12h46m33.828145533s) [upload=13246]: Waiting for error group managing segments... Aug 7 22:26:34.238842697 (12h46m33.828187684s) [upload=13246]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:26:34.238847656 (12h46m33.828192633s) [upload=13246]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:26:34.238851193 (12h46m33.82819618s) [upload=13246]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:03.590020724 (12h47m3.179365711s) [upload=13246]: Done uploading segments... Aug 7 22:27:03.590046854 (12h47m3.179391861s) [upload=13246]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:27:08.455168905 (12h47m8.044513892s) [upload=13253]: Starting upload Aug 7 22:27:08.455224000 (12h47m8.044568977s) [upload=13253]: Done starting upload Aug 7 22:27:08.455266011 (12h47m8.044610998s) [upload=13253]: Uploading segments... Aug 7 22:27:08.455292903 (12h47m8.04463789s) [upload=13253]: Got next segment. Inline: true Aug 7 22:27:08.455296630 (12h47m8.044641607s) [upload=13253]: Waiting for error group managing segments... Aug 7 22:27:08.455327689 (12h47m8.044672676s) [upload=13253]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:27:08.455331626 (12h47m8.044676613s) [upload=13253]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:27:08.455334372 (12h47m8.044679349s) [upload=13253]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:27:37.813944606 (12h47m37.403289643s) [upload=13253]: Done uploading segments... Aug 7 22:27:37.813971608 (12h47m37.403316615s) [upload=13253]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:28:03.718747816 (12h48m3.308092803s) [upload=13266]: Starting upload Aug 7 22:28:03.718803663 (12h48m3.30814865s) [upload=13266]: Done starting upload Aug 7 22:28:03.718851525 (12h48m3.308196512s) [upload=13266]: Uploading segments... Aug 7 22:28:03.718879849 (12h48m3.308224836s) [upload=13266]: Got next segment. Inline: true Aug 7 22:28:03.718883686 (12h48m3.308228673s) [upload=13266]: Waiting for error group managing segments... Aug 7 22:28:03.718918023 (12h48m3.30826301s) [upload=13266]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:03.718922461 (12h48m3.308267448s) [upload=13266]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:03.718924966 (12h48m3.308269953s) [upload=13266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:04.001556505 (12h48m3.590901502s) [upload=13266]: Done uploading segments... Aug 7 22:28:04.001574169 (12h48m3.590919156s) [upload=13266]: Upload finished. err: <nil> Aug 7 22:28:53.653874975 (12h48m53.243219952s) [upload=13280]: Starting upload Aug 7 22:28:53.653923989 (12h48m53.243268976s) [upload=13280]: Done starting upload Aug 7 22:28:53.653970338 (12h48m53.243315315s) [upload=13280]: Uploading segments... Aug 7 22:28:53.654001658 (12h48m53.243346645s) [upload=13280]: Got next segment. Inline: true Aug 7 22:28:53.654005195 (12h48m53.243350182s) [upload=13280]: Waiting for error group managing segments... Aug 7 22:28:53.654037516 (12h48m53.243382503s) [upload=13280]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:28:53.654041253 (12h48m53.24338623s) [upload=13280]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:28:53.654043598 (12h48m53.243388575s) [upload=13280]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:28:53.931677747 (12h48m53.521022724s) [upload=13280]: Done uploading segments... Aug 7 22:28:53.931693927 (12h48m53.521038914s) [upload=13280]: Upload finished. err: <nil> Aug 7 22:29:13.069018260 (12h49m12.658363247s) [upload=13284]: Starting upload Aug 7 22:29:13.069075108 (12h49m12.658420096s) [upload=13284]: Done starting upload Aug 7 22:29:13.069124593 (12h49m12.65846958s) [upload=13284]: Uploading segments... Aug 7 22:29:13.069159800 (12h49m12.658504787s) [upload=13284]: Got next segment. Inline: true Aug 7 22:29:13.069163377 (12h49m12.658508364s) [upload=13284]: Waiting for error group managing segments... Aug 7 22:29:13.069195007 (12h49m12.658539995s) [upload=13284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:29:13.069198645 (12h49m12.658543632s) [upload=13284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:29:13.069201010 (12h49m12.658545997s) [upload=13284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:29:42.426836268 (12h49m42.016181256s) [upload=13284]: Done uploading segments... Aug 7 22:29:42.426899691 (12h49m42.016244678s) [upload=13284]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.909045361s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [........................XXXXXXXXXXXXXXXXX.X.................] duration: 5.518676861s FAIL (2025-08-07 22:42:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.518695538s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 639.921154ms target: https://us1.storj.io/login FAIL (2025-08-07 22:42:23 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 639.943998ms core satellite us1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)