Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 3.391060413s FAIL (2025-10-28 11:04:21 UTC) orbiter warning: context deadline exceeded Duration: 3.391095609s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 3.417798557s FAIL (2025-10-28 11:03:50 UTC) orbiter warning: context deadline exceeded Duration: 3.417823674s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 4.054145225s FAIL (2025-10-28 11:02:44 UTC) orbiter warning: context deadline exceeded Duration: 4.054187916s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 5.000591292s FAIL (2025-10-28 11:02:18 UTC) orbiter warning: context deadline exceeded Duration: 5.00063158s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.561472773s FAIL (2025-10-28 10:05:26 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 09:20:59.362633688 (18h38m16.711940974s) [upload=16414]: Starting upload Oct 28 09:20:59.362695796 (18h38m16.712003082s) [upload=16414]: Done starting upload Oct 28 09:20:59.362893874 (18h38m16.71220115s) [upload=16414]: Uploading segments... Oct 28 09:20:59.362942907 (18h38m16.712250183s) [upload=16414]: Got next segment. Inline: false Oct 28 09:20:59.362982592 (18h38m16.712289878s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 28 09:20:59.362989917 (18h38m16.712297203s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 28 09:21:01.021344744 (18h38m18.37065203s) [seg_pos={0 0}, upload=16414]: Begin upload segment... Oct 28 09:21:01.023040152 (18h38m18.372347438s) [seg_pos={0 0}, upload=16414]: Done begin upload segment. Oct 28 09:21:01.023120105 (18h38m18.372427391s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.023839364 (18h38m18.37314665s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024279452 (18h38m18.373586778s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024909652 (18h38m18.374216938s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025349159 (18h38m18.374656446s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025638280 (18h38m18.374945566s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.026032871 (18h38m18.375340157s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.027098981 (18h38m18.376406267s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028007251 (18h38m18.377314537s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028880052 (18h38m18.378187339s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029267360 (18h38m18.378574646s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029838027 (18h38m18.379145313s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030217620 (18h38m18.379524906s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030834975 (18h38m18.380142261s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030959723 (18h38m18.380267009s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.031485544 (18h38m18.38079283s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:02.782872101 (18h38m20.132179387s) [upload=16414]: Next returned nil segment Oct 28 09:21:02.782895556 (18h38m20.132202842s) [upload=16414]: Waiting for error group managing segments... Oct 28 09:21:03.066789825 (18h38m20.416097121s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.071416355 (18h38m20.420723641s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.072983429 (18h38m20.422290715s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.077582706 (18h38m20.426889992s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.078844368 (18h38m20.428151655s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.139198845 (18h38m20.488506131s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.141506871 (18h38m20.490814167s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.170875915 (18h38m20.520183201s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.222706510 (18h38m20.572013796s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.247291775 (18h38m20.596599062s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.259905544 (18h38m20.60921283s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.281169995 (18h38m20.630477281s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312400144 (18h38m20.66170743s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312442985 (18h38m20.661750271s) [seg_pos={0 0}, upload=16414]: Segment reached optimal threshold of 13 pieces. Oct 28 09:21:03.312804424 (18h38m20.66211172s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection Oct 28 09:21:03.312830723 (18h38m20.662138019s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312894104 (18h38m20.6622014s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 28 09:21:03.312902871 (18h38m20.662210157s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312912750 (18h38m20.662220126s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:40022->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 28 09:21:03.312923039 (18h38m20.662230335s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312959218 (18h38m20.662266504s) [seg_pos={0 0}, upload=16414]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 28 09:21:03.313075419 (18h38m20.662382705s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 28 09:21:03.313081431 (18h38m20.662388717s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 28 09:21:03.646425392 (18h38m20.995732688s) [upload=16414]: Done uploading segments... Oct 28 09:21:03.646449338 (18h38m20.995756624s) [upload=16414]: Upload finished. err: <nil> Duration: 6.561503522s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 6.032183867s FAIL (2025-10-28 10:04:21 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 09:20:59.362633688 (18h38m16.711940974s) [upload=16414]: Starting upload Oct 28 09:20:59.362695796 (18h38m16.712003082s) [upload=16414]: Done starting upload Oct 28 09:20:59.362893874 (18h38m16.71220115s) [upload=16414]: Uploading segments... Oct 28 09:20:59.362942907 (18h38m16.712250183s) [upload=16414]: Got next segment. Inline: false Oct 28 09:20:59.362982592 (18h38m16.712289878s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 28 09:20:59.362989917 (18h38m16.712297203s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 28 09:21:01.021344744 (18h38m18.37065203s) [seg_pos={0 0}, upload=16414]: Begin upload segment... Oct 28 09:21:01.023040152 (18h38m18.372347438s) [seg_pos={0 0}, upload=16414]: Done begin upload segment. Oct 28 09:21:01.023120105 (18h38m18.372427391s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.023839364 (18h38m18.37314665s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024279452 (18h38m18.373586778s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024909652 (18h38m18.374216938s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025349159 (18h38m18.374656446s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025638280 (18h38m18.374945566s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.026032871 (18h38m18.375340157s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.027098981 (18h38m18.376406267s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028007251 (18h38m18.377314537s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028880052 (18h38m18.378187339s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029267360 (18h38m18.378574646s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029838027 (18h38m18.379145313s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030217620 (18h38m18.379524906s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030834975 (18h38m18.380142261s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030959723 (18h38m18.380267009s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.031485544 (18h38m18.38079283s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:02.782872101 (18h38m20.132179387s) [upload=16414]: Next returned nil segment Oct 28 09:21:02.782895556 (18h38m20.132202842s) [upload=16414]: Waiting for error group managing segments... Oct 28 09:21:03.066789825 (18h38m20.416097121s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.071416355 (18h38m20.420723641s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.072983429 (18h38m20.422290715s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.077582706 (18h38m20.426889992s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.078844368 (18h38m20.428151655s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.139198845 (18h38m20.488506131s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.141506871 (18h38m20.490814167s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.170875915 (18h38m20.520183201s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.222706510 (18h38m20.572013796s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.247291775 (18h38m20.596599062s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.259905544 (18h38m20.60921283s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.281169995 (18h38m20.630477281s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312400144 (18h38m20.66170743s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312442985 (18h38m20.661750271s) [seg_pos={0 0}, upload=16414]: Segment reached optimal threshold of 13 pieces. Oct 28 09:21:03.312804424 (18h38m20.66211172s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection Oct 28 09:21:03.312830723 (18h38m20.662138019s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312894104 (18h38m20.6622014s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 28 09:21:03.312902871 (18h38m20.662210157s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312912750 (18h38m20.662220126s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:40022->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 28 09:21:03.312923039 (18h38m20.662230335s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312959218 (18h38m20.662266504s) [seg_pos={0 0}, upload=16414]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 28 09:21:03.313075419 (18h38m20.662382705s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 28 09:21:03.313081431 (18h38m20.662388717s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 28 09:21:03.646425392 (18h38m20.995732688s) [upload=16414]: Done uploading segments... Oct 28 09:21:03.646449338 (18h38m20.995756624s) [upload=16414]: Upload finished. err: <nil> Duration: 6.032203735s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 15.517938387s FAIL (2025-10-28 10:03:37 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 09:20:59.362633688 (18h38m16.711940974s) [upload=16414]: Starting upload Oct 28 09:20:59.362695796 (18h38m16.712003082s) [upload=16414]: Done starting upload Oct 28 09:20:59.362893874 (18h38m16.71220115s) [upload=16414]: Uploading segments... Oct 28 09:20:59.362942907 (18h38m16.712250183s) [upload=16414]: Got next segment. Inline: false Oct 28 09:20:59.362982592 (18h38m16.712289878s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 28 09:20:59.362989917 (18h38m16.712297203s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 28 09:21:01.021344744 (18h38m18.37065203s) [seg_pos={0 0}, upload=16414]: Begin upload segment... Oct 28 09:21:01.023040152 (18h38m18.372347438s) [seg_pos={0 0}, upload=16414]: Done begin upload segment. Oct 28 09:21:01.023120105 (18h38m18.372427391s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.023839364 (18h38m18.37314665s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024279452 (18h38m18.373586778s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024909652 (18h38m18.374216938s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025349159 (18h38m18.374656446s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025638280 (18h38m18.374945566s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.026032871 (18h38m18.375340157s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.027098981 (18h38m18.376406267s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028007251 (18h38m18.377314537s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028880052 (18h38m18.378187339s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029267360 (18h38m18.378574646s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029838027 (18h38m18.379145313s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030217620 (18h38m18.379524906s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030834975 (18h38m18.380142261s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030959723 (18h38m18.380267009s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.031485544 (18h38m18.38079283s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:02.782872101 (18h38m20.132179387s) [upload=16414]: Next returned nil segment Oct 28 09:21:02.782895556 (18h38m20.132202842s) [upload=16414]: Waiting for error group managing segments... Oct 28 09:21:03.066789825 (18h38m20.416097121s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.071416355 (18h38m20.420723641s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.072983429 (18h38m20.422290715s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.077582706 (18h38m20.426889992s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.078844368 (18h38m20.428151655s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.139198845 (18h38m20.488506131s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.141506871 (18h38m20.490814167s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.170875915 (18h38m20.520183201s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.222706510 (18h38m20.572013796s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.247291775 (18h38m20.596599062s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.259905544 (18h38m20.60921283s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.281169995 (18h38m20.630477281s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312400144 (18h38m20.66170743s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312442985 (18h38m20.661750271s) [seg_pos={0 0}, upload=16414]: Segment reached optimal threshold of 13 pieces. Oct 28 09:21:03.312804424 (18h38m20.66211172s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection Oct 28 09:21:03.312830723 (18h38m20.662138019s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312894104 (18h38m20.6622014s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 28 09:21:03.312902871 (18h38m20.662210157s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312912750 (18h38m20.662220126s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:40022->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 28 09:21:03.312923039 (18h38m20.662230335s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312959218 (18h38m20.662266504s) [seg_pos={0 0}, upload=16414]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 28 09:21:03.313075419 (18h38m20.662382705s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 28 09:21:03.313081431 (18h38m20.662388717s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 28 09:21:03.646425392 (18h38m20.995732688s) [upload=16414]: Done uploading segments... Oct 28 09:21:03.646449338 (18h38m20.995756624s) [upload=16414]: Upload finished. err: <nil> Duration: 15.517965308s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 6.264701018s FAIL (2025-10-28 10:02:13 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 09:20:59.362633688 (18h38m16.711940974s) [upload=16414]: Starting upload Oct 28 09:20:59.362695796 (18h38m16.712003082s) [upload=16414]: Done starting upload Oct 28 09:20:59.362893874 (18h38m16.71220115s) [upload=16414]: Uploading segments... Oct 28 09:20:59.362942907 (18h38m16.712250183s) [upload=16414]: Got next segment. Inline: false Oct 28 09:20:59.362982592 (18h38m16.712289878s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 28 09:20:59.362989917 (18h38m16.712297203s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 28 09:21:01.021344744 (18h38m18.37065203s) [seg_pos={0 0}, upload=16414]: Begin upload segment... Oct 28 09:21:01.023040152 (18h38m18.372347438s) [seg_pos={0 0}, upload=16414]: Done begin upload segment. Oct 28 09:21:01.023120105 (18h38m18.372427391s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.023839364 (18h38m18.37314665s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024279452 (18h38m18.373586778s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024909652 (18h38m18.374216938s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025349159 (18h38m18.374656446s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025638280 (18h38m18.374945566s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.026032871 (18h38m18.375340157s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.027098981 (18h38m18.376406267s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028007251 (18h38m18.377314537s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028880052 (18h38m18.378187339s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029267360 (18h38m18.378574646s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029838027 (18h38m18.379145313s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030217620 (18h38m18.379524906s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030834975 (18h38m18.380142261s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030959723 (18h38m18.380267009s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.031485544 (18h38m18.38079283s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:02.782872101 (18h38m20.132179387s) [upload=16414]: Next returned nil segment Oct 28 09:21:02.782895556 (18h38m20.132202842s) [upload=16414]: Waiting for error group managing segments... Oct 28 09:21:03.066789825 (18h38m20.416097121s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.071416355 (18h38m20.420723641s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.072983429 (18h38m20.422290715s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.077582706 (18h38m20.426889992s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.078844368 (18h38m20.428151655s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.139198845 (18h38m20.488506131s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.141506871 (18h38m20.490814167s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.170875915 (18h38m20.520183201s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.222706510 (18h38m20.572013796s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.247291775 (18h38m20.596599062s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.259905544 (18h38m20.60921283s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.281169995 (18h38m20.630477281s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312400144 (18h38m20.66170743s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312442985 (18h38m20.661750271s) [seg_pos={0 0}, upload=16414]: Segment reached optimal threshold of 13 pieces. Oct 28 09:21:03.312804424 (18h38m20.66211172s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection Oct 28 09:21:03.312830723 (18h38m20.662138019s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312894104 (18h38m20.6622014s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 28 09:21:03.312902871 (18h38m20.662210157s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312912750 (18h38m20.662220126s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:40022->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 28 09:21:03.312923039 (18h38m20.662230335s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312959218 (18h38m20.662266504s) [seg_pos={0 0}, upload=16414]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 28 09:21:03.313075419 (18h38m20.662382705s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 28 09:21:03.313081431 (18h38m20.662388717s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 28 09:21:03.646425392 (18h38m20.995732688s) [upload=16414]: Done uploading segments... Oct 28 09:21:03.646449338 (18h38m20.995756624s) [upload=16414]: Upload finished. err: <nil> Duration: 6.264737056s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 6.00543068s FAIL (2025-10-28 10:01:21 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 09:20:59.362633688 (18h38m16.711940974s) [upload=16414]: Starting upload Oct 28 09:20:59.362695796 (18h38m16.712003082s) [upload=16414]: Done starting upload Oct 28 09:20:59.362893874 (18h38m16.71220115s) [upload=16414]: Uploading segments... Oct 28 09:20:59.362942907 (18h38m16.712250183s) [upload=16414]: Got next segment. Inline: false Oct 28 09:20:59.362982592 (18h38m16.712289878s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 28 09:20:59.362989917 (18h38m16.712297203s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 28 09:21:01.021344744 (18h38m18.37065203s) [seg_pos={0 0}, upload=16414]: Begin upload segment... Oct 28 09:21:01.023040152 (18h38m18.372347438s) [seg_pos={0 0}, upload=16414]: Done begin upload segment. Oct 28 09:21:01.023120105 (18h38m18.372427391s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.023839364 (18h38m18.37314665s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024279452 (18h38m18.373586778s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024909652 (18h38m18.374216938s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025349159 (18h38m18.374656446s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025638280 (18h38m18.374945566s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.026032871 (18h38m18.375340157s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.027098981 (18h38m18.376406267s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028007251 (18h38m18.377314537s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028880052 (18h38m18.378187339s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029267360 (18h38m18.378574646s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029838027 (18h38m18.379145313s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030217620 (18h38m18.379524906s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030834975 (18h38m18.380142261s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030959723 (18h38m18.380267009s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.031485544 (18h38m18.38079283s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:02.782872101 (18h38m20.132179387s) [upload=16414]: Next returned nil segment Oct 28 09:21:02.782895556 (18h38m20.132202842s) [upload=16414]: Waiting for error group managing segments... Oct 28 09:21:03.066789825 (18h38m20.416097121s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.071416355 (18h38m20.420723641s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.072983429 (18h38m20.422290715s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.077582706 (18h38m20.426889992s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.078844368 (18h38m20.428151655s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.139198845 (18h38m20.488506131s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.141506871 (18h38m20.490814167s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.170875915 (18h38m20.520183201s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.222706510 (18h38m20.572013796s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.247291775 (18h38m20.596599062s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.259905544 (18h38m20.60921283s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.281169995 (18h38m20.630477281s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312400144 (18h38m20.66170743s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312442985 (18h38m20.661750271s) [seg_pos={0 0}, upload=16414]: Segment reached optimal threshold of 13 pieces. Oct 28 09:21:03.312804424 (18h38m20.66211172s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection Oct 28 09:21:03.312830723 (18h38m20.662138019s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312894104 (18h38m20.6622014s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 28 09:21:03.312902871 (18h38m20.662210157s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312912750 (18h38m20.662220126s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:40022->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 28 09:21:03.312923039 (18h38m20.662230335s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312959218 (18h38m20.662266504s) [seg_pos={0 0}, upload=16414]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 28 09:21:03.313075419 (18h38m20.662382705s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 28 09:21:03.313081431 (18h38m20.662388717s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 28 09:21:03.646425392 (18h38m20.995732688s) [upload=16414]: Done uploading segments... Oct 28 09:21:03.646449338 (18h38m20.995756624s) [upload=16414]: Upload finished. err: <nil> Duration: 6.005461809s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.990275745s FAIL (2025-10-28 10:00:54 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 09:20:59.362633688 (18h38m16.711940974s) [upload=16414]: Starting upload Oct 28 09:20:59.362695796 (18h38m16.712003082s) [upload=16414]: Done starting upload Oct 28 09:20:59.362893874 (18h38m16.71220115s) [upload=16414]: Uploading segments... Oct 28 09:20:59.362942907 (18h38m16.712250183s) [upload=16414]: Got next segment. Inline: false Oct 28 09:20:59.362982592 (18h38m16.712289878s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 28 09:20:59.362989917 (18h38m16.712297203s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 28 09:21:01.021344744 (18h38m18.37065203s) [seg_pos={0 0}, upload=16414]: Begin upload segment... Oct 28 09:21:01.023040152 (18h38m18.372347438s) [seg_pos={0 0}, upload=16414]: Done begin upload segment. Oct 28 09:21:01.023120105 (18h38m18.372427391s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.023839364 (18h38m18.37314665s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024279452 (18h38m18.373586778s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024909652 (18h38m18.374216938s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025349159 (18h38m18.374656446s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025638280 (18h38m18.374945566s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.026032871 (18h38m18.375340157s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.027098981 (18h38m18.376406267s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028007251 (18h38m18.377314537s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028880052 (18h38m18.378187339s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029267360 (18h38m18.378574646s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029838027 (18h38m18.379145313s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030217620 (18h38m18.379524906s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030834975 (18h38m18.380142261s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030959723 (18h38m18.380267009s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.031485544 (18h38m18.38079283s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:02.782872101 (18h38m20.132179387s) [upload=16414]: Next returned nil segment Oct 28 09:21:02.782895556 (18h38m20.132202842s) [upload=16414]: Waiting for error group managing segments... Oct 28 09:21:03.066789825 (18h38m20.416097121s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.071416355 (18h38m20.420723641s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.072983429 (18h38m20.422290715s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.077582706 (18h38m20.426889992s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.078844368 (18h38m20.428151655s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.139198845 (18h38m20.488506131s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.141506871 (18h38m20.490814167s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.170875915 (18h38m20.520183201s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.222706510 (18h38m20.572013796s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.247291775 (18h38m20.596599062s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.259905544 (18h38m20.60921283s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.281169995 (18h38m20.630477281s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312400144 (18h38m20.66170743s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312442985 (18h38m20.661750271s) [seg_pos={0 0}, upload=16414]: Segment reached optimal threshold of 13 pieces. Oct 28 09:21:03.312804424 (18h38m20.66211172s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection Oct 28 09:21:03.312830723 (18h38m20.662138019s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312894104 (18h38m20.6622014s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 28 09:21:03.312902871 (18h38m20.662210157s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312912750 (18h38m20.662220126s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:40022->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 28 09:21:03.312923039 (18h38m20.662230335s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312959218 (18h38m20.662266504s) [seg_pos={0 0}, upload=16414]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 28 09:21:03.313075419 (18h38m20.662382705s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 28 09:21:03.313081431 (18h38m20.662388717s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 28 09:21:03.646425392 (18h38m20.995732688s) [upload=16414]: Done uploading segments... Oct 28 09:21:03.646449338 (18h38m20.995756624s) [upload=16414]: Upload finished. err: <nil> Duration: 5.990310632s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 6.148385441s FAIL (2025-10-28 10:00:20 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 09:20:59.362633688 (18h38m16.711940974s) [upload=16414]: Starting upload Oct 28 09:20:59.362695796 (18h38m16.712003082s) [upload=16414]: Done starting upload Oct 28 09:20:59.362893874 (18h38m16.71220115s) [upload=16414]: Uploading segments... Oct 28 09:20:59.362942907 (18h38m16.712250183s) [upload=16414]: Got next segment. Inline: false Oct 28 09:20:59.362982592 (18h38m16.712289878s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 28 09:20:59.362989917 (18h38m16.712297203s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 28 09:21:01.021344744 (18h38m18.37065203s) [seg_pos={0 0}, upload=16414]: Begin upload segment... Oct 28 09:21:01.023040152 (18h38m18.372347438s) [seg_pos={0 0}, upload=16414]: Done begin upload segment. Oct 28 09:21:01.023120105 (18h38m18.372427391s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.023839364 (18h38m18.37314665s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024279452 (18h38m18.373586778s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024909652 (18h38m18.374216938s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025349159 (18h38m18.374656446s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025638280 (18h38m18.374945566s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.026032871 (18h38m18.375340157s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.027098981 (18h38m18.376406267s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028007251 (18h38m18.377314537s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028880052 (18h38m18.378187339s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029267360 (18h38m18.378574646s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029838027 (18h38m18.379145313s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030217620 (18h38m18.379524906s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030834975 (18h38m18.380142261s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030959723 (18h38m18.380267009s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.031485544 (18h38m18.38079283s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:02.782872101 (18h38m20.132179387s) [upload=16414]: Next returned nil segment Oct 28 09:21:02.782895556 (18h38m20.132202842s) [upload=16414]: Waiting for error group managing segments... Oct 28 09:21:03.066789825 (18h38m20.416097121s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.071416355 (18h38m20.420723641s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.072983429 (18h38m20.422290715s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.077582706 (18h38m20.426889992s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.078844368 (18h38m20.428151655s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.139198845 (18h38m20.488506131s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.141506871 (18h38m20.490814167s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.170875915 (18h38m20.520183201s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.222706510 (18h38m20.572013796s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.247291775 (18h38m20.596599062s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.259905544 (18h38m20.60921283s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.281169995 (18h38m20.630477281s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312400144 (18h38m20.66170743s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312442985 (18h38m20.661750271s) [seg_pos={0 0}, upload=16414]: Segment reached optimal threshold of 13 pieces. Oct 28 09:21:03.312804424 (18h38m20.66211172s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection Oct 28 09:21:03.312830723 (18h38m20.662138019s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312894104 (18h38m20.6622014s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 28 09:21:03.312902871 (18h38m20.662210157s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312912750 (18h38m20.662220126s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:40022->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 28 09:21:03.312923039 (18h38m20.662230335s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312959218 (18h38m20.662266504s) [seg_pos={0 0}, upload=16414]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 28 09:21:03.313075419 (18h38m20.662382705s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 28 09:21:03.313081431 (18h38m20.662388717s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 28 09:21:03.646425392 (18h38m20.995732688s) [upload=16414]: Done uploading segments... Oct 28 09:21:03.646449338 (18h38m20.995756624s) [upload=16414]: Upload finished. err: <nil> Duration: 6.148420015s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.860985954s FAIL (2025-10-28 09:59:37 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 09:20:59.362633688 (18h38m16.711940974s) [upload=16414]: Starting upload Oct 28 09:20:59.362695796 (18h38m16.712003082s) [upload=16414]: Done starting upload Oct 28 09:20:59.362893874 (18h38m16.71220115s) [upload=16414]: Uploading segments... Oct 28 09:20:59.362942907 (18h38m16.712250183s) [upload=16414]: Got next segment. Inline: false Oct 28 09:20:59.362982592 (18h38m16.712289878s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 28 09:20:59.362989917 (18h38m16.712297203s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 28 09:21:01.021344744 (18h38m18.37065203s) [seg_pos={0 0}, upload=16414]: Begin upload segment... Oct 28 09:21:01.023040152 (18h38m18.372347438s) [seg_pos={0 0}, upload=16414]: Done begin upload segment. Oct 28 09:21:01.023120105 (18h38m18.372427391s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.023839364 (18h38m18.37314665s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024279452 (18h38m18.373586778s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024909652 (18h38m18.374216938s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025349159 (18h38m18.374656446s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025638280 (18h38m18.374945566s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.026032871 (18h38m18.375340157s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.027098981 (18h38m18.376406267s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028007251 (18h38m18.377314537s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028880052 (18h38m18.378187339s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029267360 (18h38m18.378574646s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029838027 (18h38m18.379145313s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030217620 (18h38m18.379524906s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030834975 (18h38m18.380142261s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030959723 (18h38m18.380267009s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.031485544 (18h38m18.38079283s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:02.782872101 (18h38m20.132179387s) [upload=16414]: Next returned nil segment Oct 28 09:21:02.782895556 (18h38m20.132202842s) [upload=16414]: Waiting for error group managing segments... Oct 28 09:21:03.066789825 (18h38m20.416097121s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.071416355 (18h38m20.420723641s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.072983429 (18h38m20.422290715s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.077582706 (18h38m20.426889992s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.078844368 (18h38m20.428151655s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.139198845 (18h38m20.488506131s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.141506871 (18h38m20.490814167s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.170875915 (18h38m20.520183201s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.222706510 (18h38m20.572013796s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.247291775 (18h38m20.596599062s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.259905544 (18h38m20.60921283s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.281169995 (18h38m20.630477281s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312400144 (18h38m20.66170743s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312442985 (18h38m20.661750271s) [seg_pos={0 0}, upload=16414]: Segment reached optimal threshold of 13 pieces. Oct 28 09:21:03.312804424 (18h38m20.66211172s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection Oct 28 09:21:03.312830723 (18h38m20.662138019s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312894104 (18h38m20.6622014s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 28 09:21:03.312902871 (18h38m20.662210157s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312912750 (18h38m20.662220126s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:40022->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 28 09:21:03.312923039 (18h38m20.662230335s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312959218 (18h38m20.662266504s) [seg_pos={0 0}, upload=16414]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 28 09:21:03.313075419 (18h38m20.662382705s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 28 09:21:03.313081431 (18h38m20.662388717s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 28 09:21:03.646425392 (18h38m20.995732688s) [upload=16414]: Done uploading segments... Oct 28 09:21:03.646449338 (18h38m20.995756624s) [upload=16414]: Upload finished. err: <nil> Duration: 5.861011121s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.228904435s FAIL (2025-10-28 09:59:06 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 09:20:59.362633688 (18h38m16.711940974s) [upload=16414]: Starting upload Oct 28 09:20:59.362695796 (18h38m16.712003082s) [upload=16414]: Done starting upload Oct 28 09:20:59.362893874 (18h38m16.71220115s) [upload=16414]: Uploading segments... Oct 28 09:20:59.362942907 (18h38m16.712250183s) [upload=16414]: Got next segment. Inline: false Oct 28 09:20:59.362982592 (18h38m16.712289878s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 28 09:20:59.362989917 (18h38m16.712297203s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 28 09:21:01.021344744 (18h38m18.37065203s) [seg_pos={0 0}, upload=16414]: Begin upload segment... Oct 28 09:21:01.023040152 (18h38m18.372347438s) [seg_pos={0 0}, upload=16414]: Done begin upload segment. Oct 28 09:21:01.023120105 (18h38m18.372427391s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.023839364 (18h38m18.37314665s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024279452 (18h38m18.373586778s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024909652 (18h38m18.374216938s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025349159 (18h38m18.374656446s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025638280 (18h38m18.374945566s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.026032871 (18h38m18.375340157s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.027098981 (18h38m18.376406267s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028007251 (18h38m18.377314537s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028880052 (18h38m18.378187339s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029267360 (18h38m18.378574646s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029838027 (18h38m18.379145313s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030217620 (18h38m18.379524906s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030834975 (18h38m18.380142261s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030959723 (18h38m18.380267009s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.031485544 (18h38m18.38079283s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:02.782872101 (18h38m20.132179387s) [upload=16414]: Next returned nil segment Oct 28 09:21:02.782895556 (18h38m20.132202842s) [upload=16414]: Waiting for error group managing segments... Oct 28 09:21:03.066789825 (18h38m20.416097121s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.071416355 (18h38m20.420723641s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.072983429 (18h38m20.422290715s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.077582706 (18h38m20.426889992s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.078844368 (18h38m20.428151655s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.139198845 (18h38m20.488506131s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.141506871 (18h38m20.490814167s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.170875915 (18h38m20.520183201s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.222706510 (18h38m20.572013796s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.247291775 (18h38m20.596599062s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.259905544 (18h38m20.60921283s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.281169995 (18h38m20.630477281s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312400144 (18h38m20.66170743s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312442985 (18h38m20.661750271s) [seg_pos={0 0}, upload=16414]: Segment reached optimal threshold of 13 pieces. Oct 28 09:21:03.312804424 (18h38m20.66211172s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection Oct 28 09:21:03.312830723 (18h38m20.662138019s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312894104 (18h38m20.6622014s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 28 09:21:03.312902871 (18h38m20.662210157s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312912750 (18h38m20.662220126s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:40022->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 28 09:21:03.312923039 (18h38m20.662230335s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312959218 (18h38m20.662266504s) [seg_pos={0 0}, upload=16414]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 28 09:21:03.313075419 (18h38m20.662382705s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 28 09:21:03.313081431 (18h38m20.662388717s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 28 09:21:03.646425392 (18h38m20.995732688s) [upload=16414]: Done uploading segments... Oct 28 09:21:03.646449338 (18h38m20.995756624s) [upload=16414]: Upload finished. err: <nil> Duration: 6.228936175s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.978206162s FAIL (2025-10-28 09:58:40 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 09:20:59.362633688 (18h38m16.711940974s) [upload=16414]: Starting upload Oct 28 09:20:59.362695796 (18h38m16.712003082s) [upload=16414]: Done starting upload Oct 28 09:20:59.362893874 (18h38m16.71220115s) [upload=16414]: Uploading segments... Oct 28 09:20:59.362942907 (18h38m16.712250183s) [upload=16414]: Got next segment. Inline: false Oct 28 09:20:59.362982592 (18h38m16.712289878s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 28 09:20:59.362989917 (18h38m16.712297203s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 28 09:21:01.021344744 (18h38m18.37065203s) [seg_pos={0 0}, upload=16414]: Begin upload segment... Oct 28 09:21:01.023040152 (18h38m18.372347438s) [seg_pos={0 0}, upload=16414]: Done begin upload segment. Oct 28 09:21:01.023120105 (18h38m18.372427391s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.023839364 (18h38m18.37314665s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024279452 (18h38m18.373586778s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024909652 (18h38m18.374216938s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025349159 (18h38m18.374656446s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025638280 (18h38m18.374945566s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.026032871 (18h38m18.375340157s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.027098981 (18h38m18.376406267s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028007251 (18h38m18.377314537s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028880052 (18h38m18.378187339s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029267360 (18h38m18.378574646s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029838027 (18h38m18.379145313s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030217620 (18h38m18.379524906s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030834975 (18h38m18.380142261s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030959723 (18h38m18.380267009s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.031485544 (18h38m18.38079283s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:02.782872101 (18h38m20.132179387s) [upload=16414]: Next returned nil segment Oct 28 09:21:02.782895556 (18h38m20.132202842s) [upload=16414]: Waiting for error group managing segments... Oct 28 09:21:03.066789825 (18h38m20.416097121s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.071416355 (18h38m20.420723641s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.072983429 (18h38m20.422290715s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.077582706 (18h38m20.426889992s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.078844368 (18h38m20.428151655s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.139198845 (18h38m20.488506131s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.141506871 (18h38m20.490814167s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.170875915 (18h38m20.520183201s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.222706510 (18h38m20.572013796s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.247291775 (18h38m20.596599062s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.259905544 (18h38m20.60921283s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.281169995 (18h38m20.630477281s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312400144 (18h38m20.66170743s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312442985 (18h38m20.661750271s) [seg_pos={0 0}, upload=16414]: Segment reached optimal threshold of 13 pieces. Oct 28 09:21:03.312804424 (18h38m20.66211172s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection Oct 28 09:21:03.312830723 (18h38m20.662138019s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312894104 (18h38m20.6622014s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 28 09:21:03.312902871 (18h38m20.662210157s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312912750 (18h38m20.662220126s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:40022->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 28 09:21:03.312923039 (18h38m20.662230335s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312959218 (18h38m20.662266504s) [seg_pos={0 0}, upload=16414]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 28 09:21:03.313075419 (18h38m20.662382705s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 28 09:21:03.313081431 (18h38m20.662388717s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 28 09:21:03.646425392 (18h38m20.995732688s) [upload=16414]: Done uploading segments... Oct 28 09:21:03.646449338 (18h38m20.995756624s) [upload=16414]: Upload finished. err: <nil> Duration: 5.978232354s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.86563414s FAIL (2025-10-28 09:58:16 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 09:20:59.362633688 (18h38m16.711940974s) [upload=16414]: Starting upload Oct 28 09:20:59.362695796 (18h38m16.712003082s) [upload=16414]: Done starting upload Oct 28 09:20:59.362893874 (18h38m16.71220115s) [upload=16414]: Uploading segments... Oct 28 09:20:59.362942907 (18h38m16.712250183s) [upload=16414]: Got next segment. Inline: false Oct 28 09:20:59.362982592 (18h38m16.712289878s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 28 09:20:59.362989917 (18h38m16.712297203s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 28 09:21:01.021344744 (18h38m18.37065203s) [seg_pos={0 0}, upload=16414]: Begin upload segment... Oct 28 09:21:01.023040152 (18h38m18.372347438s) [seg_pos={0 0}, upload=16414]: Done begin upload segment. Oct 28 09:21:01.023120105 (18h38m18.372427391s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.023839364 (18h38m18.37314665s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024279452 (18h38m18.373586778s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024909652 (18h38m18.374216938s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025349159 (18h38m18.374656446s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025638280 (18h38m18.374945566s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.026032871 (18h38m18.375340157s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.027098981 (18h38m18.376406267s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028007251 (18h38m18.377314537s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028880052 (18h38m18.378187339s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029267360 (18h38m18.378574646s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029838027 (18h38m18.379145313s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030217620 (18h38m18.379524906s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030834975 (18h38m18.380142261s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030959723 (18h38m18.380267009s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.031485544 (18h38m18.38079283s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:02.782872101 (18h38m20.132179387s) [upload=16414]: Next returned nil segment Oct 28 09:21:02.782895556 (18h38m20.132202842s) [upload=16414]: Waiting for error group managing segments... Oct 28 09:21:03.066789825 (18h38m20.416097121s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.071416355 (18h38m20.420723641s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.072983429 (18h38m20.422290715s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.077582706 (18h38m20.426889992s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.078844368 (18h38m20.428151655s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.139198845 (18h38m20.488506131s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.141506871 (18h38m20.490814167s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.170875915 (18h38m20.520183201s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.222706510 (18h38m20.572013796s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.247291775 (18h38m20.596599062s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.259905544 (18h38m20.60921283s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.281169995 (18h38m20.630477281s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312400144 (18h38m20.66170743s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312442985 (18h38m20.661750271s) [seg_pos={0 0}, upload=16414]: Segment reached optimal threshold of 13 pieces. Oct 28 09:21:03.312804424 (18h38m20.66211172s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection Oct 28 09:21:03.312830723 (18h38m20.662138019s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312894104 (18h38m20.6622014s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 28 09:21:03.312902871 (18h38m20.662210157s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312912750 (18h38m20.662220126s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:40022->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 28 09:21:03.312923039 (18h38m20.662230335s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312959218 (18h38m20.662266504s) [seg_pos={0 0}, upload=16414]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 28 09:21:03.313075419 (18h38m20.662382705s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 28 09:21:03.313081431 (18h38m20.662388717s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 28 09:21:03.646425392 (18h38m20.995732688s) [upload=16414]: Done uploading segments... Oct 28 09:21:03.646449338 (18h38m20.995756624s) [upload=16414]: Upload finished. err: <nil> Duration: 5.865664878s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.535609086s FAIL (2025-10-28 09:57:13 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 09:20:59.362633688 (18h38m16.711940974s) [upload=16414]: Starting upload Oct 28 09:20:59.362695796 (18h38m16.712003082s) [upload=16414]: Done starting upload Oct 28 09:20:59.362893874 (18h38m16.71220115s) [upload=16414]: Uploading segments... Oct 28 09:20:59.362942907 (18h38m16.712250183s) [upload=16414]: Got next segment. Inline: false Oct 28 09:20:59.362982592 (18h38m16.712289878s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 28 09:20:59.362989917 (18h38m16.712297203s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 28 09:21:01.021344744 (18h38m18.37065203s) [seg_pos={0 0}, upload=16414]: Begin upload segment... Oct 28 09:21:01.023040152 (18h38m18.372347438s) [seg_pos={0 0}, upload=16414]: Done begin upload segment. Oct 28 09:21:01.023120105 (18h38m18.372427391s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.023839364 (18h38m18.37314665s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024279452 (18h38m18.373586778s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024909652 (18h38m18.374216938s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025349159 (18h38m18.374656446s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025638280 (18h38m18.374945566s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.026032871 (18h38m18.375340157s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.027098981 (18h38m18.376406267s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028007251 (18h38m18.377314537s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028880052 (18h38m18.378187339s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029267360 (18h38m18.378574646s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029838027 (18h38m18.379145313s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030217620 (18h38m18.379524906s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030834975 (18h38m18.380142261s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030959723 (18h38m18.380267009s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.031485544 (18h38m18.38079283s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:02.782872101 (18h38m20.132179387s) [upload=16414]: Next returned nil segment Oct 28 09:21:02.782895556 (18h38m20.132202842s) [upload=16414]: Waiting for error group managing segments... Oct 28 09:21:03.066789825 (18h38m20.416097121s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.071416355 (18h38m20.420723641s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.072983429 (18h38m20.422290715s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.077582706 (18h38m20.426889992s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.078844368 (18h38m20.428151655s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.139198845 (18h38m20.488506131s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.141506871 (18h38m20.490814167s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.170875915 (18h38m20.520183201s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.222706510 (18h38m20.572013796s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.247291775 (18h38m20.596599062s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.259905544 (18h38m20.60921283s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.281169995 (18h38m20.630477281s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312400144 (18h38m20.66170743s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312442985 (18h38m20.661750271s) [seg_pos={0 0}, upload=16414]: Segment reached optimal threshold of 13 pieces. Oct 28 09:21:03.312804424 (18h38m20.66211172s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection Oct 28 09:21:03.312830723 (18h38m20.662138019s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312894104 (18h38m20.6622014s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 28 09:21:03.312902871 (18h38m20.662210157s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312912750 (18h38m20.662220126s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:40022->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 28 09:21:03.312923039 (18h38m20.662230335s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312959218 (18h38m20.662266504s) [seg_pos={0 0}, upload=16414]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 28 09:21:03.313075419 (18h38m20.662382705s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 28 09:21:03.313081431 (18h38m20.662388717s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 28 09:21:03.646425392 (18h38m20.995732688s) [upload=16414]: Done uploading segments... Oct 28 09:21:03.646449338 (18h38m20.995756624s) [upload=16414]: Upload finished. err: <nil> Duration: 6.535715149s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.796969517s FAIL (2025-10-28 09:56:03 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 09:20:59.362633688 (18h38m16.711940974s) [upload=16414]: Starting upload Oct 28 09:20:59.362695796 (18h38m16.712003082s) [upload=16414]: Done starting upload Oct 28 09:20:59.362893874 (18h38m16.71220115s) [upload=16414]: Uploading segments... Oct 28 09:20:59.362942907 (18h38m16.712250183s) [upload=16414]: Got next segment. Inline: false Oct 28 09:20:59.362982592 (18h38m16.712289878s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 28 09:20:59.362989917 (18h38m16.712297203s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 28 09:21:01.021344744 (18h38m18.37065203s) [seg_pos={0 0}, upload=16414]: Begin upload segment... Oct 28 09:21:01.023040152 (18h38m18.372347438s) [seg_pos={0 0}, upload=16414]: Done begin upload segment. Oct 28 09:21:01.023120105 (18h38m18.372427391s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.023839364 (18h38m18.37314665s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024279452 (18h38m18.373586778s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024909652 (18h38m18.374216938s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025349159 (18h38m18.374656446s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025638280 (18h38m18.374945566s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.026032871 (18h38m18.375340157s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.027098981 (18h38m18.376406267s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028007251 (18h38m18.377314537s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028880052 (18h38m18.378187339s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029267360 (18h38m18.378574646s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029838027 (18h38m18.379145313s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030217620 (18h38m18.379524906s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030834975 (18h38m18.380142261s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030959723 (18h38m18.380267009s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.031485544 (18h38m18.38079283s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:02.782872101 (18h38m20.132179387s) [upload=16414]: Next returned nil segment Oct 28 09:21:02.782895556 (18h38m20.132202842s) [upload=16414]: Waiting for error group managing segments... Oct 28 09:21:03.066789825 (18h38m20.416097121s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.071416355 (18h38m20.420723641s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.072983429 (18h38m20.422290715s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.077582706 (18h38m20.426889992s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.078844368 (18h38m20.428151655s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.139198845 (18h38m20.488506131s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.141506871 (18h38m20.490814167s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.170875915 (18h38m20.520183201s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.222706510 (18h38m20.572013796s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.247291775 (18h38m20.596599062s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.259905544 (18h38m20.60921283s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.281169995 (18h38m20.630477281s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312400144 (18h38m20.66170743s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312442985 (18h38m20.661750271s) [seg_pos={0 0}, upload=16414]: Segment reached optimal threshold of 13 pieces. Oct 28 09:21:03.312804424 (18h38m20.66211172s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection Oct 28 09:21:03.312830723 (18h38m20.662138019s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312894104 (18h38m20.6622014s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 28 09:21:03.312902871 (18h38m20.662210157s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312912750 (18h38m20.662220126s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:40022->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 28 09:21:03.312923039 (18h38m20.662230335s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312959218 (18h38m20.662266504s) [seg_pos={0 0}, upload=16414]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 28 09:21:03.313075419 (18h38m20.662382705s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 28 09:21:03.313081431 (18h38m20.662388717s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 28 09:21:03.646425392 (18h38m20.995732688s) [upload=16414]: Done uploading segments... Oct 28 09:21:03.646449338 (18h38m20.995756624s) [upload=16414]: Upload finished. err: <nil> Duration: 5.797107889s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.919413987s FAIL (2025-10-28 09:54:55 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 09:20:59.362633688 (18h38m16.711940974s) [upload=16414]: Starting upload Oct 28 09:20:59.362695796 (18h38m16.712003082s) [upload=16414]: Done starting upload Oct 28 09:20:59.362893874 (18h38m16.71220115s) [upload=16414]: Uploading segments... Oct 28 09:20:59.362942907 (18h38m16.712250183s) [upload=16414]: Got next segment. Inline: false Oct 28 09:20:59.362982592 (18h38m16.712289878s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 28 09:20:59.362989917 (18h38m16.712297203s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 28 09:21:01.021344744 (18h38m18.37065203s) [seg_pos={0 0}, upload=16414]: Begin upload segment... Oct 28 09:21:01.023040152 (18h38m18.372347438s) [seg_pos={0 0}, upload=16414]: Done begin upload segment. Oct 28 09:21:01.023120105 (18h38m18.372427391s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.023839364 (18h38m18.37314665s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024279452 (18h38m18.373586778s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024909652 (18h38m18.374216938s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025349159 (18h38m18.374656446s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025638280 (18h38m18.374945566s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.026032871 (18h38m18.375340157s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.027098981 (18h38m18.376406267s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028007251 (18h38m18.377314537s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028880052 (18h38m18.378187339s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029267360 (18h38m18.378574646s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029838027 (18h38m18.379145313s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030217620 (18h38m18.379524906s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030834975 (18h38m18.380142261s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030959723 (18h38m18.380267009s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.031485544 (18h38m18.38079283s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:02.782872101 (18h38m20.132179387s) [upload=16414]: Next returned nil segment Oct 28 09:21:02.782895556 (18h38m20.132202842s) [upload=16414]: Waiting for error group managing segments... Oct 28 09:21:03.066789825 (18h38m20.416097121s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.071416355 (18h38m20.420723641s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.072983429 (18h38m20.422290715s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.077582706 (18h38m20.426889992s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.078844368 (18h38m20.428151655s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.139198845 (18h38m20.488506131s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.141506871 (18h38m20.490814167s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.170875915 (18h38m20.520183201s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.222706510 (18h38m20.572013796s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.247291775 (18h38m20.596599062s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.259905544 (18h38m20.60921283s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.281169995 (18h38m20.630477281s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312400144 (18h38m20.66170743s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312442985 (18h38m20.661750271s) [seg_pos={0 0}, upload=16414]: Segment reached optimal threshold of 13 pieces. Oct 28 09:21:03.312804424 (18h38m20.66211172s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection Oct 28 09:21:03.312830723 (18h38m20.662138019s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312894104 (18h38m20.6622014s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 28 09:21:03.312902871 (18h38m20.662210157s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312912750 (18h38m20.662220126s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:40022->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 28 09:21:03.312923039 (18h38m20.662230335s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312959218 (18h38m20.662266504s) [seg_pos={0 0}, upload=16414]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 28 09:21:03.313075419 (18h38m20.662382705s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 28 09:21:03.313081431 (18h38m20.662388717s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 28 09:21:03.646425392 (18h38m20.995732688s) [upload=16414]: Done uploading segments... Oct 28 09:21:03.646449338 (18h38m20.995756624s) [upload=16414]: Upload finished. err: <nil> Duration: 5.919449164s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.894737939s FAIL (2025-10-28 09:54:09 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 09:20:59.362633688 (18h38m16.711940974s) [upload=16414]: Starting upload Oct 28 09:20:59.362695796 (18h38m16.712003082s) [upload=16414]: Done starting upload Oct 28 09:20:59.362893874 (18h38m16.71220115s) [upload=16414]: Uploading segments... Oct 28 09:20:59.362942907 (18h38m16.712250183s) [upload=16414]: Got next segment. Inline: false Oct 28 09:20:59.362982592 (18h38m16.712289878s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 28 09:20:59.362989917 (18h38m16.712297203s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 28 09:21:01.021344744 (18h38m18.37065203s) [seg_pos={0 0}, upload=16414]: Begin upload segment... Oct 28 09:21:01.023040152 (18h38m18.372347438s) [seg_pos={0 0}, upload=16414]: Done begin upload segment. Oct 28 09:21:01.023120105 (18h38m18.372427391s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.023839364 (18h38m18.37314665s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024279452 (18h38m18.373586778s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024909652 (18h38m18.374216938s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025349159 (18h38m18.374656446s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025638280 (18h38m18.374945566s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.026032871 (18h38m18.375340157s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.027098981 (18h38m18.376406267s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028007251 (18h38m18.377314537s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028880052 (18h38m18.378187339s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029267360 (18h38m18.378574646s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029838027 (18h38m18.379145313s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030217620 (18h38m18.379524906s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030834975 (18h38m18.380142261s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030959723 (18h38m18.380267009s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.031485544 (18h38m18.38079283s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:02.782872101 (18h38m20.132179387s) [upload=16414]: Next returned nil segment Oct 28 09:21:02.782895556 (18h38m20.132202842s) [upload=16414]: Waiting for error group managing segments... Oct 28 09:21:03.066789825 (18h38m20.416097121s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.071416355 (18h38m20.420723641s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.072983429 (18h38m20.422290715s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.077582706 (18h38m20.426889992s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.078844368 (18h38m20.428151655s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.139198845 (18h38m20.488506131s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.141506871 (18h38m20.490814167s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.170875915 (18h38m20.520183201s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.222706510 (18h38m20.572013796s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.247291775 (18h38m20.596599062s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.259905544 (18h38m20.60921283s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.281169995 (18h38m20.630477281s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312400144 (18h38m20.66170743s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312442985 (18h38m20.661750271s) [seg_pos={0 0}, upload=16414]: Segment reached optimal threshold of 13 pieces. Oct 28 09:21:03.312804424 (18h38m20.66211172s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection Oct 28 09:21:03.312830723 (18h38m20.662138019s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312894104 (18h38m20.6622014s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 28 09:21:03.312902871 (18h38m20.662210157s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312912750 (18h38m20.662220126s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:40022->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 28 09:21:03.312923039 (18h38m20.662230335s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312959218 (18h38m20.662266504s) [seg_pos={0 0}, upload=16414]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 28 09:21:03.313075419 (18h38m20.662382705s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 28 09:21:03.313081431 (18h38m20.662388717s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 28 09:21:03.646425392 (18h38m20.995732688s) [upload=16414]: Done uploading segments... Oct 28 09:21:03.646449338 (18h38m20.995756624s) [upload=16414]: Upload finished. err: <nil> Duration: 5.894765721s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.872662647s FAIL (2025-10-28 09:53:15 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 09:20:59.362633688 (18h38m16.711940974s) [upload=16414]: Starting upload Oct 28 09:20:59.362695796 (18h38m16.712003082s) [upload=16414]: Done starting upload Oct 28 09:20:59.362893874 (18h38m16.71220115s) [upload=16414]: Uploading segments... Oct 28 09:20:59.362942907 (18h38m16.712250183s) [upload=16414]: Got next segment. Inline: false Oct 28 09:20:59.362982592 (18h38m16.712289878s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 28 09:20:59.362989917 (18h38m16.712297203s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 28 09:21:01.021344744 (18h38m18.37065203s) [seg_pos={0 0}, upload=16414]: Begin upload segment... Oct 28 09:21:01.023040152 (18h38m18.372347438s) [seg_pos={0 0}, upload=16414]: Done begin upload segment. Oct 28 09:21:01.023120105 (18h38m18.372427391s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.023839364 (18h38m18.37314665s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024279452 (18h38m18.373586778s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024909652 (18h38m18.374216938s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025349159 (18h38m18.374656446s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025638280 (18h38m18.374945566s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.026032871 (18h38m18.375340157s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.027098981 (18h38m18.376406267s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028007251 (18h38m18.377314537s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028880052 (18h38m18.378187339s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029267360 (18h38m18.378574646s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029838027 (18h38m18.379145313s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030217620 (18h38m18.379524906s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030834975 (18h38m18.380142261s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030959723 (18h38m18.380267009s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.031485544 (18h38m18.38079283s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:02.782872101 (18h38m20.132179387s) [upload=16414]: Next returned nil segment Oct 28 09:21:02.782895556 (18h38m20.132202842s) [upload=16414]: Waiting for error group managing segments... Oct 28 09:21:03.066789825 (18h38m20.416097121s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.071416355 (18h38m20.420723641s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.072983429 (18h38m20.422290715s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.077582706 (18h38m20.426889992s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.078844368 (18h38m20.428151655s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.139198845 (18h38m20.488506131s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.141506871 (18h38m20.490814167s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.170875915 (18h38m20.520183201s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.222706510 (18h38m20.572013796s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.247291775 (18h38m20.596599062s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.259905544 (18h38m20.60921283s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.281169995 (18h38m20.630477281s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312400144 (18h38m20.66170743s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312442985 (18h38m20.661750271s) [seg_pos={0 0}, upload=16414]: Segment reached optimal threshold of 13 pieces. Oct 28 09:21:03.312804424 (18h38m20.66211172s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection Oct 28 09:21:03.312830723 (18h38m20.662138019s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312894104 (18h38m20.6622014s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 28 09:21:03.312902871 (18h38m20.662210157s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312912750 (18h38m20.662220126s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:40022->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 28 09:21:03.312923039 (18h38m20.662230335s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312959218 (18h38m20.662266504s) [seg_pos={0 0}, upload=16414]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 28 09:21:03.313075419 (18h38m20.662382705s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 28 09:21:03.313081431 (18h38m20.662388717s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 28 09:21:03.646425392 (18h38m20.995732688s) [upload=16414]: Done uploading segments... Oct 28 09:21:03.646449338 (18h38m20.995756624s) [upload=16414]: Upload finished. err: <nil> Duration: 5.872692884s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.799701995s FAIL (2025-10-28 09:52:40 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 09:20:59.362633688 (18h38m16.711940974s) [upload=16414]: Starting upload Oct 28 09:20:59.362695796 (18h38m16.712003082s) [upload=16414]: Done starting upload Oct 28 09:20:59.362893874 (18h38m16.71220115s) [upload=16414]: Uploading segments... Oct 28 09:20:59.362942907 (18h38m16.712250183s) [upload=16414]: Got next segment. Inline: false Oct 28 09:20:59.362982592 (18h38m16.712289878s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 28 09:20:59.362989917 (18h38m16.712297203s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 28 09:21:01.021344744 (18h38m18.37065203s) [seg_pos={0 0}, upload=16414]: Begin upload segment... Oct 28 09:21:01.023040152 (18h38m18.372347438s) [seg_pos={0 0}, upload=16414]: Done begin upload segment. Oct 28 09:21:01.023120105 (18h38m18.372427391s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.023839364 (18h38m18.37314665s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024279452 (18h38m18.373586778s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024909652 (18h38m18.374216938s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025349159 (18h38m18.374656446s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025638280 (18h38m18.374945566s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.026032871 (18h38m18.375340157s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.027098981 (18h38m18.376406267s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028007251 (18h38m18.377314537s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028880052 (18h38m18.378187339s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029267360 (18h38m18.378574646s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029838027 (18h38m18.379145313s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030217620 (18h38m18.379524906s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030834975 (18h38m18.380142261s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030959723 (18h38m18.380267009s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.031485544 (18h38m18.38079283s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:02.782872101 (18h38m20.132179387s) [upload=16414]: Next returned nil segment Oct 28 09:21:02.782895556 (18h38m20.132202842s) [upload=16414]: Waiting for error group managing segments... Oct 28 09:21:03.066789825 (18h38m20.416097121s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.071416355 (18h38m20.420723641s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.072983429 (18h38m20.422290715s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.077582706 (18h38m20.426889992s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.078844368 (18h38m20.428151655s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.139198845 (18h38m20.488506131s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.141506871 (18h38m20.490814167s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.170875915 (18h38m20.520183201s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.222706510 (18h38m20.572013796s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.247291775 (18h38m20.596599062s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.259905544 (18h38m20.60921283s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.281169995 (18h38m20.630477281s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312400144 (18h38m20.66170743s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312442985 (18h38m20.661750271s) [seg_pos={0 0}, upload=16414]: Segment reached optimal threshold of 13 pieces. Oct 28 09:21:03.312804424 (18h38m20.66211172s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection Oct 28 09:21:03.312830723 (18h38m20.662138019s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312894104 (18h38m20.6622014s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 28 09:21:03.312902871 (18h38m20.662210157s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312912750 (18h38m20.662220126s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:40022->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 28 09:21:03.312923039 (18h38m20.662230335s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312959218 (18h38m20.662266504s) [seg_pos={0 0}, upload=16414]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 28 09:21:03.313075419 (18h38m20.662382705s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 28 09:21:03.313081431 (18h38m20.662388717s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 28 09:21:03.646425392 (18h38m20.995732688s) [upload=16414]: Done uploading segments... Oct 28 09:21:03.646449338 (18h38m20.995756624s) [upload=16414]: Upload finished. err: <nil> Duration: 5.799730548s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.997161131s FAIL (2025-10-28 09:51:33 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 09:20:59.362633688 (18h38m16.711940974s) [upload=16414]: Starting upload Oct 28 09:20:59.362695796 (18h38m16.712003082s) [upload=16414]: Done starting upload Oct 28 09:20:59.362893874 (18h38m16.71220115s) [upload=16414]: Uploading segments... Oct 28 09:20:59.362942907 (18h38m16.712250183s) [upload=16414]: Got next segment. Inline: false Oct 28 09:20:59.362982592 (18h38m16.712289878s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 28 09:20:59.362989917 (18h38m16.712297203s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 28 09:21:01.021344744 (18h38m18.37065203s) [seg_pos={0 0}, upload=16414]: Begin upload segment... Oct 28 09:21:01.023040152 (18h38m18.372347438s) [seg_pos={0 0}, upload=16414]: Done begin upload segment. Oct 28 09:21:01.023120105 (18h38m18.372427391s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.023839364 (18h38m18.37314665s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024279452 (18h38m18.373586778s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024909652 (18h38m18.374216938s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025349159 (18h38m18.374656446s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025638280 (18h38m18.374945566s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.026032871 (18h38m18.375340157s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.027098981 (18h38m18.376406267s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028007251 (18h38m18.377314537s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028880052 (18h38m18.378187339s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029267360 (18h38m18.378574646s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029838027 (18h38m18.379145313s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030217620 (18h38m18.379524906s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030834975 (18h38m18.380142261s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030959723 (18h38m18.380267009s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.031485544 (18h38m18.38079283s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:02.782872101 (18h38m20.132179387s) [upload=16414]: Next returned nil segment Oct 28 09:21:02.782895556 (18h38m20.132202842s) [upload=16414]: Waiting for error group managing segments... Oct 28 09:21:03.066789825 (18h38m20.416097121s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.071416355 (18h38m20.420723641s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.072983429 (18h38m20.422290715s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.077582706 (18h38m20.426889992s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.078844368 (18h38m20.428151655s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.139198845 (18h38m20.488506131s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.141506871 (18h38m20.490814167s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.170875915 (18h38m20.520183201s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.222706510 (18h38m20.572013796s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.247291775 (18h38m20.596599062s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.259905544 (18h38m20.60921283s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.281169995 (18h38m20.630477281s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312400144 (18h38m20.66170743s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312442985 (18h38m20.661750271s) [seg_pos={0 0}, upload=16414]: Segment reached optimal threshold of 13 pieces. Oct 28 09:21:03.312804424 (18h38m20.66211172s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection Oct 28 09:21:03.312830723 (18h38m20.662138019s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312894104 (18h38m20.6622014s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 28 09:21:03.312902871 (18h38m20.662210157s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312912750 (18h38m20.662220126s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:40022->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 28 09:21:03.312923039 (18h38m20.662230335s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312959218 (18h38m20.662266504s) [seg_pos={0 0}, upload=16414]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 28 09:21:03.313075419 (18h38m20.662382705s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 28 09:21:03.313081431 (18h38m20.662388717s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 28 09:21:03.646425392 (18h38m20.995732688s) [upload=16414]: Done uploading segments... Oct 28 09:21:03.646449338 (18h38m20.995756624s) [upload=16414]: Upload finished. err: <nil> Duration: 5.997188955s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.961031239s FAIL (2025-10-28 09:50:50 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 09:20:59.362633688 (18h38m16.711940974s) [upload=16414]: Starting upload Oct 28 09:20:59.362695796 (18h38m16.712003082s) [upload=16414]: Done starting upload Oct 28 09:20:59.362893874 (18h38m16.71220115s) [upload=16414]: Uploading segments... Oct 28 09:20:59.362942907 (18h38m16.712250183s) [upload=16414]: Got next segment. Inline: false Oct 28 09:20:59.362982592 (18h38m16.712289878s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 28 09:20:59.362989917 (18h38m16.712297203s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 28 09:21:01.021344744 (18h38m18.37065203s) [seg_pos={0 0}, upload=16414]: Begin upload segment... Oct 28 09:21:01.023040152 (18h38m18.372347438s) [seg_pos={0 0}, upload=16414]: Done begin upload segment. Oct 28 09:21:01.023120105 (18h38m18.372427391s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.023839364 (18h38m18.37314665s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024279452 (18h38m18.373586778s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024909652 (18h38m18.374216938s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025349159 (18h38m18.374656446s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025638280 (18h38m18.374945566s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.026032871 (18h38m18.375340157s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.027098981 (18h38m18.376406267s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028007251 (18h38m18.377314537s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028880052 (18h38m18.378187339s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029267360 (18h38m18.378574646s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029838027 (18h38m18.379145313s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030217620 (18h38m18.379524906s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030834975 (18h38m18.380142261s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030959723 (18h38m18.380267009s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.031485544 (18h38m18.38079283s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:02.782872101 (18h38m20.132179387s) [upload=16414]: Next returned nil segment Oct 28 09:21:02.782895556 (18h38m20.132202842s) [upload=16414]: Waiting for error group managing segments... Oct 28 09:21:03.066789825 (18h38m20.416097121s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.071416355 (18h38m20.420723641s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.072983429 (18h38m20.422290715s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.077582706 (18h38m20.426889992s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.078844368 (18h38m20.428151655s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.139198845 (18h38m20.488506131s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.141506871 (18h38m20.490814167s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.170875915 (18h38m20.520183201s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.222706510 (18h38m20.572013796s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.247291775 (18h38m20.596599062s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.259905544 (18h38m20.60921283s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.281169995 (18h38m20.630477281s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312400144 (18h38m20.66170743s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312442985 (18h38m20.661750271s) [seg_pos={0 0}, upload=16414]: Segment reached optimal threshold of 13 pieces. Oct 28 09:21:03.312804424 (18h38m20.66211172s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection Oct 28 09:21:03.312830723 (18h38m20.662138019s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312894104 (18h38m20.6622014s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 28 09:21:03.312902871 (18h38m20.662210157s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312912750 (18h38m20.662220126s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:40022->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 28 09:21:03.312923039 (18h38m20.662230335s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312959218 (18h38m20.662266504s) [seg_pos={0 0}, upload=16414]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 28 09:21:03.313075419 (18h38m20.662382705s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 28 09:21:03.313081431 (18h38m20.662388717s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 28 09:21:03.646425392 (18h38m20.995732688s) [upload=16414]: Done uploading segments... Oct 28 09:21:03.646449338 (18h38m20.995756624s) [upload=16414]: Upload finished. err: <nil> Duration: 5.96106283s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.975692309s FAIL (2025-10-28 09:50:21 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 09:20:59.362633688 (18h38m16.711940974s) [upload=16414]: Starting upload Oct 28 09:20:59.362695796 (18h38m16.712003082s) [upload=16414]: Done starting upload Oct 28 09:20:59.362893874 (18h38m16.71220115s) [upload=16414]: Uploading segments... Oct 28 09:20:59.362942907 (18h38m16.712250183s) [upload=16414]: Got next segment. Inline: false Oct 28 09:20:59.362982592 (18h38m16.712289878s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 28 09:20:59.362989917 (18h38m16.712297203s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 28 09:21:01.021344744 (18h38m18.37065203s) [seg_pos={0 0}, upload=16414]: Begin upload segment... Oct 28 09:21:01.023040152 (18h38m18.372347438s) [seg_pos={0 0}, upload=16414]: Done begin upload segment. Oct 28 09:21:01.023120105 (18h38m18.372427391s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.023839364 (18h38m18.37314665s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024279452 (18h38m18.373586778s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024909652 (18h38m18.374216938s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025349159 (18h38m18.374656446s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025638280 (18h38m18.374945566s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.026032871 (18h38m18.375340157s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.027098981 (18h38m18.376406267s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028007251 (18h38m18.377314537s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028880052 (18h38m18.378187339s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029267360 (18h38m18.378574646s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029838027 (18h38m18.379145313s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030217620 (18h38m18.379524906s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030834975 (18h38m18.380142261s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030959723 (18h38m18.380267009s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.031485544 (18h38m18.38079283s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:02.782872101 (18h38m20.132179387s) [upload=16414]: Next returned nil segment Oct 28 09:21:02.782895556 (18h38m20.132202842s) [upload=16414]: Waiting for error group managing segments... Oct 28 09:21:03.066789825 (18h38m20.416097121s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.071416355 (18h38m20.420723641s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.072983429 (18h38m20.422290715s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.077582706 (18h38m20.426889992s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.078844368 (18h38m20.428151655s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.139198845 (18h38m20.488506131s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.141506871 (18h38m20.490814167s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.170875915 (18h38m20.520183201s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.222706510 (18h38m20.572013796s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.247291775 (18h38m20.596599062s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.259905544 (18h38m20.60921283s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.281169995 (18h38m20.630477281s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312400144 (18h38m20.66170743s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312442985 (18h38m20.661750271s) [seg_pos={0 0}, upload=16414]: Segment reached optimal threshold of 13 pieces. Oct 28 09:21:03.312804424 (18h38m20.66211172s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection Oct 28 09:21:03.312830723 (18h38m20.662138019s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312894104 (18h38m20.6622014s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 28 09:21:03.312902871 (18h38m20.662210157s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312912750 (18h38m20.662220126s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:40022->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 28 09:21:03.312923039 (18h38m20.662230335s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312959218 (18h38m20.662266504s) [seg_pos={0 0}, upload=16414]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 28 09:21:03.313075419 (18h38m20.662382705s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 28 09:21:03.313081431 (18h38m20.662388717s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 28 09:21:03.646425392 (18h38m20.995732688s) [upload=16414]: Done uploading segments... Oct 28 09:21:03.646449338 (18h38m20.995756624s) [upload=16414]: Upload finished. err: <nil> Duration: 5.97572333s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 6.058716835s FAIL (2025-10-28 09:49:59 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 09:20:59.362633688 (18h38m16.711940974s) [upload=16414]: Starting upload Oct 28 09:20:59.362695796 (18h38m16.712003082s) [upload=16414]: Done starting upload Oct 28 09:20:59.362893874 (18h38m16.71220115s) [upload=16414]: Uploading segments... Oct 28 09:20:59.362942907 (18h38m16.712250183s) [upload=16414]: Got next segment. Inline: false Oct 28 09:20:59.362982592 (18h38m16.712289878s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 28 09:20:59.362989917 (18h38m16.712297203s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 28 09:21:01.021344744 (18h38m18.37065203s) [seg_pos={0 0}, upload=16414]: Begin upload segment... Oct 28 09:21:01.023040152 (18h38m18.372347438s) [seg_pos={0 0}, upload=16414]: Done begin upload segment. Oct 28 09:21:01.023120105 (18h38m18.372427391s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.023839364 (18h38m18.37314665s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024279452 (18h38m18.373586778s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024909652 (18h38m18.374216938s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025349159 (18h38m18.374656446s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025638280 (18h38m18.374945566s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.026032871 (18h38m18.375340157s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.027098981 (18h38m18.376406267s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028007251 (18h38m18.377314537s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028880052 (18h38m18.378187339s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029267360 (18h38m18.378574646s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029838027 (18h38m18.379145313s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030217620 (18h38m18.379524906s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030834975 (18h38m18.380142261s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030959723 (18h38m18.380267009s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.031485544 (18h38m18.38079283s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:02.782872101 (18h38m20.132179387s) [upload=16414]: Next returned nil segment Oct 28 09:21:02.782895556 (18h38m20.132202842s) [upload=16414]: Waiting for error group managing segments... Oct 28 09:21:03.066789825 (18h38m20.416097121s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.071416355 (18h38m20.420723641s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.072983429 (18h38m20.422290715s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.077582706 (18h38m20.426889992s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.078844368 (18h38m20.428151655s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.139198845 (18h38m20.488506131s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.141506871 (18h38m20.490814167s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.170875915 (18h38m20.520183201s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.222706510 (18h38m20.572013796s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.247291775 (18h38m20.596599062s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.259905544 (18h38m20.60921283s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.281169995 (18h38m20.630477281s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312400144 (18h38m20.66170743s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312442985 (18h38m20.661750271s) [seg_pos={0 0}, upload=16414]: Segment reached optimal threshold of 13 pieces. Oct 28 09:21:03.312804424 (18h38m20.66211172s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection Oct 28 09:21:03.312830723 (18h38m20.662138019s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312894104 (18h38m20.6622014s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 28 09:21:03.312902871 (18h38m20.662210157s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312912750 (18h38m20.662220126s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:40022->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 28 09:21:03.312923039 (18h38m20.662230335s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312959218 (18h38m20.662266504s) [seg_pos={0 0}, upload=16414]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 28 09:21:03.313075419 (18h38m20.662382705s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 28 09:21:03.313081431 (18h38m20.662388717s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 28 09:21:03.646425392 (18h38m20.995732688s) [upload=16414]: Done uploading segments... Oct 28 09:21:03.646449338 (18h38m20.995756624s) [upload=16414]: Upload finished. err: <nil> Duration: 6.058760508s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 6.026368456s FAIL (2025-10-28 09:49:21 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 09:20:59.362633688 (18h38m16.711940974s) [upload=16414]: Starting upload Oct 28 09:20:59.362695796 (18h38m16.712003082s) [upload=16414]: Done starting upload Oct 28 09:20:59.362893874 (18h38m16.71220115s) [upload=16414]: Uploading segments... Oct 28 09:20:59.362942907 (18h38m16.712250183s) [upload=16414]: Got next segment. Inline: false Oct 28 09:20:59.362982592 (18h38m16.712289878s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 28 09:20:59.362989917 (18h38m16.712297203s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 28 09:21:01.021344744 (18h38m18.37065203s) [seg_pos={0 0}, upload=16414]: Begin upload segment... Oct 28 09:21:01.023040152 (18h38m18.372347438s) [seg_pos={0 0}, upload=16414]: Done begin upload segment. Oct 28 09:21:01.023120105 (18h38m18.372427391s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.023839364 (18h38m18.37314665s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024279452 (18h38m18.373586778s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024909652 (18h38m18.374216938s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025349159 (18h38m18.374656446s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025638280 (18h38m18.374945566s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.026032871 (18h38m18.375340157s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.027098981 (18h38m18.376406267s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028007251 (18h38m18.377314537s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028880052 (18h38m18.378187339s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029267360 (18h38m18.378574646s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029838027 (18h38m18.379145313s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030217620 (18h38m18.379524906s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030834975 (18h38m18.380142261s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030959723 (18h38m18.380267009s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.031485544 (18h38m18.38079283s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:02.782872101 (18h38m20.132179387s) [upload=16414]: Next returned nil segment Oct 28 09:21:02.782895556 (18h38m20.132202842s) [upload=16414]: Waiting for error group managing segments... Oct 28 09:21:03.066789825 (18h38m20.416097121s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.071416355 (18h38m20.420723641s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.072983429 (18h38m20.422290715s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.077582706 (18h38m20.426889992s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.078844368 (18h38m20.428151655s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.139198845 (18h38m20.488506131s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.141506871 (18h38m20.490814167s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.170875915 (18h38m20.520183201s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.222706510 (18h38m20.572013796s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.247291775 (18h38m20.596599062s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.259905544 (18h38m20.60921283s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.281169995 (18h38m20.630477281s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312400144 (18h38m20.66170743s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312442985 (18h38m20.661750271s) [seg_pos={0 0}, upload=16414]: Segment reached optimal threshold of 13 pieces. Oct 28 09:21:03.312804424 (18h38m20.66211172s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection Oct 28 09:21:03.312830723 (18h38m20.662138019s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312894104 (18h38m20.6622014s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 28 09:21:03.312902871 (18h38m20.662210157s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312912750 (18h38m20.662220126s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:40022->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 28 09:21:03.312923039 (18h38m20.662230335s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312959218 (18h38m20.662266504s) [seg_pos={0 0}, upload=16414]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 28 09:21:03.313075419 (18h38m20.662382705s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 28 09:21:03.313081431 (18h38m20.662388717s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 28 09:21:03.646425392 (18h38m20.995732688s) [upload=16414]: Done uploading segments... Oct 28 09:21:03.646449338 (18h38m20.995756624s) [upload=16414]: Upload finished. err: <nil> Duration: 6.026393383s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.272490562s FAIL (2025-10-28 09:48:32 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 09:20:59.362633688 (18h38m16.711940974s) [upload=16414]: Starting upload Oct 28 09:20:59.362695796 (18h38m16.712003082s) [upload=16414]: Done starting upload Oct 28 09:20:59.362893874 (18h38m16.71220115s) [upload=16414]: Uploading segments... Oct 28 09:20:59.362942907 (18h38m16.712250183s) [upload=16414]: Got next segment. Inline: false Oct 28 09:20:59.362982592 (18h38m16.712289878s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 28 09:20:59.362989917 (18h38m16.712297203s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 28 09:21:01.021344744 (18h38m18.37065203s) [seg_pos={0 0}, upload=16414]: Begin upload segment... Oct 28 09:21:01.023040152 (18h38m18.372347438s) [seg_pos={0 0}, upload=16414]: Done begin upload segment. Oct 28 09:21:01.023120105 (18h38m18.372427391s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.023839364 (18h38m18.37314665s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024279452 (18h38m18.373586778s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024909652 (18h38m18.374216938s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025349159 (18h38m18.374656446s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025638280 (18h38m18.374945566s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.026032871 (18h38m18.375340157s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.027098981 (18h38m18.376406267s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028007251 (18h38m18.377314537s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028880052 (18h38m18.378187339s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029267360 (18h38m18.378574646s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029838027 (18h38m18.379145313s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030217620 (18h38m18.379524906s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030834975 (18h38m18.380142261s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030959723 (18h38m18.380267009s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.031485544 (18h38m18.38079283s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:02.782872101 (18h38m20.132179387s) [upload=16414]: Next returned nil segment Oct 28 09:21:02.782895556 (18h38m20.132202842s) [upload=16414]: Waiting for error group managing segments... Oct 28 09:21:03.066789825 (18h38m20.416097121s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.071416355 (18h38m20.420723641s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.072983429 (18h38m20.422290715s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.077582706 (18h38m20.426889992s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.078844368 (18h38m20.428151655s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.139198845 (18h38m20.488506131s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.141506871 (18h38m20.490814167s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.170875915 (18h38m20.520183201s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.222706510 (18h38m20.572013796s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.247291775 (18h38m20.596599062s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.259905544 (18h38m20.60921283s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.281169995 (18h38m20.630477281s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312400144 (18h38m20.66170743s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312442985 (18h38m20.661750271s) [seg_pos={0 0}, upload=16414]: Segment reached optimal threshold of 13 pieces. Oct 28 09:21:03.312804424 (18h38m20.66211172s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection Oct 28 09:21:03.312830723 (18h38m20.662138019s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312894104 (18h38m20.6622014s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 28 09:21:03.312902871 (18h38m20.662210157s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312912750 (18h38m20.662220126s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:40022->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 28 09:21:03.312923039 (18h38m20.662230335s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312959218 (18h38m20.662266504s) [seg_pos={0 0}, upload=16414]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 28 09:21:03.313075419 (18h38m20.662382705s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 28 09:21:03.313081431 (18h38m20.662388717s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 28 09:21:03.646425392 (18h38m20.995732688s) [upload=16414]: Done uploading segments... Oct 28 09:21:03.646449338 (18h38m20.995756624s) [upload=16414]: Upload finished. err: <nil> Duration: 6.272521101s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.970282201s FAIL (2025-10-28 09:48:02 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 09:20:59.362633688 (18h38m16.711940974s) [upload=16414]: Starting upload Oct 28 09:20:59.362695796 (18h38m16.712003082s) [upload=16414]: Done starting upload Oct 28 09:20:59.362893874 (18h38m16.71220115s) [upload=16414]: Uploading segments... Oct 28 09:20:59.362942907 (18h38m16.712250183s) [upload=16414]: Got next segment. Inline: false Oct 28 09:20:59.362982592 (18h38m16.712289878s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 28 09:20:59.362989917 (18h38m16.712297203s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 28 09:21:01.021344744 (18h38m18.37065203s) [seg_pos={0 0}, upload=16414]: Begin upload segment... Oct 28 09:21:01.023040152 (18h38m18.372347438s) [seg_pos={0 0}, upload=16414]: Done begin upload segment. Oct 28 09:21:01.023120105 (18h38m18.372427391s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.023839364 (18h38m18.37314665s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024279452 (18h38m18.373586778s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024909652 (18h38m18.374216938s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025349159 (18h38m18.374656446s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025638280 (18h38m18.374945566s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.026032871 (18h38m18.375340157s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.027098981 (18h38m18.376406267s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028007251 (18h38m18.377314537s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028880052 (18h38m18.378187339s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029267360 (18h38m18.378574646s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029838027 (18h38m18.379145313s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030217620 (18h38m18.379524906s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030834975 (18h38m18.380142261s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030959723 (18h38m18.380267009s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.031485544 (18h38m18.38079283s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:02.782872101 (18h38m20.132179387s) [upload=16414]: Next returned nil segment Oct 28 09:21:02.782895556 (18h38m20.132202842s) [upload=16414]: Waiting for error group managing segments... Oct 28 09:21:03.066789825 (18h38m20.416097121s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.071416355 (18h38m20.420723641s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.072983429 (18h38m20.422290715s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.077582706 (18h38m20.426889992s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.078844368 (18h38m20.428151655s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.139198845 (18h38m20.488506131s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.141506871 (18h38m20.490814167s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.170875915 (18h38m20.520183201s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.222706510 (18h38m20.572013796s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.247291775 (18h38m20.596599062s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.259905544 (18h38m20.60921283s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.281169995 (18h38m20.630477281s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312400144 (18h38m20.66170743s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312442985 (18h38m20.661750271s) [seg_pos={0 0}, upload=16414]: Segment reached optimal threshold of 13 pieces. Oct 28 09:21:03.312804424 (18h38m20.66211172s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection Oct 28 09:21:03.312830723 (18h38m20.662138019s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312894104 (18h38m20.6622014s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 28 09:21:03.312902871 (18h38m20.662210157s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312912750 (18h38m20.662220126s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:40022->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 28 09:21:03.312923039 (18h38m20.662230335s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312959218 (18h38m20.662266504s) [seg_pos={0 0}, upload=16414]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 28 09:21:03.313075419 (18h38m20.662382705s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 28 09:21:03.313081431 (18h38m20.662388717s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 28 09:21:03.646425392 (18h38m20.995732688s) [upload=16414]: Done uploading segments... Oct 28 09:21:03.646449338 (18h38m20.995756624s) [upload=16414]: Upload finished. err: <nil> Duration: 5.970453557s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.787974479s FAIL (2025-10-28 09:47:13 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 09:20:59.362633688 (18h38m16.711940974s) [upload=16414]: Starting upload Oct 28 09:20:59.362695796 (18h38m16.712003082s) [upload=16414]: Done starting upload Oct 28 09:20:59.362893874 (18h38m16.71220115s) [upload=16414]: Uploading segments... Oct 28 09:20:59.362942907 (18h38m16.712250183s) [upload=16414]: Got next segment. Inline: false Oct 28 09:20:59.362982592 (18h38m16.712289878s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 28 09:20:59.362989917 (18h38m16.712297203s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 28 09:21:01.021344744 (18h38m18.37065203s) [seg_pos={0 0}, upload=16414]: Begin upload segment... Oct 28 09:21:01.023040152 (18h38m18.372347438s) [seg_pos={0 0}, upload=16414]: Done begin upload segment. Oct 28 09:21:01.023120105 (18h38m18.372427391s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.023839364 (18h38m18.37314665s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024279452 (18h38m18.373586778s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024909652 (18h38m18.374216938s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025349159 (18h38m18.374656446s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025638280 (18h38m18.374945566s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.026032871 (18h38m18.375340157s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.027098981 (18h38m18.376406267s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028007251 (18h38m18.377314537s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028880052 (18h38m18.378187339s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029267360 (18h38m18.378574646s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029838027 (18h38m18.379145313s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030217620 (18h38m18.379524906s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030834975 (18h38m18.380142261s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030959723 (18h38m18.380267009s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.031485544 (18h38m18.38079283s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:02.782872101 (18h38m20.132179387s) [upload=16414]: Next returned nil segment Oct 28 09:21:02.782895556 (18h38m20.132202842s) [upload=16414]: Waiting for error group managing segments... Oct 28 09:21:03.066789825 (18h38m20.416097121s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.071416355 (18h38m20.420723641s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.072983429 (18h38m20.422290715s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.077582706 (18h38m20.426889992s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.078844368 (18h38m20.428151655s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.139198845 (18h38m20.488506131s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.141506871 (18h38m20.490814167s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.170875915 (18h38m20.520183201s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.222706510 (18h38m20.572013796s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.247291775 (18h38m20.596599062s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.259905544 (18h38m20.60921283s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.281169995 (18h38m20.630477281s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312400144 (18h38m20.66170743s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312442985 (18h38m20.661750271s) [seg_pos={0 0}, upload=16414]: Segment reached optimal threshold of 13 pieces. Oct 28 09:21:03.312804424 (18h38m20.66211172s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection Oct 28 09:21:03.312830723 (18h38m20.662138019s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312894104 (18h38m20.6622014s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 28 09:21:03.312902871 (18h38m20.662210157s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312912750 (18h38m20.662220126s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:40022->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 28 09:21:03.312923039 (18h38m20.662230335s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312959218 (18h38m20.662266504s) [seg_pos={0 0}, upload=16414]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 28 09:21:03.313075419 (18h38m20.662382705s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 28 09:21:03.313081431 (18h38m20.662388717s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 28 09:21:03.646425392 (18h38m20.995732688s) [upload=16414]: Done uploading segments... Oct 28 09:21:03.646449338 (18h38m20.995756624s) [upload=16414]: Upload finished. err: <nil> Duration: 5.788001351s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.974423642s FAIL (2025-10-28 09:46:52 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 09:20:59.362633688 (18h38m16.711940974s) [upload=16414]: Starting upload Oct 28 09:20:59.362695796 (18h38m16.712003082s) [upload=16414]: Done starting upload Oct 28 09:20:59.362893874 (18h38m16.71220115s) [upload=16414]: Uploading segments... Oct 28 09:20:59.362942907 (18h38m16.712250183s) [upload=16414]: Got next segment. Inline: false Oct 28 09:20:59.362982592 (18h38m16.712289878s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 28 09:20:59.362989917 (18h38m16.712297203s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 28 09:21:01.021344744 (18h38m18.37065203s) [seg_pos={0 0}, upload=16414]: Begin upload segment... Oct 28 09:21:01.023040152 (18h38m18.372347438s) [seg_pos={0 0}, upload=16414]: Done begin upload segment. Oct 28 09:21:01.023120105 (18h38m18.372427391s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.023839364 (18h38m18.37314665s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024279452 (18h38m18.373586778s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024909652 (18h38m18.374216938s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025349159 (18h38m18.374656446s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025638280 (18h38m18.374945566s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.026032871 (18h38m18.375340157s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.027098981 (18h38m18.376406267s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028007251 (18h38m18.377314537s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028880052 (18h38m18.378187339s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029267360 (18h38m18.378574646s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029838027 (18h38m18.379145313s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030217620 (18h38m18.379524906s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030834975 (18h38m18.380142261s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030959723 (18h38m18.380267009s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.031485544 (18h38m18.38079283s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:02.782872101 (18h38m20.132179387s) [upload=16414]: Next returned nil segment Oct 28 09:21:02.782895556 (18h38m20.132202842s) [upload=16414]: Waiting for error group managing segments... Oct 28 09:21:03.066789825 (18h38m20.416097121s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.071416355 (18h38m20.420723641s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.072983429 (18h38m20.422290715s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.077582706 (18h38m20.426889992s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.078844368 (18h38m20.428151655s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.139198845 (18h38m20.488506131s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.141506871 (18h38m20.490814167s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.170875915 (18h38m20.520183201s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.222706510 (18h38m20.572013796s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.247291775 (18h38m20.596599062s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.259905544 (18h38m20.60921283s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.281169995 (18h38m20.630477281s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312400144 (18h38m20.66170743s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312442985 (18h38m20.661750271s) [seg_pos={0 0}, upload=16414]: Segment reached optimal threshold of 13 pieces. Oct 28 09:21:03.312804424 (18h38m20.66211172s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection Oct 28 09:21:03.312830723 (18h38m20.662138019s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312894104 (18h38m20.6622014s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 28 09:21:03.312902871 (18h38m20.662210157s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312912750 (18h38m20.662220126s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:40022->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 28 09:21:03.312923039 (18h38m20.662230335s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312959218 (18h38m20.662266504s) [seg_pos={0 0}, upload=16414]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 28 09:21:03.313075419 (18h38m20.662382705s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 28 09:21:03.313081431 (18h38m20.662388717s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 28 09:21:03.646425392 (18h38m20.995732688s) [upload=16414]: Done uploading segments... Oct 28 09:21:03.646449338 (18h38m20.995756624s) [upload=16414]: Upload finished. err: <nil> Duration: 5.974451214s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.901687307s FAIL (2025-10-28 09:45:46 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 09:20:59.362633688 (18h38m16.711940974s) [upload=16414]: Starting upload Oct 28 09:20:59.362695796 (18h38m16.712003082s) [upload=16414]: Done starting upload Oct 28 09:20:59.362893874 (18h38m16.71220115s) [upload=16414]: Uploading segments... Oct 28 09:20:59.362942907 (18h38m16.712250183s) [upload=16414]: Got next segment. Inline: false Oct 28 09:20:59.362982592 (18h38m16.712289878s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 28 09:20:59.362989917 (18h38m16.712297203s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 28 09:21:01.021344744 (18h38m18.37065203s) [seg_pos={0 0}, upload=16414]: Begin upload segment... Oct 28 09:21:01.023040152 (18h38m18.372347438s) [seg_pos={0 0}, upload=16414]: Done begin upload segment. Oct 28 09:21:01.023120105 (18h38m18.372427391s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.023839364 (18h38m18.37314665s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024279452 (18h38m18.373586778s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024909652 (18h38m18.374216938s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025349159 (18h38m18.374656446s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025638280 (18h38m18.374945566s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.026032871 (18h38m18.375340157s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.027098981 (18h38m18.376406267s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028007251 (18h38m18.377314537s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028880052 (18h38m18.378187339s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029267360 (18h38m18.378574646s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029838027 (18h38m18.379145313s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030217620 (18h38m18.379524906s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030834975 (18h38m18.380142261s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030959723 (18h38m18.380267009s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.031485544 (18h38m18.38079283s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:02.782872101 (18h38m20.132179387s) [upload=16414]: Next returned nil segment Oct 28 09:21:02.782895556 (18h38m20.132202842s) [upload=16414]: Waiting for error group managing segments... Oct 28 09:21:03.066789825 (18h38m20.416097121s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.071416355 (18h38m20.420723641s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.072983429 (18h38m20.422290715s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.077582706 (18h38m20.426889992s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.078844368 (18h38m20.428151655s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.139198845 (18h38m20.488506131s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.141506871 (18h38m20.490814167s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.170875915 (18h38m20.520183201s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.222706510 (18h38m20.572013796s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.247291775 (18h38m20.596599062s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.259905544 (18h38m20.60921283s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.281169995 (18h38m20.630477281s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312400144 (18h38m20.66170743s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312442985 (18h38m20.661750271s) [seg_pos={0 0}, upload=16414]: Segment reached optimal threshold of 13 pieces. Oct 28 09:21:03.312804424 (18h38m20.66211172s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection Oct 28 09:21:03.312830723 (18h38m20.662138019s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312894104 (18h38m20.6622014s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 28 09:21:03.312902871 (18h38m20.662210157s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312912750 (18h38m20.662220126s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:40022->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 28 09:21:03.312923039 (18h38m20.662230335s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312959218 (18h38m20.662266504s) [seg_pos={0 0}, upload=16414]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 28 09:21:03.313075419 (18h38m20.662382705s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 28 09:21:03.313081431 (18h38m20.662388717s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 28 09:21:03.646425392 (18h38m20.995732688s) [upload=16414]: Done uploading segments... Oct 28 09:21:03.646449338 (18h38m20.995756624s) [upload=16414]: Upload finished. err: <nil> Duration: 5.901717866s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.847448782s FAIL (2025-10-28 09:44:33 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 09:20:59.362633688 (18h38m16.711940974s) [upload=16414]: Starting upload Oct 28 09:20:59.362695796 (18h38m16.712003082s) [upload=16414]: Done starting upload Oct 28 09:20:59.362893874 (18h38m16.71220115s) [upload=16414]: Uploading segments... Oct 28 09:20:59.362942907 (18h38m16.712250183s) [upload=16414]: Got next segment. Inline: false Oct 28 09:20:59.362982592 (18h38m16.712289878s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 28 09:20:59.362989917 (18h38m16.712297203s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 28 09:21:01.021344744 (18h38m18.37065203s) [seg_pos={0 0}, upload=16414]: Begin upload segment... Oct 28 09:21:01.023040152 (18h38m18.372347438s) [seg_pos={0 0}, upload=16414]: Done begin upload segment. Oct 28 09:21:01.023120105 (18h38m18.372427391s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.023839364 (18h38m18.37314665s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024279452 (18h38m18.373586778s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024909652 (18h38m18.374216938s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025349159 (18h38m18.374656446s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025638280 (18h38m18.374945566s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.026032871 (18h38m18.375340157s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.027098981 (18h38m18.376406267s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028007251 (18h38m18.377314537s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028880052 (18h38m18.378187339s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029267360 (18h38m18.378574646s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029838027 (18h38m18.379145313s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030217620 (18h38m18.379524906s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030834975 (18h38m18.380142261s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030959723 (18h38m18.380267009s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.031485544 (18h38m18.38079283s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:02.782872101 (18h38m20.132179387s) [upload=16414]: Next returned nil segment Oct 28 09:21:02.782895556 (18h38m20.132202842s) [upload=16414]: Waiting for error group managing segments... Oct 28 09:21:03.066789825 (18h38m20.416097121s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.071416355 (18h38m20.420723641s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.072983429 (18h38m20.422290715s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.077582706 (18h38m20.426889992s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.078844368 (18h38m20.428151655s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.139198845 (18h38m20.488506131s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.141506871 (18h38m20.490814167s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.170875915 (18h38m20.520183201s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.222706510 (18h38m20.572013796s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.247291775 (18h38m20.596599062s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.259905544 (18h38m20.60921283s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.281169995 (18h38m20.630477281s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312400144 (18h38m20.66170743s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312442985 (18h38m20.661750271s) [seg_pos={0 0}, upload=16414]: Segment reached optimal threshold of 13 pieces. Oct 28 09:21:03.312804424 (18h38m20.66211172s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection Oct 28 09:21:03.312830723 (18h38m20.662138019s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312894104 (18h38m20.6622014s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 28 09:21:03.312902871 (18h38m20.662210157s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312912750 (18h38m20.662220126s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:40022->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 28 09:21:03.312923039 (18h38m20.662230335s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312959218 (18h38m20.662266504s) [seg_pos={0 0}, upload=16414]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 28 09:21:03.313075419 (18h38m20.662382705s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 28 09:21:03.313081431 (18h38m20.662388717s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 28 09:21:03.646425392 (18h38m20.995732688s) [upload=16414]: Done uploading segments... Oct 28 09:21:03.646449338 (18h38m20.995756624s) [upload=16414]: Upload finished. err: <nil> Duration: 5.847477106s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 6.395457273s FAIL (2025-10-28 09:43:38 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 09:20:59.362633688 (18h38m16.711940974s) [upload=16414]: Starting upload Oct 28 09:20:59.362695796 (18h38m16.712003082s) [upload=16414]: Done starting upload Oct 28 09:20:59.362893874 (18h38m16.71220115s) [upload=16414]: Uploading segments... Oct 28 09:20:59.362942907 (18h38m16.712250183s) [upload=16414]: Got next segment. Inline: false Oct 28 09:20:59.362982592 (18h38m16.712289878s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 28 09:20:59.362989917 (18h38m16.712297203s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 28 09:21:01.021344744 (18h38m18.37065203s) [seg_pos={0 0}, upload=16414]: Begin upload segment... Oct 28 09:21:01.023040152 (18h38m18.372347438s) [seg_pos={0 0}, upload=16414]: Done begin upload segment. Oct 28 09:21:01.023120105 (18h38m18.372427391s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.023839364 (18h38m18.37314665s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024279452 (18h38m18.373586778s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024909652 (18h38m18.374216938s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025349159 (18h38m18.374656446s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025638280 (18h38m18.374945566s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.026032871 (18h38m18.375340157s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.027098981 (18h38m18.376406267s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028007251 (18h38m18.377314537s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028880052 (18h38m18.378187339s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029267360 (18h38m18.378574646s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029838027 (18h38m18.379145313s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030217620 (18h38m18.379524906s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030834975 (18h38m18.380142261s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030959723 (18h38m18.380267009s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.031485544 (18h38m18.38079283s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:02.782872101 (18h38m20.132179387s) [upload=16414]: Next returned nil segment Oct 28 09:21:02.782895556 (18h38m20.132202842s) [upload=16414]: Waiting for error group managing segments... Oct 28 09:21:03.066789825 (18h38m20.416097121s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.071416355 (18h38m20.420723641s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.072983429 (18h38m20.422290715s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.077582706 (18h38m20.426889992s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.078844368 (18h38m20.428151655s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.139198845 (18h38m20.488506131s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.141506871 (18h38m20.490814167s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.170875915 (18h38m20.520183201s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.222706510 (18h38m20.572013796s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.247291775 (18h38m20.596599062s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.259905544 (18h38m20.60921283s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.281169995 (18h38m20.630477281s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312400144 (18h38m20.66170743s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312442985 (18h38m20.661750271s) [seg_pos={0 0}, upload=16414]: Segment reached optimal threshold of 13 pieces. Oct 28 09:21:03.312804424 (18h38m20.66211172s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection Oct 28 09:21:03.312830723 (18h38m20.662138019s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312894104 (18h38m20.6622014s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 28 09:21:03.312902871 (18h38m20.662210157s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312912750 (18h38m20.662220126s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:40022->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 28 09:21:03.312923039 (18h38m20.662230335s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312959218 (18h38m20.662266504s) [seg_pos={0 0}, upload=16414]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 28 09:21:03.313075419 (18h38m20.662382705s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 28 09:21:03.313081431 (18h38m20.662388717s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 28 09:21:03.646425392 (18h38m20.995732688s) [upload=16414]: Done uploading segments... Oct 28 09:21:03.646449338 (18h38m20.995756624s) [upload=16414]: Upload finished. err: <nil> Duration: 6.395572081s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.965477909s FAIL (2025-10-28 09:43:06 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 09:20:59.362633688 (18h38m16.711940974s) [upload=16414]: Starting upload Oct 28 09:20:59.362695796 (18h38m16.712003082s) [upload=16414]: Done starting upload Oct 28 09:20:59.362893874 (18h38m16.71220115s) [upload=16414]: Uploading segments... Oct 28 09:20:59.362942907 (18h38m16.712250183s) [upload=16414]: Got next segment. Inline: false Oct 28 09:20:59.362982592 (18h38m16.712289878s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 28 09:20:59.362989917 (18h38m16.712297203s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 28 09:21:01.021344744 (18h38m18.37065203s) [seg_pos={0 0}, upload=16414]: Begin upload segment... Oct 28 09:21:01.023040152 (18h38m18.372347438s) [seg_pos={0 0}, upload=16414]: Done begin upload segment. Oct 28 09:21:01.023120105 (18h38m18.372427391s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.023839364 (18h38m18.37314665s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024279452 (18h38m18.373586778s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024909652 (18h38m18.374216938s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025349159 (18h38m18.374656446s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025638280 (18h38m18.374945566s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.026032871 (18h38m18.375340157s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.027098981 (18h38m18.376406267s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028007251 (18h38m18.377314537s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028880052 (18h38m18.378187339s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029267360 (18h38m18.378574646s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029838027 (18h38m18.379145313s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030217620 (18h38m18.379524906s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030834975 (18h38m18.380142261s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030959723 (18h38m18.380267009s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.031485544 (18h38m18.38079283s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:02.782872101 (18h38m20.132179387s) [upload=16414]: Next returned nil segment Oct 28 09:21:02.782895556 (18h38m20.132202842s) [upload=16414]: Waiting for error group managing segments... Oct 28 09:21:03.066789825 (18h38m20.416097121s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.071416355 (18h38m20.420723641s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.072983429 (18h38m20.422290715s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.077582706 (18h38m20.426889992s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.078844368 (18h38m20.428151655s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.139198845 (18h38m20.488506131s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.141506871 (18h38m20.490814167s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.170875915 (18h38m20.520183201s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.222706510 (18h38m20.572013796s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.247291775 (18h38m20.596599062s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.259905544 (18h38m20.60921283s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.281169995 (18h38m20.630477281s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312400144 (18h38m20.66170743s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312442985 (18h38m20.661750271s) [seg_pos={0 0}, upload=16414]: Segment reached optimal threshold of 13 pieces. Oct 28 09:21:03.312804424 (18h38m20.66211172s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection Oct 28 09:21:03.312830723 (18h38m20.662138019s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312894104 (18h38m20.6622014s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 28 09:21:03.312902871 (18h38m20.662210157s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312912750 (18h38m20.662220126s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:40022->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 28 09:21:03.312923039 (18h38m20.662230335s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312959218 (18h38m20.662266504s) [seg_pos={0 0}, upload=16414]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 28 09:21:03.313075419 (18h38m20.662382705s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 28 09:21:03.313081431 (18h38m20.662388717s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 28 09:21:03.646425392 (18h38m20.995732688s) [upload=16414]: Done uploading segments... Oct 28 09:21:03.646449338 (18h38m20.995756624s) [upload=16414]: Upload finished. err: <nil> Duration: 5.965512857s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.920797627s FAIL (2025-10-28 09:42:06 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 09:20:59.362633688 (18h38m16.711940974s) [upload=16414]: Starting upload Oct 28 09:20:59.362695796 (18h38m16.712003082s) [upload=16414]: Done starting upload Oct 28 09:20:59.362893874 (18h38m16.71220115s) [upload=16414]: Uploading segments... Oct 28 09:20:59.362942907 (18h38m16.712250183s) [upload=16414]: Got next segment. Inline: false Oct 28 09:20:59.362982592 (18h38m16.712289878s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 28 09:20:59.362989917 (18h38m16.712297203s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 28 09:21:01.021344744 (18h38m18.37065203s) [seg_pos={0 0}, upload=16414]: Begin upload segment... Oct 28 09:21:01.023040152 (18h38m18.372347438s) [seg_pos={0 0}, upload=16414]: Done begin upload segment. Oct 28 09:21:01.023120105 (18h38m18.372427391s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.023839364 (18h38m18.37314665s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024279452 (18h38m18.373586778s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024909652 (18h38m18.374216938s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025349159 (18h38m18.374656446s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025638280 (18h38m18.374945566s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.026032871 (18h38m18.375340157s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.027098981 (18h38m18.376406267s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028007251 (18h38m18.377314537s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028880052 (18h38m18.378187339s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029267360 (18h38m18.378574646s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029838027 (18h38m18.379145313s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030217620 (18h38m18.379524906s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030834975 (18h38m18.380142261s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030959723 (18h38m18.380267009s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.031485544 (18h38m18.38079283s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:02.782872101 (18h38m20.132179387s) [upload=16414]: Next returned nil segment Oct 28 09:21:02.782895556 (18h38m20.132202842s) [upload=16414]: Waiting for error group managing segments... Oct 28 09:21:03.066789825 (18h38m20.416097121s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.071416355 (18h38m20.420723641s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.072983429 (18h38m20.422290715s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.077582706 (18h38m20.426889992s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.078844368 (18h38m20.428151655s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.139198845 (18h38m20.488506131s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.141506871 (18h38m20.490814167s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.170875915 (18h38m20.520183201s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.222706510 (18h38m20.572013796s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.247291775 (18h38m20.596599062s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.259905544 (18h38m20.60921283s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.281169995 (18h38m20.630477281s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312400144 (18h38m20.66170743s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312442985 (18h38m20.661750271s) [seg_pos={0 0}, upload=16414]: Segment reached optimal threshold of 13 pieces. Oct 28 09:21:03.312804424 (18h38m20.66211172s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection Oct 28 09:21:03.312830723 (18h38m20.662138019s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312894104 (18h38m20.6622014s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 28 09:21:03.312902871 (18h38m20.662210157s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312912750 (18h38m20.662220126s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:40022->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 28 09:21:03.312923039 (18h38m20.662230335s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312959218 (18h38m20.662266504s) [seg_pos={0 0}, upload=16414]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 28 09:21:03.313075419 (18h38m20.662382705s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 28 09:21:03.313081431 (18h38m20.662388717s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 28 09:21:03.646425392 (18h38m20.995732688s) [upload=16414]: Done uploading segments... Oct 28 09:21:03.646449338 (18h38m20.995756624s) [upload=16414]: Upload finished. err: <nil> Duration: 5.920828988s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 6.033057294s FAIL (2025-10-28 09:41:34 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 09:20:59.362633688 (18h38m16.711940974s) [upload=16414]: Starting upload Oct 28 09:20:59.362695796 (18h38m16.712003082s) [upload=16414]: Done starting upload Oct 28 09:20:59.362893874 (18h38m16.71220115s) [upload=16414]: Uploading segments... Oct 28 09:20:59.362942907 (18h38m16.712250183s) [upload=16414]: Got next segment. Inline: false Oct 28 09:20:59.362982592 (18h38m16.712289878s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 28 09:20:59.362989917 (18h38m16.712297203s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 28 09:21:01.021344744 (18h38m18.37065203s) [seg_pos={0 0}, upload=16414]: Begin upload segment... Oct 28 09:21:01.023040152 (18h38m18.372347438s) [seg_pos={0 0}, upload=16414]: Done begin upload segment. Oct 28 09:21:01.023120105 (18h38m18.372427391s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.023839364 (18h38m18.37314665s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024279452 (18h38m18.373586778s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024909652 (18h38m18.374216938s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025349159 (18h38m18.374656446s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025638280 (18h38m18.374945566s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.026032871 (18h38m18.375340157s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.027098981 (18h38m18.376406267s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028007251 (18h38m18.377314537s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028880052 (18h38m18.378187339s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029267360 (18h38m18.378574646s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029838027 (18h38m18.379145313s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030217620 (18h38m18.379524906s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030834975 (18h38m18.380142261s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030959723 (18h38m18.380267009s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.031485544 (18h38m18.38079283s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:02.782872101 (18h38m20.132179387s) [upload=16414]: Next returned nil segment Oct 28 09:21:02.782895556 (18h38m20.132202842s) [upload=16414]: Waiting for error group managing segments... Oct 28 09:21:03.066789825 (18h38m20.416097121s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.071416355 (18h38m20.420723641s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.072983429 (18h38m20.422290715s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.077582706 (18h38m20.426889992s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.078844368 (18h38m20.428151655s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.139198845 (18h38m20.488506131s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.141506871 (18h38m20.490814167s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.170875915 (18h38m20.520183201s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.222706510 (18h38m20.572013796s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.247291775 (18h38m20.596599062s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.259905544 (18h38m20.60921283s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.281169995 (18h38m20.630477281s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312400144 (18h38m20.66170743s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312442985 (18h38m20.661750271s) [seg_pos={0 0}, upload=16414]: Segment reached optimal threshold of 13 pieces. Oct 28 09:21:03.312804424 (18h38m20.66211172s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection Oct 28 09:21:03.312830723 (18h38m20.662138019s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312894104 (18h38m20.6622014s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 28 09:21:03.312902871 (18h38m20.662210157s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312912750 (18h38m20.662220126s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:40022->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 28 09:21:03.312923039 (18h38m20.662230335s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312959218 (18h38m20.662266504s) [seg_pos={0 0}, upload=16414]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 28 09:21:03.313075419 (18h38m20.662382705s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 28 09:21:03.313081431 (18h38m20.662388717s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 28 09:21:03.646425392 (18h38m20.995732688s) [upload=16414]: Done uploading segments... Oct 28 09:21:03.646449338 (18h38m20.995756624s) [upload=16414]: Upload finished. err: <nil> Duration: 6.033098633s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.873957082s FAIL (2025-10-28 09:40:36 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 09:20:59.362633688 (18h38m16.711940974s) [upload=16414]: Starting upload Oct 28 09:20:59.362695796 (18h38m16.712003082s) [upload=16414]: Done starting upload Oct 28 09:20:59.362893874 (18h38m16.71220115s) [upload=16414]: Uploading segments... Oct 28 09:20:59.362942907 (18h38m16.712250183s) [upload=16414]: Got next segment. Inline: false Oct 28 09:20:59.362982592 (18h38m16.712289878s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 28 09:20:59.362989917 (18h38m16.712297203s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 28 09:21:01.021344744 (18h38m18.37065203s) [seg_pos={0 0}, upload=16414]: Begin upload segment... Oct 28 09:21:01.023040152 (18h38m18.372347438s) [seg_pos={0 0}, upload=16414]: Done begin upload segment. Oct 28 09:21:01.023120105 (18h38m18.372427391s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.023839364 (18h38m18.37314665s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024279452 (18h38m18.373586778s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024909652 (18h38m18.374216938s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025349159 (18h38m18.374656446s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025638280 (18h38m18.374945566s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.026032871 (18h38m18.375340157s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.027098981 (18h38m18.376406267s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028007251 (18h38m18.377314537s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028880052 (18h38m18.378187339s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029267360 (18h38m18.378574646s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029838027 (18h38m18.379145313s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030217620 (18h38m18.379524906s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030834975 (18h38m18.380142261s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030959723 (18h38m18.380267009s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.031485544 (18h38m18.38079283s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:02.782872101 (18h38m20.132179387s) [upload=16414]: Next returned nil segment Oct 28 09:21:02.782895556 (18h38m20.132202842s) [upload=16414]: Waiting for error group managing segments... Oct 28 09:21:03.066789825 (18h38m20.416097121s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.071416355 (18h38m20.420723641s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.072983429 (18h38m20.422290715s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.077582706 (18h38m20.426889992s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.078844368 (18h38m20.428151655s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.139198845 (18h38m20.488506131s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.141506871 (18h38m20.490814167s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.170875915 (18h38m20.520183201s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.222706510 (18h38m20.572013796s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.247291775 (18h38m20.596599062s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.259905544 (18h38m20.60921283s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.281169995 (18h38m20.630477281s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312400144 (18h38m20.66170743s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312442985 (18h38m20.661750271s) [seg_pos={0 0}, upload=16414]: Segment reached optimal threshold of 13 pieces. Oct 28 09:21:03.312804424 (18h38m20.66211172s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection Oct 28 09:21:03.312830723 (18h38m20.662138019s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312894104 (18h38m20.6622014s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 28 09:21:03.312902871 (18h38m20.662210157s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312912750 (18h38m20.662220126s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:40022->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 28 09:21:03.312923039 (18h38m20.662230335s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312959218 (18h38m20.662266504s) [seg_pos={0 0}, upload=16414]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 28 09:21:03.313075419 (18h38m20.662382705s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 28 09:21:03.313081431 (18h38m20.662388717s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 28 09:21:03.646425392 (18h38m20.995732688s) [upload=16414]: Done uploading segments... Oct 28 09:21:03.646449338 (18h38m20.995756624s) [upload=16414]: Upload finished. err: <nil> Duration: 5.873983784s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 6.009754091s FAIL (2025-10-28 09:39:51 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 09:20:59.362633688 (18h38m16.711940974s) [upload=16414]: Starting upload Oct 28 09:20:59.362695796 (18h38m16.712003082s) [upload=16414]: Done starting upload Oct 28 09:20:59.362893874 (18h38m16.71220115s) [upload=16414]: Uploading segments... Oct 28 09:20:59.362942907 (18h38m16.712250183s) [upload=16414]: Got next segment. Inline: false Oct 28 09:20:59.362982592 (18h38m16.712289878s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 28 09:20:59.362989917 (18h38m16.712297203s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 28 09:21:01.021344744 (18h38m18.37065203s) [seg_pos={0 0}, upload=16414]: Begin upload segment... Oct 28 09:21:01.023040152 (18h38m18.372347438s) [seg_pos={0 0}, upload=16414]: Done begin upload segment. Oct 28 09:21:01.023120105 (18h38m18.372427391s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.023839364 (18h38m18.37314665s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024279452 (18h38m18.373586778s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024909652 (18h38m18.374216938s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025349159 (18h38m18.374656446s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025638280 (18h38m18.374945566s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.026032871 (18h38m18.375340157s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.027098981 (18h38m18.376406267s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028007251 (18h38m18.377314537s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028880052 (18h38m18.378187339s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029267360 (18h38m18.378574646s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029838027 (18h38m18.379145313s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030217620 (18h38m18.379524906s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030834975 (18h38m18.380142261s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030959723 (18h38m18.380267009s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.031485544 (18h38m18.38079283s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:02.782872101 (18h38m20.132179387s) [upload=16414]: Next returned nil segment Oct 28 09:21:02.782895556 (18h38m20.132202842s) [upload=16414]: Waiting for error group managing segments... Oct 28 09:21:03.066789825 (18h38m20.416097121s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.071416355 (18h38m20.420723641s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.072983429 (18h38m20.422290715s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.077582706 (18h38m20.426889992s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.078844368 (18h38m20.428151655s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.139198845 (18h38m20.488506131s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.141506871 (18h38m20.490814167s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.170875915 (18h38m20.520183201s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.222706510 (18h38m20.572013796s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.247291775 (18h38m20.596599062s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.259905544 (18h38m20.60921283s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.281169995 (18h38m20.630477281s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312400144 (18h38m20.66170743s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312442985 (18h38m20.661750271s) [seg_pos={0 0}, upload=16414]: Segment reached optimal threshold of 13 pieces. Oct 28 09:21:03.312804424 (18h38m20.66211172s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection Oct 28 09:21:03.312830723 (18h38m20.662138019s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312894104 (18h38m20.6622014s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 28 09:21:03.312902871 (18h38m20.662210157s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312912750 (18h38m20.662220126s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:40022->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 28 09:21:03.312923039 (18h38m20.662230335s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312959218 (18h38m20.662266504s) [seg_pos={0 0}, upload=16414]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 28 09:21:03.313075419 (18h38m20.662382705s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 28 09:21:03.313081431 (18h38m20.662388717s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 28 09:21:03.646425392 (18h38m20.995732688s) [upload=16414]: Done uploading segments... Oct 28 09:21:03.646449338 (18h38m20.995756624s) [upload=16414]: Upload finished. err: <nil> Duration: 6.009783027s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.829587904s FAIL (2025-10-28 09:38:50 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 09:20:59.362633688 (18h38m16.711940974s) [upload=16414]: Starting upload Oct 28 09:20:59.362695796 (18h38m16.712003082s) [upload=16414]: Done starting upload Oct 28 09:20:59.362893874 (18h38m16.71220115s) [upload=16414]: Uploading segments... Oct 28 09:20:59.362942907 (18h38m16.712250183s) [upload=16414]: Got next segment. Inline: false Oct 28 09:20:59.362982592 (18h38m16.712289878s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 28 09:20:59.362989917 (18h38m16.712297203s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 28 09:21:01.021344744 (18h38m18.37065203s) [seg_pos={0 0}, upload=16414]: Begin upload segment... Oct 28 09:21:01.023040152 (18h38m18.372347438s) [seg_pos={0 0}, upload=16414]: Done begin upload segment. Oct 28 09:21:01.023120105 (18h38m18.372427391s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.023839364 (18h38m18.37314665s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024279452 (18h38m18.373586778s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024909652 (18h38m18.374216938s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025349159 (18h38m18.374656446s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025638280 (18h38m18.374945566s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.026032871 (18h38m18.375340157s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.027098981 (18h38m18.376406267s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028007251 (18h38m18.377314537s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028880052 (18h38m18.378187339s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029267360 (18h38m18.378574646s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029838027 (18h38m18.379145313s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030217620 (18h38m18.379524906s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030834975 (18h38m18.380142261s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030959723 (18h38m18.380267009s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.031485544 (18h38m18.38079283s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:02.782872101 (18h38m20.132179387s) [upload=16414]: Next returned nil segment Oct 28 09:21:02.782895556 (18h38m20.132202842s) [upload=16414]: Waiting for error group managing segments... Oct 28 09:21:03.066789825 (18h38m20.416097121s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.071416355 (18h38m20.420723641s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.072983429 (18h38m20.422290715s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.077582706 (18h38m20.426889992s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.078844368 (18h38m20.428151655s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.139198845 (18h38m20.488506131s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.141506871 (18h38m20.490814167s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.170875915 (18h38m20.520183201s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.222706510 (18h38m20.572013796s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.247291775 (18h38m20.596599062s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.259905544 (18h38m20.60921283s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.281169995 (18h38m20.630477281s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312400144 (18h38m20.66170743s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312442985 (18h38m20.661750271s) [seg_pos={0 0}, upload=16414]: Segment reached optimal threshold of 13 pieces. Oct 28 09:21:03.312804424 (18h38m20.66211172s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection Oct 28 09:21:03.312830723 (18h38m20.662138019s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312894104 (18h38m20.6622014s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 28 09:21:03.312902871 (18h38m20.662210157s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312912750 (18h38m20.662220126s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:40022->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 28 09:21:03.312923039 (18h38m20.662230335s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312959218 (18h38m20.662266504s) [seg_pos={0 0}, upload=16414]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 28 09:21:03.313075419 (18h38m20.662382705s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 28 09:21:03.313081431 (18h38m20.662388717s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 28 09:21:03.646425392 (18h38m20.995732688s) [upload=16414]: Done uploading segments... Oct 28 09:21:03.646449338 (18h38m20.995756624s) [upload=16414]: Upload finished. err: <nil> Duration: 5.829615316s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 6.034614421s FAIL (2025-10-28 09:37:50 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 09:20:59.362633688 (18h38m16.711940974s) [upload=16414]: Starting upload Oct 28 09:20:59.362695796 (18h38m16.712003082s) [upload=16414]: Done starting upload Oct 28 09:20:59.362893874 (18h38m16.71220115s) [upload=16414]: Uploading segments... Oct 28 09:20:59.362942907 (18h38m16.712250183s) [upload=16414]: Got next segment. Inline: false Oct 28 09:20:59.362982592 (18h38m16.712289878s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 28 09:20:59.362989917 (18h38m16.712297203s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 28 09:21:01.021344744 (18h38m18.37065203s) [seg_pos={0 0}, upload=16414]: Begin upload segment... Oct 28 09:21:01.023040152 (18h38m18.372347438s) [seg_pos={0 0}, upload=16414]: Done begin upload segment. Oct 28 09:21:01.023120105 (18h38m18.372427391s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.023839364 (18h38m18.37314665s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024279452 (18h38m18.373586778s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024909652 (18h38m18.374216938s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025349159 (18h38m18.374656446s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025638280 (18h38m18.374945566s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.026032871 (18h38m18.375340157s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.027098981 (18h38m18.376406267s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028007251 (18h38m18.377314537s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028880052 (18h38m18.378187339s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029267360 (18h38m18.378574646s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029838027 (18h38m18.379145313s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030217620 (18h38m18.379524906s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030834975 (18h38m18.380142261s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030959723 (18h38m18.380267009s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.031485544 (18h38m18.38079283s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:02.782872101 (18h38m20.132179387s) [upload=16414]: Next returned nil segment Oct 28 09:21:02.782895556 (18h38m20.132202842s) [upload=16414]: Waiting for error group managing segments... Oct 28 09:21:03.066789825 (18h38m20.416097121s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.071416355 (18h38m20.420723641s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.072983429 (18h38m20.422290715s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.077582706 (18h38m20.426889992s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.078844368 (18h38m20.428151655s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.139198845 (18h38m20.488506131s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.141506871 (18h38m20.490814167s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.170875915 (18h38m20.520183201s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.222706510 (18h38m20.572013796s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.247291775 (18h38m20.596599062s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.259905544 (18h38m20.60921283s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.281169995 (18h38m20.630477281s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312400144 (18h38m20.66170743s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312442985 (18h38m20.661750271s) [seg_pos={0 0}, upload=16414]: Segment reached optimal threshold of 13 pieces. Oct 28 09:21:03.312804424 (18h38m20.66211172s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection Oct 28 09:21:03.312830723 (18h38m20.662138019s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312894104 (18h38m20.6622014s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 28 09:21:03.312902871 (18h38m20.662210157s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312912750 (18h38m20.662220126s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:40022->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 28 09:21:03.312923039 (18h38m20.662230335s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312959218 (18h38m20.662266504s) [seg_pos={0 0}, upload=16414]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 28 09:21:03.313075419 (18h38m20.662382705s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 28 09:21:03.313081431 (18h38m20.662388717s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 28 09:21:03.646425392 (18h38m20.995732688s) [upload=16414]: Done uploading segments... Oct 28 09:21:03.646449338 (18h38m20.995756624s) [upload=16414]: Upload finished. err: <nil> Duration: 6.034645469s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.868569661s FAIL (2025-10-28 09:36:47 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 09:20:59.362633688 (18h38m16.711940974s) [upload=16414]: Starting upload Oct 28 09:20:59.362695796 (18h38m16.712003082s) [upload=16414]: Done starting upload Oct 28 09:20:59.362893874 (18h38m16.71220115s) [upload=16414]: Uploading segments... Oct 28 09:20:59.362942907 (18h38m16.712250183s) [upload=16414]: Got next segment. Inline: false Oct 28 09:20:59.362982592 (18h38m16.712289878s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 28 09:20:59.362989917 (18h38m16.712297203s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 28 09:21:01.021344744 (18h38m18.37065203s) [seg_pos={0 0}, upload=16414]: Begin upload segment... Oct 28 09:21:01.023040152 (18h38m18.372347438s) [seg_pos={0 0}, upload=16414]: Done begin upload segment. Oct 28 09:21:01.023120105 (18h38m18.372427391s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.023839364 (18h38m18.37314665s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024279452 (18h38m18.373586778s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024909652 (18h38m18.374216938s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025349159 (18h38m18.374656446s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025638280 (18h38m18.374945566s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.026032871 (18h38m18.375340157s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.027098981 (18h38m18.376406267s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028007251 (18h38m18.377314537s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028880052 (18h38m18.378187339s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029267360 (18h38m18.378574646s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029838027 (18h38m18.379145313s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030217620 (18h38m18.379524906s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030834975 (18h38m18.380142261s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030959723 (18h38m18.380267009s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.031485544 (18h38m18.38079283s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:02.782872101 (18h38m20.132179387s) [upload=16414]: Next returned nil segment Oct 28 09:21:02.782895556 (18h38m20.132202842s) [upload=16414]: Waiting for error group managing segments... Oct 28 09:21:03.066789825 (18h38m20.416097121s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.071416355 (18h38m20.420723641s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.072983429 (18h38m20.422290715s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.077582706 (18h38m20.426889992s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.078844368 (18h38m20.428151655s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.139198845 (18h38m20.488506131s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.141506871 (18h38m20.490814167s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.170875915 (18h38m20.520183201s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.222706510 (18h38m20.572013796s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.247291775 (18h38m20.596599062s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.259905544 (18h38m20.60921283s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.281169995 (18h38m20.630477281s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312400144 (18h38m20.66170743s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312442985 (18h38m20.661750271s) [seg_pos={0 0}, upload=16414]: Segment reached optimal threshold of 13 pieces. Oct 28 09:21:03.312804424 (18h38m20.66211172s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection Oct 28 09:21:03.312830723 (18h38m20.662138019s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312894104 (18h38m20.6622014s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 28 09:21:03.312902871 (18h38m20.662210157s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312912750 (18h38m20.662220126s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:40022->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 28 09:21:03.312923039 (18h38m20.662230335s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312959218 (18h38m20.662266504s) [seg_pos={0 0}, upload=16414]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 28 09:21:03.313075419 (18h38m20.662382705s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 28 09:21:03.313081431 (18h38m20.662388717s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 28 09:21:03.646425392 (18h38m20.995732688s) [upload=16414]: Done uploading segments... Oct 28 09:21:03.646449338 (18h38m20.995756624s) [upload=16414]: Upload finished. err: <nil> Duration: 5.868602372s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.901204098s FAIL (2025-10-28 09:36:31 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 09:20:59.362633688 (18h38m16.711940974s) [upload=16414]: Starting upload Oct 28 09:20:59.362695796 (18h38m16.712003082s) [upload=16414]: Done starting upload Oct 28 09:20:59.362893874 (18h38m16.71220115s) [upload=16414]: Uploading segments... Oct 28 09:20:59.362942907 (18h38m16.712250183s) [upload=16414]: Got next segment. Inline: false Oct 28 09:20:59.362982592 (18h38m16.712289878s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 28 09:20:59.362989917 (18h38m16.712297203s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 28 09:21:01.021344744 (18h38m18.37065203s) [seg_pos={0 0}, upload=16414]: Begin upload segment... Oct 28 09:21:01.023040152 (18h38m18.372347438s) [seg_pos={0 0}, upload=16414]: Done begin upload segment. Oct 28 09:21:01.023120105 (18h38m18.372427391s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.023839364 (18h38m18.37314665s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024279452 (18h38m18.373586778s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024909652 (18h38m18.374216938s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025349159 (18h38m18.374656446s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025638280 (18h38m18.374945566s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.026032871 (18h38m18.375340157s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.027098981 (18h38m18.376406267s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028007251 (18h38m18.377314537s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028880052 (18h38m18.378187339s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029267360 (18h38m18.378574646s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029838027 (18h38m18.379145313s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030217620 (18h38m18.379524906s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030834975 (18h38m18.380142261s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030959723 (18h38m18.380267009s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.031485544 (18h38m18.38079283s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:02.782872101 (18h38m20.132179387s) [upload=16414]: Next returned nil segment Oct 28 09:21:02.782895556 (18h38m20.132202842s) [upload=16414]: Waiting for error group managing segments... Oct 28 09:21:03.066789825 (18h38m20.416097121s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.071416355 (18h38m20.420723641s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.072983429 (18h38m20.422290715s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.077582706 (18h38m20.426889992s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.078844368 (18h38m20.428151655s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.139198845 (18h38m20.488506131s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.141506871 (18h38m20.490814167s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.170875915 (18h38m20.520183201s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.222706510 (18h38m20.572013796s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.247291775 (18h38m20.596599062s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.259905544 (18h38m20.60921283s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.281169995 (18h38m20.630477281s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312400144 (18h38m20.66170743s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312442985 (18h38m20.661750271s) [seg_pos={0 0}, upload=16414]: Segment reached optimal threshold of 13 pieces. Oct 28 09:21:03.312804424 (18h38m20.66211172s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection Oct 28 09:21:03.312830723 (18h38m20.662138019s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312894104 (18h38m20.6622014s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 28 09:21:03.312902871 (18h38m20.662210157s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312912750 (18h38m20.662220126s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:40022->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 28 09:21:03.312923039 (18h38m20.662230335s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312959218 (18h38m20.662266504s) [seg_pos={0 0}, upload=16414]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 28 09:21:03.313075419 (18h38m20.662382705s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 28 09:21:03.313081431 (18h38m20.662388717s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 28 09:21:03.646425392 (18h38m20.995732688s) [upload=16414]: Done uploading segments... Oct 28 09:21:03.646449338 (18h38m20.995756624s) [upload=16414]: Upload finished. err: <nil> Duration: 5.901235588s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.916104866s FAIL (2025-10-28 09:36:04 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 09:20:59.362633688 (18h38m16.711940974s) [upload=16414]: Starting upload Oct 28 09:20:59.362695796 (18h38m16.712003082s) [upload=16414]: Done starting upload Oct 28 09:20:59.362893874 (18h38m16.71220115s) [upload=16414]: Uploading segments... Oct 28 09:20:59.362942907 (18h38m16.712250183s) [upload=16414]: Got next segment. Inline: false Oct 28 09:20:59.362982592 (18h38m16.712289878s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 28 09:20:59.362989917 (18h38m16.712297203s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 28 09:21:01.021344744 (18h38m18.37065203s) [seg_pos={0 0}, upload=16414]: Begin upload segment... Oct 28 09:21:01.023040152 (18h38m18.372347438s) [seg_pos={0 0}, upload=16414]: Done begin upload segment. Oct 28 09:21:01.023120105 (18h38m18.372427391s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.023839364 (18h38m18.37314665s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024279452 (18h38m18.373586778s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024909652 (18h38m18.374216938s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025349159 (18h38m18.374656446s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025638280 (18h38m18.374945566s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.026032871 (18h38m18.375340157s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.027098981 (18h38m18.376406267s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028007251 (18h38m18.377314537s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028880052 (18h38m18.378187339s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029267360 (18h38m18.378574646s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029838027 (18h38m18.379145313s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030217620 (18h38m18.379524906s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030834975 (18h38m18.380142261s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030959723 (18h38m18.380267009s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.031485544 (18h38m18.38079283s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:02.782872101 (18h38m20.132179387s) [upload=16414]: Next returned nil segment Oct 28 09:21:02.782895556 (18h38m20.132202842s) [upload=16414]: Waiting for error group managing segments... Oct 28 09:21:03.066789825 (18h38m20.416097121s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.071416355 (18h38m20.420723641s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.072983429 (18h38m20.422290715s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.077582706 (18h38m20.426889992s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.078844368 (18h38m20.428151655s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.139198845 (18h38m20.488506131s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.141506871 (18h38m20.490814167s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.170875915 (18h38m20.520183201s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.222706510 (18h38m20.572013796s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.247291775 (18h38m20.596599062s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.259905544 (18h38m20.60921283s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.281169995 (18h38m20.630477281s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312400144 (18h38m20.66170743s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312442985 (18h38m20.661750271s) [seg_pos={0 0}, upload=16414]: Segment reached optimal threshold of 13 pieces. Oct 28 09:21:03.312804424 (18h38m20.66211172s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection Oct 28 09:21:03.312830723 (18h38m20.662138019s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312894104 (18h38m20.6622014s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 28 09:21:03.312902871 (18h38m20.662210157s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312912750 (18h38m20.662220126s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:40022->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 28 09:21:03.312923039 (18h38m20.662230335s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312959218 (18h38m20.662266504s) [seg_pos={0 0}, upload=16414]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 28 09:21:03.313075419 (18h38m20.662382705s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 28 09:21:03.313081431 (18h38m20.662388717s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 28 09:21:03.646425392 (18h38m20.995732688s) [upload=16414]: Done uploading segments... Oct 28 09:21:03.646449338 (18h38m20.995756624s) [upload=16414]: Upload finished. err: <nil> Duration: 5.916134022s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.878725813s FAIL (2025-10-28 09:35:19 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 09:20:59.362633688 (18h38m16.711940974s) [upload=16414]: Starting upload Oct 28 09:20:59.362695796 (18h38m16.712003082s) [upload=16414]: Done starting upload Oct 28 09:20:59.362893874 (18h38m16.71220115s) [upload=16414]: Uploading segments... Oct 28 09:20:59.362942907 (18h38m16.712250183s) [upload=16414]: Got next segment. Inline: false Oct 28 09:20:59.362982592 (18h38m16.712289878s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 28 09:20:59.362989917 (18h38m16.712297203s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 28 09:21:01.021344744 (18h38m18.37065203s) [seg_pos={0 0}, upload=16414]: Begin upload segment... Oct 28 09:21:01.023040152 (18h38m18.372347438s) [seg_pos={0 0}, upload=16414]: Done begin upload segment. Oct 28 09:21:01.023120105 (18h38m18.372427391s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.023839364 (18h38m18.37314665s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024279452 (18h38m18.373586778s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024909652 (18h38m18.374216938s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025349159 (18h38m18.374656446s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025638280 (18h38m18.374945566s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.026032871 (18h38m18.375340157s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.027098981 (18h38m18.376406267s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028007251 (18h38m18.377314537s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028880052 (18h38m18.378187339s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029267360 (18h38m18.378574646s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029838027 (18h38m18.379145313s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030217620 (18h38m18.379524906s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030834975 (18h38m18.380142261s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030959723 (18h38m18.380267009s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.031485544 (18h38m18.38079283s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:02.782872101 (18h38m20.132179387s) [upload=16414]: Next returned nil segment Oct 28 09:21:02.782895556 (18h38m20.132202842s) [upload=16414]: Waiting for error group managing segments... Oct 28 09:21:03.066789825 (18h38m20.416097121s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.071416355 (18h38m20.420723641s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.072983429 (18h38m20.422290715s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.077582706 (18h38m20.426889992s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.078844368 (18h38m20.428151655s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.139198845 (18h38m20.488506131s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.141506871 (18h38m20.490814167s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.170875915 (18h38m20.520183201s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.222706510 (18h38m20.572013796s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.247291775 (18h38m20.596599062s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.259905544 (18h38m20.60921283s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.281169995 (18h38m20.630477281s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312400144 (18h38m20.66170743s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312442985 (18h38m20.661750271s) [seg_pos={0 0}, upload=16414]: Segment reached optimal threshold of 13 pieces. Oct 28 09:21:03.312804424 (18h38m20.66211172s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection Oct 28 09:21:03.312830723 (18h38m20.662138019s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312894104 (18h38m20.6622014s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 28 09:21:03.312902871 (18h38m20.662210157s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312912750 (18h38m20.662220126s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:40022->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 28 09:21:03.312923039 (18h38m20.662230335s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312959218 (18h38m20.662266504s) [seg_pos={0 0}, upload=16414]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 28 09:21:03.313075419 (18h38m20.662382705s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 28 09:21:03.313081431 (18h38m20.662388717s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 28 09:21:03.646425392 (18h38m20.995732688s) [upload=16414]: Done uploading segments... Oct 28 09:21:03.646449338 (18h38m20.995756624s) [upload=16414]: Upload finished. err: <nil> Duration: 5.878759778s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 6.10323729s FAIL (2025-10-28 09:34:21 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 09:20:59.362633688 (18h38m16.711940974s) [upload=16414]: Starting upload Oct 28 09:20:59.362695796 (18h38m16.712003082s) [upload=16414]: Done starting upload Oct 28 09:20:59.362893874 (18h38m16.71220115s) [upload=16414]: Uploading segments... Oct 28 09:20:59.362942907 (18h38m16.712250183s) [upload=16414]: Got next segment. Inline: false Oct 28 09:20:59.362982592 (18h38m16.712289878s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 28 09:20:59.362989917 (18h38m16.712297203s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 28 09:21:01.021344744 (18h38m18.37065203s) [seg_pos={0 0}, upload=16414]: Begin upload segment... Oct 28 09:21:01.023040152 (18h38m18.372347438s) [seg_pos={0 0}, upload=16414]: Done begin upload segment. Oct 28 09:21:01.023120105 (18h38m18.372427391s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.023839364 (18h38m18.37314665s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024279452 (18h38m18.373586778s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024909652 (18h38m18.374216938s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025349159 (18h38m18.374656446s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025638280 (18h38m18.374945566s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.026032871 (18h38m18.375340157s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.027098981 (18h38m18.376406267s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028007251 (18h38m18.377314537s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028880052 (18h38m18.378187339s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029267360 (18h38m18.378574646s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029838027 (18h38m18.379145313s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030217620 (18h38m18.379524906s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030834975 (18h38m18.380142261s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030959723 (18h38m18.380267009s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.031485544 (18h38m18.38079283s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:02.782872101 (18h38m20.132179387s) [upload=16414]: Next returned nil segment Oct 28 09:21:02.782895556 (18h38m20.132202842s) [upload=16414]: Waiting for error group managing segments... Oct 28 09:21:03.066789825 (18h38m20.416097121s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.071416355 (18h38m20.420723641s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.072983429 (18h38m20.422290715s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.077582706 (18h38m20.426889992s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.078844368 (18h38m20.428151655s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.139198845 (18h38m20.488506131s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.141506871 (18h38m20.490814167s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.170875915 (18h38m20.520183201s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.222706510 (18h38m20.572013796s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.247291775 (18h38m20.596599062s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.259905544 (18h38m20.60921283s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.281169995 (18h38m20.630477281s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312400144 (18h38m20.66170743s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312442985 (18h38m20.661750271s) [seg_pos={0 0}, upload=16414]: Segment reached optimal threshold of 13 pieces. Oct 28 09:21:03.312804424 (18h38m20.66211172s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection Oct 28 09:21:03.312830723 (18h38m20.662138019s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312894104 (18h38m20.6622014s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 28 09:21:03.312902871 (18h38m20.662210157s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312912750 (18h38m20.662220126s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:40022->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 28 09:21:03.312923039 (18h38m20.662230335s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312959218 (18h38m20.662266504s) [seg_pos={0 0}, upload=16414]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 28 09:21:03.313075419 (18h38m20.662382705s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 28 09:21:03.313081431 (18h38m20.662388717s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 28 09:21:03.646425392 (18h38m20.995732688s) [upload=16414]: Done uploading segments... Oct 28 09:21:03.646449338 (18h38m20.995756624s) [upload=16414]: Upload finished. err: <nil> Duration: 6.103269863s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.916907602s FAIL (2025-10-28 09:33:51 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 09:20:59.362633688 (18h38m16.711940974s) [upload=16414]: Starting upload Oct 28 09:20:59.362695796 (18h38m16.712003082s) [upload=16414]: Done starting upload Oct 28 09:20:59.362893874 (18h38m16.71220115s) [upload=16414]: Uploading segments... Oct 28 09:20:59.362942907 (18h38m16.712250183s) [upload=16414]: Got next segment. Inline: false Oct 28 09:20:59.362982592 (18h38m16.712289878s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 28 09:20:59.362989917 (18h38m16.712297203s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 28 09:21:01.021344744 (18h38m18.37065203s) [seg_pos={0 0}, upload=16414]: Begin upload segment... Oct 28 09:21:01.023040152 (18h38m18.372347438s) [seg_pos={0 0}, upload=16414]: Done begin upload segment. Oct 28 09:21:01.023120105 (18h38m18.372427391s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.023839364 (18h38m18.37314665s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024279452 (18h38m18.373586778s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024909652 (18h38m18.374216938s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025349159 (18h38m18.374656446s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025638280 (18h38m18.374945566s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.026032871 (18h38m18.375340157s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.027098981 (18h38m18.376406267s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028007251 (18h38m18.377314537s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028880052 (18h38m18.378187339s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029267360 (18h38m18.378574646s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029838027 (18h38m18.379145313s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030217620 (18h38m18.379524906s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030834975 (18h38m18.380142261s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030959723 (18h38m18.380267009s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.031485544 (18h38m18.38079283s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:02.782872101 (18h38m20.132179387s) [upload=16414]: Next returned nil segment Oct 28 09:21:02.782895556 (18h38m20.132202842s) [upload=16414]: Waiting for error group managing segments... Oct 28 09:21:03.066789825 (18h38m20.416097121s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.071416355 (18h38m20.420723641s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.072983429 (18h38m20.422290715s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.077582706 (18h38m20.426889992s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.078844368 (18h38m20.428151655s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.139198845 (18h38m20.488506131s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.141506871 (18h38m20.490814167s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.170875915 (18h38m20.520183201s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.222706510 (18h38m20.572013796s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.247291775 (18h38m20.596599062s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.259905544 (18h38m20.60921283s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.281169995 (18h38m20.630477281s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312400144 (18h38m20.66170743s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312442985 (18h38m20.661750271s) [seg_pos={0 0}, upload=16414]: Segment reached optimal threshold of 13 pieces. Oct 28 09:21:03.312804424 (18h38m20.66211172s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection Oct 28 09:21:03.312830723 (18h38m20.662138019s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312894104 (18h38m20.6622014s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 28 09:21:03.312902871 (18h38m20.662210157s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312912750 (18h38m20.662220126s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:40022->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 28 09:21:03.312923039 (18h38m20.662230335s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312959218 (18h38m20.662266504s) [seg_pos={0 0}, upload=16414]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 28 09:21:03.313075419 (18h38m20.662382705s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 28 09:21:03.313081431 (18h38m20.662388717s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 28 09:21:03.646425392 (18h38m20.995732688s) [upload=16414]: Done uploading segments... Oct 28 09:21:03.646449338 (18h38m20.995756624s) [upload=16414]: Upload finished. err: <nil> Duration: 5.916940294s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.916337214s FAIL (2025-10-28 09:32:41 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 09:20:59.362633688 (18h38m16.711940974s) [upload=16414]: Starting upload Oct 28 09:20:59.362695796 (18h38m16.712003082s) [upload=16414]: Done starting upload Oct 28 09:20:59.362893874 (18h38m16.71220115s) [upload=16414]: Uploading segments... Oct 28 09:20:59.362942907 (18h38m16.712250183s) [upload=16414]: Got next segment. Inline: false Oct 28 09:20:59.362982592 (18h38m16.712289878s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 28 09:20:59.362989917 (18h38m16.712297203s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 28 09:21:01.021344744 (18h38m18.37065203s) [seg_pos={0 0}, upload=16414]: Begin upload segment... Oct 28 09:21:01.023040152 (18h38m18.372347438s) [seg_pos={0 0}, upload=16414]: Done begin upload segment. Oct 28 09:21:01.023120105 (18h38m18.372427391s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.023839364 (18h38m18.37314665s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024279452 (18h38m18.373586778s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024909652 (18h38m18.374216938s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025349159 (18h38m18.374656446s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025638280 (18h38m18.374945566s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.026032871 (18h38m18.375340157s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.027098981 (18h38m18.376406267s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028007251 (18h38m18.377314537s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028880052 (18h38m18.378187339s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029267360 (18h38m18.378574646s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029838027 (18h38m18.379145313s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030217620 (18h38m18.379524906s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030834975 (18h38m18.380142261s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030959723 (18h38m18.380267009s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.031485544 (18h38m18.38079283s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:02.782872101 (18h38m20.132179387s) [upload=16414]: Next returned nil segment Oct 28 09:21:02.782895556 (18h38m20.132202842s) [upload=16414]: Waiting for error group managing segments... Oct 28 09:21:03.066789825 (18h38m20.416097121s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.071416355 (18h38m20.420723641s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.072983429 (18h38m20.422290715s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.077582706 (18h38m20.426889992s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.078844368 (18h38m20.428151655s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.139198845 (18h38m20.488506131s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.141506871 (18h38m20.490814167s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.170875915 (18h38m20.520183201s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.222706510 (18h38m20.572013796s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.247291775 (18h38m20.596599062s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.259905544 (18h38m20.60921283s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.281169995 (18h38m20.630477281s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312400144 (18h38m20.66170743s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312442985 (18h38m20.661750271s) [seg_pos={0 0}, upload=16414]: Segment reached optimal threshold of 13 pieces. Oct 28 09:21:03.312804424 (18h38m20.66211172s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection Oct 28 09:21:03.312830723 (18h38m20.662138019s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312894104 (18h38m20.6622014s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 28 09:21:03.312902871 (18h38m20.662210157s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312912750 (18h38m20.662220126s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:40022->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 28 09:21:03.312923039 (18h38m20.662230335s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312959218 (18h38m20.662266504s) [seg_pos={0 0}, upload=16414]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 28 09:21:03.313075419 (18h38m20.662382705s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 28 09:21:03.313081431 (18h38m20.662388717s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 28 09:21:03.646425392 (18h38m20.995732688s) [upload=16414]: Done uploading segments... Oct 28 09:21:03.646449338 (18h38m20.995756624s) [upload=16414]: Upload finished. err: <nil> Duration: 5.916377922s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 6.035835585s FAIL (2025-10-28 09:31:30 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 09:20:59.362633688 (18h38m16.711940974s) [upload=16414]: Starting upload Oct 28 09:20:59.362695796 (18h38m16.712003082s) [upload=16414]: Done starting upload Oct 28 09:20:59.362893874 (18h38m16.71220115s) [upload=16414]: Uploading segments... Oct 28 09:20:59.362942907 (18h38m16.712250183s) [upload=16414]: Got next segment. Inline: false Oct 28 09:20:59.362982592 (18h38m16.712289878s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 28 09:20:59.362989917 (18h38m16.712297203s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 28 09:21:01.021344744 (18h38m18.37065203s) [seg_pos={0 0}, upload=16414]: Begin upload segment... Oct 28 09:21:01.023040152 (18h38m18.372347438s) [seg_pos={0 0}, upload=16414]: Done begin upload segment. Oct 28 09:21:01.023120105 (18h38m18.372427391s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.023839364 (18h38m18.37314665s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024279452 (18h38m18.373586778s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024909652 (18h38m18.374216938s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025349159 (18h38m18.374656446s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025638280 (18h38m18.374945566s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.026032871 (18h38m18.375340157s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.027098981 (18h38m18.376406267s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028007251 (18h38m18.377314537s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028880052 (18h38m18.378187339s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029267360 (18h38m18.378574646s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029838027 (18h38m18.379145313s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030217620 (18h38m18.379524906s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030834975 (18h38m18.380142261s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030959723 (18h38m18.380267009s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.031485544 (18h38m18.38079283s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:02.782872101 (18h38m20.132179387s) [upload=16414]: Next returned nil segment Oct 28 09:21:02.782895556 (18h38m20.132202842s) [upload=16414]: Waiting for error group managing segments... Oct 28 09:21:03.066789825 (18h38m20.416097121s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.071416355 (18h38m20.420723641s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.072983429 (18h38m20.422290715s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.077582706 (18h38m20.426889992s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.078844368 (18h38m20.428151655s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.139198845 (18h38m20.488506131s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.141506871 (18h38m20.490814167s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.170875915 (18h38m20.520183201s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.222706510 (18h38m20.572013796s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.247291775 (18h38m20.596599062s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.259905544 (18h38m20.60921283s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.281169995 (18h38m20.630477281s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312400144 (18h38m20.66170743s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312442985 (18h38m20.661750271s) [seg_pos={0 0}, upload=16414]: Segment reached optimal threshold of 13 pieces. Oct 28 09:21:03.312804424 (18h38m20.66211172s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection Oct 28 09:21:03.312830723 (18h38m20.662138019s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312894104 (18h38m20.6622014s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 28 09:21:03.312902871 (18h38m20.662210157s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312912750 (18h38m20.662220126s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:40022->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 28 09:21:03.312923039 (18h38m20.662230335s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312959218 (18h38m20.662266504s) [seg_pos={0 0}, upload=16414]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 28 09:21:03.313075419 (18h38m20.662382705s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 28 09:21:03.313081431 (18h38m20.662388717s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 28 09:21:03.646425392 (18h38m20.995732688s) [upload=16414]: Done uploading segments... Oct 28 09:21:03.646449338 (18h38m20.995756624s) [upload=16414]: Upload finished. err: <nil> Duration: 6.035866543s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.883128988s FAIL (2025-10-28 09:30:33 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 09:20:59.362633688 (18h38m16.711940974s) [upload=16414]: Starting upload Oct 28 09:20:59.362695796 (18h38m16.712003082s) [upload=16414]: Done starting upload Oct 28 09:20:59.362893874 (18h38m16.71220115s) [upload=16414]: Uploading segments... Oct 28 09:20:59.362942907 (18h38m16.712250183s) [upload=16414]: Got next segment. Inline: false Oct 28 09:20:59.362982592 (18h38m16.712289878s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 28 09:20:59.362989917 (18h38m16.712297203s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 28 09:21:01.021344744 (18h38m18.37065203s) [seg_pos={0 0}, upload=16414]: Begin upload segment... Oct 28 09:21:01.023040152 (18h38m18.372347438s) [seg_pos={0 0}, upload=16414]: Done begin upload segment. Oct 28 09:21:01.023120105 (18h38m18.372427391s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.023839364 (18h38m18.37314665s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024279452 (18h38m18.373586778s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024909652 (18h38m18.374216938s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025349159 (18h38m18.374656446s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025638280 (18h38m18.374945566s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.026032871 (18h38m18.375340157s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.027098981 (18h38m18.376406267s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028007251 (18h38m18.377314537s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028880052 (18h38m18.378187339s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029267360 (18h38m18.378574646s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029838027 (18h38m18.379145313s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030217620 (18h38m18.379524906s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030834975 (18h38m18.380142261s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030959723 (18h38m18.380267009s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.031485544 (18h38m18.38079283s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:02.782872101 (18h38m20.132179387s) [upload=16414]: Next returned nil segment Oct 28 09:21:02.782895556 (18h38m20.132202842s) [upload=16414]: Waiting for error group managing segments... Oct 28 09:21:03.066789825 (18h38m20.416097121s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.071416355 (18h38m20.420723641s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.072983429 (18h38m20.422290715s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.077582706 (18h38m20.426889992s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.078844368 (18h38m20.428151655s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.139198845 (18h38m20.488506131s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.141506871 (18h38m20.490814167s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.170875915 (18h38m20.520183201s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.222706510 (18h38m20.572013796s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.247291775 (18h38m20.596599062s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.259905544 (18h38m20.60921283s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.281169995 (18h38m20.630477281s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312400144 (18h38m20.66170743s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312442985 (18h38m20.661750271s) [seg_pos={0 0}, upload=16414]: Segment reached optimal threshold of 13 pieces. Oct 28 09:21:03.312804424 (18h38m20.66211172s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection Oct 28 09:21:03.312830723 (18h38m20.662138019s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312894104 (18h38m20.6622014s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 28 09:21:03.312902871 (18h38m20.662210157s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312912750 (18h38m20.662220126s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:40022->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 28 09:21:03.312923039 (18h38m20.662230335s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312959218 (18h38m20.662266504s) [seg_pos={0 0}, upload=16414]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 28 09:21:03.313075419 (18h38m20.662382705s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 28 09:21:03.313081431 (18h38m20.662388717s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 28 09:21:03.646425392 (18h38m20.995732688s) [upload=16414]: Done uploading segments... Oct 28 09:21:03.646449338 (18h38m20.995756624s) [upload=16414]: Upload finished. err: <nil> Duration: 5.883164175s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.021515724s FAIL (2025-10-28 09:30:01 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 09:20:59.362633688 (18h38m16.711940974s) [upload=16414]: Starting upload Oct 28 09:20:59.362695796 (18h38m16.712003082s) [upload=16414]: Done starting upload Oct 28 09:20:59.362893874 (18h38m16.71220115s) [upload=16414]: Uploading segments... Oct 28 09:20:59.362942907 (18h38m16.712250183s) [upload=16414]: Got next segment. Inline: false Oct 28 09:20:59.362982592 (18h38m16.712289878s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 28 09:20:59.362989917 (18h38m16.712297203s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 28 09:21:01.021344744 (18h38m18.37065203s) [seg_pos={0 0}, upload=16414]: Begin upload segment... Oct 28 09:21:01.023040152 (18h38m18.372347438s) [seg_pos={0 0}, upload=16414]: Done begin upload segment. Oct 28 09:21:01.023120105 (18h38m18.372427391s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.023839364 (18h38m18.37314665s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024279452 (18h38m18.373586778s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.024909652 (18h38m18.374216938s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025349159 (18h38m18.374656446s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.025638280 (18h38m18.374945566s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.026032871 (18h38m18.375340157s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.027098981 (18h38m18.376406267s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028007251 (18h38m18.377314537s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.028880052 (18h38m18.378187339s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029267360 (18h38m18.378574646s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.029838027 (18h38m18.379145313s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030217620 (18h38m18.379524906s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030834975 (18h38m18.380142261s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.030959723 (18h38m18.380267009s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:01.031485544 (18h38m18.38079283s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Uploading piece... Oct 28 09:21:02.782872101 (18h38m20.132179387s) [upload=16414]: Next returned nil segment Oct 28 09:21:02.782895556 (18h38m20.132202842s) [upload=16414]: Waiting for error group managing segments... Oct 28 09:21:03.066789825 (18h38m20.416097121s) [noise=true , address=145.79.227.4:4185 , piece_id=VKHFX2IQ34AWY53POYYF6WSFKFHCPEXDT3WNXHYO5Z6Y5TUOCU4A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.071416355 (18h38m20.420723641s) [noise=true , address=145.79.227.22:4510 , piece_id=ZX7VX6SIL5ESFFNX4WRCN6DC4XFJQ6GLXMABCVSWDJTM5UEC7XTA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.072983429 (18h38m20.422290715s) [noise=true , address=145.79.227.6:4259 , piece_id=PXFB3YPE5B73ITEKMVKY65NN5OE2JOSMNVLT24JEUBBNWMXNT45Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.077582706 (18h38m20.426889992s) [noise=true , address=145.79.227.8:4329 , piece_id=E2OLUGERCDDYS2ACT5556GHQGD2V3KUYTZDNGXGA4OEPKMEBKTUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.078844368 (18h38m20.428151655s) [noise=true , address=145.79.227.26:4654 , piece_id=OELCXRUD472YQ6CK5IXNDHRYVCJBVXRNCSIM2HB67LC2UKQP2SIA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.139198845 (18h38m20.488506131s) [noise=true , address=145.79.227.24:4583 , piece_id=2IHHOIER7BPTU3QIFNCYXUKCNSDLHXAMRHFTLC3AYOET3QE2BRRQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.141506871 (18h38m20.490814167s) [noise=true , address=145.79.227.16:4910 , piece_id=EH43PWXNIMF3TZJNXYAWWDYJ35QBN6VWP5LK7762HG4OHLSAQPJQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.170875915 (18h38m20.520183201s) [noise=true , address=145.79.227.11:4732 , piece_id=TAWQEIFVOV4JKJST2R2VEAKABUX3KTCZLL4QIAB5WUUHAWHFYVOQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.222706510 (18h38m20.572013796s) [noise=true , address=45.91.116.4:4938 , piece_id=JYMURV333VXIWAFVTX6XWHFI7HLAB26C7OKRJOVLRSIFGLY4OVUQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.247291775 (18h38m20.596599062s) [noise=true , address=45.91.116.11:4117 , piece_id=WMPWS2QUOJYKDQ42DZ6HQ365AEPANPQXK6FUDMES2QGPOY5IFW4Q, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.259905544 (18h38m20.60921283s) [noise=true , address=77.90.31.19:4842 , piece_id=IWPKOBYKX3R3AJUUJ4ULBQVQUG2LXJHLA6R4DOZV3ZZEIWS3BLDA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.281169995 (18h38m20.630477281s) [noise=true , address=45.91.116.13:4153 , piece_id=7KKSLVWKXVILILRWNR4GWLB7HGZEJB7ZCB62DHG5J3LSVVMQHS3A, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312400144 (18h38m20.66170743s) [noise=true , address=77.90.31.11:4555 , piece_id=3HSSPNJU6GK7BC7D7ALKHXXHHGPSSFD3PVMIIQ64QJKKFZRS6DHQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: <nil> Oct 28 09:21:03.312442985 (18h38m20.661750271s) [seg_pos={0 0}, upload=16414]: Segment reached optimal threshold of 13 pieces. Oct 28 09:21:03.312804424 (18h38m20.66211172s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35130->45.91.116.10:4083: use of closed network connection Oct 28 09:21:03.312830723 (18h38m20.662138019s) [noise=true , address=45.91.116.10:4083 , piece_id=OFB32DYCC2W3EOBPQXM3C4CUDGZ24YR7QOSAS2ZUGF6RLYQZNHFQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312894104 (18h38m20.6622014s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 28 09:21:03.312902871 (18h38m20.662210157s) [noise=true , address=77.90.31.14:4662 , piece_id=72H45A5SQUWQRVFJVJYV4XZVT6JSGVFLUWQDJSKII3UC2VRV5EPA, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312912750 (18h38m20.662220126s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:40022->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 28 09:21:03.312923039 (18h38m20.662230335s) [noise=true , address=77.90.31.10:4518 , piece_id=USOXVBQROBFOQKN77ZYVUAITFWOUXPGP26WBNUCZJEV5FEUZMOPQ, seg_pos={0 0}, upload=16414]: Long tail cancellation due to optimal threshold Oct 28 09:21:03.312959218 (18h38m20.662266504s) [seg_pos={0 0}, upload=16414]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 28 09:21:03.313075419 (18h38m20.662382705s) [upload=16414]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 28 09:21:03.313081431 (18h38m20.662388717s) [upload=16414]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 28 09:21:03.646425392 (18h38m20.995732688s) [upload=16414]: Done uploading segments... Oct 28 09:21:03.646449338 (18h38m20.995756624s) [upload=16414]: Upload finished. err: <nil> Duration: 6.021549118s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)