Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X.X...........................] duration: 6.019935887s FAIL (2025-07-18 01:30:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== T467KJ2SHEV5EJW2LJCYJSSHAG2CJIU2A, seg_pos={0 0}, upload=473444]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: write tcp 188.166.225.170:43510->77.90.31.16:4733: use of closed network connection; context canceled Jul 18 01:09:36.439338648 (591h59m11.209979983s) [seg_pos={0 0}, upload=473444]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 01:09:36.439407128 (591h59m11.210048464s) [upload=473444]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 01:09:36.439411076 (591h59m11.210052411s) [upload=473444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 01:09:36.756883131 (591h59m11.527524466s) [upload=473444]: Done uploading segments... Jul 18 01:09:36.756897708 (591h59m11.527539044s) [upload=473444]: Upload finished. err: <nil> Jul 18 01:10:39.025598816 (592h0m13.796240151s) [upload=473459]: Starting upload Jul 18 01:10:39.025658440 (592h0m13.796299765s) [upload=473459]: Done starting upload Jul 18 01:10:39.025759353 (592h0m13.796400688s) [upload=473459]: Uploading segments... Jul 18 01:10:39.025907185 (592h0m13.79654852s) [upload=473459]: Got next segment. Inline: false Jul 18 01:10:39.025950799 (592h0m13.796592124s) [upload=473459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 01:10:39.025957632 (592h0m13.796598967s) [upload=473459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 01:10:39.286495526 (592h0m14.057136861s) [seg_pos={0 0}, upload=473459]: Begin upload segment... Jul 18 01:10:39.286547334 (592h0m14.057188699s) [seg_pos={0 0}, upload=473459]: Done begin upload segment. Jul 18 01:10:39.286596759 (592h0m14.057238094s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.286620685 (592h0m14.05726202s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287116924 (592h0m14.057758259s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287176458 (592h0m14.057817793s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287488926 (592h0m14.058130261s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287772308 (592h0m14.058413643s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287863583 (592h0m14.058504918s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288079506 (592h0m14.058720841s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288237839 (592h0m14.058879184s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288418594 (592h0m14.059059929s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288629087 (592h0m14.059270422s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289488541 (592h0m14.060129877s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289738971 (592h0m14.060380306s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289989741 (592h0m14.060631086s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.290216545 (592h0m14.06085788s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.297416701 (592h0m14.068058036s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:41.030217031 (592h0m15.800858356s) [upload=473459]: Next returned nil segment Jul 18 01:10:41.030234745 (592h0m15.80087608s) [upload=473459]: Waiting for error group managing segments... Jul 18 01:10:41.308832612 (592h0m16.079473947s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.358887261 (592h0m16.129528596s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.365986644 (592h0m16.136627979s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.375224368 (592h0m16.145865704s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.419739118 (592h0m16.190380453s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.430360108 (592h0m16.201001444s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.450861434 (592h0m16.221502769s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468109615 (592h0m16.23875095s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468727326 (592h0m16.239368661s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.516013518 (592h0m16.286654863s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.522587155 (592h0m16.29322849s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.553218502 (592h0m16.323859837s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:11:08.331622589 (592h0m43.102263924s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38902->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331668216 (592h0m43.102309551s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:08.331710026 (592h0m43.102351361s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:60806->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331772906 (592h0m43.102414241s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331804857 (592h0m43.102446182s) [seg_pos={0 0}, upload=473459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 01:11:08.598012820 (592h0m43.368654156s) [upload=473459]: Done uploading segments... Jul 18 01:11:08.598041896 (592h0m43.368683231s) [upload=473459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.019969803s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X.X..........................] duration: 6.271062726s FAIL (2025-07-18 01:29:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== T467KJ2SHEV5EJW2LJCYJSSHAG2CJIU2A, seg_pos={0 0}, upload=473444]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: write tcp 188.166.225.170:43510->77.90.31.16:4733: use of closed network connection; context canceled Jul 18 01:09:36.439338648 (591h59m11.209979983s) [seg_pos={0 0}, upload=473444]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 01:09:36.439407128 (591h59m11.210048464s) [upload=473444]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 01:09:36.439411076 (591h59m11.210052411s) [upload=473444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 01:09:36.756883131 (591h59m11.527524466s) [upload=473444]: Done uploading segments... Jul 18 01:09:36.756897708 (591h59m11.527539044s) [upload=473444]: Upload finished. err: <nil> Jul 18 01:10:39.025598816 (592h0m13.796240151s) [upload=473459]: Starting upload Jul 18 01:10:39.025658440 (592h0m13.796299765s) [upload=473459]: Done starting upload Jul 18 01:10:39.025759353 (592h0m13.796400688s) [upload=473459]: Uploading segments... Jul 18 01:10:39.025907185 (592h0m13.79654852s) [upload=473459]: Got next segment. Inline: false Jul 18 01:10:39.025950799 (592h0m13.796592124s) [upload=473459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 01:10:39.025957632 (592h0m13.796598967s) [upload=473459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 01:10:39.286495526 (592h0m14.057136861s) [seg_pos={0 0}, upload=473459]: Begin upload segment... Jul 18 01:10:39.286547334 (592h0m14.057188699s) [seg_pos={0 0}, upload=473459]: Done begin upload segment. Jul 18 01:10:39.286596759 (592h0m14.057238094s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.286620685 (592h0m14.05726202s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287116924 (592h0m14.057758259s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287176458 (592h0m14.057817793s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287488926 (592h0m14.058130261s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287772308 (592h0m14.058413643s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287863583 (592h0m14.058504918s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288079506 (592h0m14.058720841s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288237839 (592h0m14.058879184s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288418594 (592h0m14.059059929s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288629087 (592h0m14.059270422s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289488541 (592h0m14.060129877s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289738971 (592h0m14.060380306s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289989741 (592h0m14.060631086s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.290216545 (592h0m14.06085788s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.297416701 (592h0m14.068058036s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:41.030217031 (592h0m15.800858356s) [upload=473459]: Next returned nil segment Jul 18 01:10:41.030234745 (592h0m15.80087608s) [upload=473459]: Waiting for error group managing segments... Jul 18 01:10:41.308832612 (592h0m16.079473947s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.358887261 (592h0m16.129528596s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.365986644 (592h0m16.136627979s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.375224368 (592h0m16.145865704s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.419739118 (592h0m16.190380453s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.430360108 (592h0m16.201001444s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.450861434 (592h0m16.221502769s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468109615 (592h0m16.23875095s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468727326 (592h0m16.239368661s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.516013518 (592h0m16.286654863s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.522587155 (592h0m16.29322849s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.553218502 (592h0m16.323859837s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:11:08.331622589 (592h0m43.102263924s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38902->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331668216 (592h0m43.102309551s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:08.331710026 (592h0m43.102351361s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:60806->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331772906 (592h0m43.102414241s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331804857 (592h0m43.102446182s) [seg_pos={0 0}, upload=473459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 01:11:08.598012820 (592h0m43.368654156s) [upload=473459]: Done uploading segments... Jul 18 01:11:08.598041896 (592h0m43.368683231s) [upload=473459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.271116579s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................................XX.......................] duration: 5.878184485s FAIL (2025-07-18 01:29:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ress=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683391198 (592h0m30.454032533s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683761526 (592h0m30.454402861s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683801033 (592h0m30.454442368s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684063464 (592h0m30.454704799s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684360503 (592h0m30.455001838s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684654846 (592h0m30.455296181s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685431742 (592h0m30.456073077s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685861425 (592h0m30.45650276s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686139176 (592h0m30.456780511s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686424722 (592h0m30.457066057s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686687836 (592h0m30.457329171s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686944527 (592h0m30.457585862s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.687208402 (592h0m30.457849737s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688069810 (592h0m30.458711145s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688391375 (592h0m30.45903271s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688755091 (592h0m30.459396426s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.689020598 (592h0m30.459661934s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:56.791786484 (592h0m31.562427839s) [upload=473462]: Next returned nil segment Jul 18 01:10:56.791810180 (592h0m31.562451535s) [upload=473462]: Waiting for error group managing segments... Jul 18 01:10:57.072508846 (592h0m31.843150181s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.084313581 (592h0m31.854954917s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.130829109 (592h0m31.901470444s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.380019692 (592h0m32.150661027s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.586116023 (592h0m32.356757358s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.590030633 (592h0m32.360671968s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.644593087 (592h0m32.415234412s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.711170438 (592h0m32.481811773s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.815350524 (592h0m32.585991869s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:11:54.506942141 (592h1m29.277583476s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: rpc: dial tcp 77.90.31.13:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.506990634 (592h1m29.277631959s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507021904 (592h1m29.277663239s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: rpc: dial tcp 77.90.31.19:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507055037 (592h1m29.277696372s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: rpc: dial tcp 77.90.31.18:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507085656 (592h1m29.277726992s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 188.166.225.170:42120->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507153856 (592h1m29.277795192s) [noise=true , address=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj, address:77.90.31.15:4704): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507228530 (592h1m29.277869865s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507325304 (592h1m29.277966639s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12eUC555Lzo37B5FafAFMfXjSkfm1zv1pRZmH71jf4xpdCWH6kG, address:77.90.31.21:4904): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507433642 (592h1m29.278074977s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507448881 (592h1m29.278090216s) [seg_pos={0 0}, upload=473462]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 18 01:11:54.756684719 (592h1m29.527326054s) [upload=473462]: Done uploading segments... Jul 18 01:11:54.756710668 (592h1m29.527352004s) [upload=473462]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 5.878211347s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X.X.........................] duration: 5.937724497s FAIL (2025-07-18 01:28:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== T467KJ2SHEV5EJW2LJCYJSSHAG2CJIU2A, seg_pos={0 0}, upload=473444]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: write tcp 188.166.225.170:43510->77.90.31.16:4733: use of closed network connection; context canceled Jul 18 01:09:36.439338648 (591h59m11.209979983s) [seg_pos={0 0}, upload=473444]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 01:09:36.439407128 (591h59m11.210048464s) [upload=473444]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 01:09:36.439411076 (591h59m11.210052411s) [upload=473444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 01:09:36.756883131 (591h59m11.527524466s) [upload=473444]: Done uploading segments... Jul 18 01:09:36.756897708 (591h59m11.527539044s) [upload=473444]: Upload finished. err: <nil> Jul 18 01:10:39.025598816 (592h0m13.796240151s) [upload=473459]: Starting upload Jul 18 01:10:39.025658440 (592h0m13.796299765s) [upload=473459]: Done starting upload Jul 18 01:10:39.025759353 (592h0m13.796400688s) [upload=473459]: Uploading segments... Jul 18 01:10:39.025907185 (592h0m13.79654852s) [upload=473459]: Got next segment. Inline: false Jul 18 01:10:39.025950799 (592h0m13.796592124s) [upload=473459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 01:10:39.025957632 (592h0m13.796598967s) [upload=473459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 01:10:39.286495526 (592h0m14.057136861s) [seg_pos={0 0}, upload=473459]: Begin upload segment... Jul 18 01:10:39.286547334 (592h0m14.057188699s) [seg_pos={0 0}, upload=473459]: Done begin upload segment. Jul 18 01:10:39.286596759 (592h0m14.057238094s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.286620685 (592h0m14.05726202s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287116924 (592h0m14.057758259s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287176458 (592h0m14.057817793s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287488926 (592h0m14.058130261s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287772308 (592h0m14.058413643s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287863583 (592h0m14.058504918s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288079506 (592h0m14.058720841s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288237839 (592h0m14.058879184s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288418594 (592h0m14.059059929s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288629087 (592h0m14.059270422s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289488541 (592h0m14.060129877s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289738971 (592h0m14.060380306s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289989741 (592h0m14.060631086s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.290216545 (592h0m14.06085788s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.297416701 (592h0m14.068058036s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:41.030217031 (592h0m15.800858356s) [upload=473459]: Next returned nil segment Jul 18 01:10:41.030234745 (592h0m15.80087608s) [upload=473459]: Waiting for error group managing segments... Jul 18 01:10:41.308832612 (592h0m16.079473947s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.358887261 (592h0m16.129528596s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.365986644 (592h0m16.136627979s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.375224368 (592h0m16.145865704s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.419739118 (592h0m16.190380453s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.430360108 (592h0m16.201001444s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.450861434 (592h0m16.221502769s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468109615 (592h0m16.23875095s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468727326 (592h0m16.239368661s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.516013518 (592h0m16.286654863s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.522587155 (592h0m16.29322849s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.553218502 (592h0m16.323859837s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:11:08.331622589 (592h0m43.102263924s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38902->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331668216 (592h0m43.102309551s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:08.331710026 (592h0m43.102351361s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:60806->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331772906 (592h0m43.102414241s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331804857 (592h0m43.102446182s) [seg_pos={0 0}, upload=473459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 01:11:08.598012820 (592h0m43.368654156s) [upload=473459]: Done uploading segments... Jul 18 01:11:08.598041896 (592h0m43.368683231s) [upload=473459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.937747681s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....................................XX......................] duration: 5.885178062s FAIL (2025-07-18 01:28:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ress=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683391198 (592h0m30.454032533s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683761526 (592h0m30.454402861s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683801033 (592h0m30.454442368s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684063464 (592h0m30.454704799s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684360503 (592h0m30.455001838s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684654846 (592h0m30.455296181s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685431742 (592h0m30.456073077s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685861425 (592h0m30.45650276s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686139176 (592h0m30.456780511s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686424722 (592h0m30.457066057s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686687836 (592h0m30.457329171s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686944527 (592h0m30.457585862s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.687208402 (592h0m30.457849737s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688069810 (592h0m30.458711145s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688391375 (592h0m30.45903271s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688755091 (592h0m30.459396426s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.689020598 (592h0m30.459661934s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:56.791786484 (592h0m31.562427839s) [upload=473462]: Next returned nil segment Jul 18 01:10:56.791810180 (592h0m31.562451535s) [upload=473462]: Waiting for error group managing segments... Jul 18 01:10:57.072508846 (592h0m31.843150181s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.084313581 (592h0m31.854954917s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.130829109 (592h0m31.901470444s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.380019692 (592h0m32.150661027s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.586116023 (592h0m32.356757358s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.590030633 (592h0m32.360671968s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.644593087 (592h0m32.415234412s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.711170438 (592h0m32.481811773s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.815350524 (592h0m32.585991869s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:11:54.506942141 (592h1m29.277583476s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: rpc: dial tcp 77.90.31.13:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.506990634 (592h1m29.277631959s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507021904 (592h1m29.277663239s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: rpc: dial tcp 77.90.31.19:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507055037 (592h1m29.277696372s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: rpc: dial tcp 77.90.31.18:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507085656 (592h1m29.277726992s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 188.166.225.170:42120->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507153856 (592h1m29.277795192s) [noise=true , address=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj, address:77.90.31.15:4704): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507228530 (592h1m29.277869865s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507325304 (592h1m29.277966639s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12eUC555Lzo37B5FafAFMfXjSkfm1zv1pRZmH71jf4xpdCWH6kG, address:77.90.31.21:4904): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507433642 (592h1m29.278074977s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507448881 (592h1m29.278090216s) [seg_pos={0 0}, upload=473462]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 18 01:11:54.756684719 (592h1m29.527326054s) [upload=473462]: Done uploading segments... Jul 18 01:11:54.756710668 (592h1m29.527352004s) [upload=473462]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 5.885205175s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X.X........................] duration: 5.919466856s FAIL (2025-07-18 01:28:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== T467KJ2SHEV5EJW2LJCYJSSHAG2CJIU2A, seg_pos={0 0}, upload=473444]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: write tcp 188.166.225.170:43510->77.90.31.16:4733: use of closed network connection; context canceled Jul 18 01:09:36.439338648 (591h59m11.209979983s) [seg_pos={0 0}, upload=473444]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 01:09:36.439407128 (591h59m11.210048464s) [upload=473444]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 01:09:36.439411076 (591h59m11.210052411s) [upload=473444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 01:09:36.756883131 (591h59m11.527524466s) [upload=473444]: Done uploading segments... Jul 18 01:09:36.756897708 (591h59m11.527539044s) [upload=473444]: Upload finished. err: <nil> Jul 18 01:10:39.025598816 (592h0m13.796240151s) [upload=473459]: Starting upload Jul 18 01:10:39.025658440 (592h0m13.796299765s) [upload=473459]: Done starting upload Jul 18 01:10:39.025759353 (592h0m13.796400688s) [upload=473459]: Uploading segments... Jul 18 01:10:39.025907185 (592h0m13.79654852s) [upload=473459]: Got next segment. Inline: false Jul 18 01:10:39.025950799 (592h0m13.796592124s) [upload=473459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 01:10:39.025957632 (592h0m13.796598967s) [upload=473459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 01:10:39.286495526 (592h0m14.057136861s) [seg_pos={0 0}, upload=473459]: Begin upload segment... Jul 18 01:10:39.286547334 (592h0m14.057188699s) [seg_pos={0 0}, upload=473459]: Done begin upload segment. Jul 18 01:10:39.286596759 (592h0m14.057238094s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.286620685 (592h0m14.05726202s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287116924 (592h0m14.057758259s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287176458 (592h0m14.057817793s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287488926 (592h0m14.058130261s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287772308 (592h0m14.058413643s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287863583 (592h0m14.058504918s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288079506 (592h0m14.058720841s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288237839 (592h0m14.058879184s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288418594 (592h0m14.059059929s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288629087 (592h0m14.059270422s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289488541 (592h0m14.060129877s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289738971 (592h0m14.060380306s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289989741 (592h0m14.060631086s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.290216545 (592h0m14.06085788s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.297416701 (592h0m14.068058036s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:41.030217031 (592h0m15.800858356s) [upload=473459]: Next returned nil segment Jul 18 01:10:41.030234745 (592h0m15.80087608s) [upload=473459]: Waiting for error group managing segments... Jul 18 01:10:41.308832612 (592h0m16.079473947s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.358887261 (592h0m16.129528596s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.365986644 (592h0m16.136627979s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.375224368 (592h0m16.145865704s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.419739118 (592h0m16.190380453s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.430360108 (592h0m16.201001444s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.450861434 (592h0m16.221502769s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468109615 (592h0m16.23875095s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468727326 (592h0m16.239368661s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.516013518 (592h0m16.286654863s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.522587155 (592h0m16.29322849s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.553218502 (592h0m16.323859837s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:11:08.331622589 (592h0m43.102263924s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38902->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331668216 (592h0m43.102309551s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:08.331710026 (592h0m43.102351361s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:60806->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331772906 (592h0m43.102414241s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331804857 (592h0m43.102446182s) [seg_pos={0 0}, upload=473459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 01:11:08.598012820 (592h0m43.368654156s) [upload=473459]: Done uploading segments... Jul 18 01:11:08.598041896 (592h0m43.368683231s) [upload=473459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.91953191s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....................................XX.....................] duration: 5.775201414s FAIL (2025-07-18 01:27:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ress=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683391198 (592h0m30.454032533s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683761526 (592h0m30.454402861s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683801033 (592h0m30.454442368s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684063464 (592h0m30.454704799s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684360503 (592h0m30.455001838s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684654846 (592h0m30.455296181s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685431742 (592h0m30.456073077s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685861425 (592h0m30.45650276s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686139176 (592h0m30.456780511s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686424722 (592h0m30.457066057s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686687836 (592h0m30.457329171s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686944527 (592h0m30.457585862s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.687208402 (592h0m30.457849737s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688069810 (592h0m30.458711145s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688391375 (592h0m30.45903271s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688755091 (592h0m30.459396426s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.689020598 (592h0m30.459661934s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:56.791786484 (592h0m31.562427839s) [upload=473462]: Next returned nil segment Jul 18 01:10:56.791810180 (592h0m31.562451535s) [upload=473462]: Waiting for error group managing segments... Jul 18 01:10:57.072508846 (592h0m31.843150181s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.084313581 (592h0m31.854954917s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.130829109 (592h0m31.901470444s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.380019692 (592h0m32.150661027s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.586116023 (592h0m32.356757358s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.590030633 (592h0m32.360671968s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.644593087 (592h0m32.415234412s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.711170438 (592h0m32.481811773s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.815350524 (592h0m32.585991869s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:11:54.506942141 (592h1m29.277583476s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: rpc: dial tcp 77.90.31.13:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.506990634 (592h1m29.277631959s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507021904 (592h1m29.277663239s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: rpc: dial tcp 77.90.31.19:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507055037 (592h1m29.277696372s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: rpc: dial tcp 77.90.31.18:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507085656 (592h1m29.277726992s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 188.166.225.170:42120->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507153856 (592h1m29.277795192s) [noise=true , address=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj, address:77.90.31.15:4704): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507228530 (592h1m29.277869865s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507325304 (592h1m29.277966639s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12eUC555Lzo37B5FafAFMfXjSkfm1zv1pRZmH71jf4xpdCWH6kG, address:77.90.31.21:4904): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507433642 (592h1m29.278074977s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507448881 (592h1m29.278090216s) [seg_pos={0 0}, upload=473462]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 18 01:11:54.756684719 (592h1m29.527326054s) [upload=473462]: Done uploading segments... Jul 18 01:11:54.756710668 (592h1m29.527352004s) [upload=473462]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 5.775230197s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X.X.......................] duration: 6.028515264s FAIL (2025-07-18 01:27:05 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== T467KJ2SHEV5EJW2LJCYJSSHAG2CJIU2A, seg_pos={0 0}, upload=473444]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: write tcp 188.166.225.170:43510->77.90.31.16:4733: use of closed network connection; context canceled Jul 18 01:09:36.439338648 (591h59m11.209979983s) [seg_pos={0 0}, upload=473444]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 01:09:36.439407128 (591h59m11.210048464s) [upload=473444]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 01:09:36.439411076 (591h59m11.210052411s) [upload=473444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 01:09:36.756883131 (591h59m11.527524466s) [upload=473444]: Done uploading segments... Jul 18 01:09:36.756897708 (591h59m11.527539044s) [upload=473444]: Upload finished. err: <nil> Jul 18 01:10:39.025598816 (592h0m13.796240151s) [upload=473459]: Starting upload Jul 18 01:10:39.025658440 (592h0m13.796299765s) [upload=473459]: Done starting upload Jul 18 01:10:39.025759353 (592h0m13.796400688s) [upload=473459]: Uploading segments... Jul 18 01:10:39.025907185 (592h0m13.79654852s) [upload=473459]: Got next segment. Inline: false Jul 18 01:10:39.025950799 (592h0m13.796592124s) [upload=473459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 01:10:39.025957632 (592h0m13.796598967s) [upload=473459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 01:10:39.286495526 (592h0m14.057136861s) [seg_pos={0 0}, upload=473459]: Begin upload segment... Jul 18 01:10:39.286547334 (592h0m14.057188699s) [seg_pos={0 0}, upload=473459]: Done begin upload segment. Jul 18 01:10:39.286596759 (592h0m14.057238094s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.286620685 (592h0m14.05726202s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287116924 (592h0m14.057758259s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287176458 (592h0m14.057817793s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287488926 (592h0m14.058130261s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287772308 (592h0m14.058413643s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287863583 (592h0m14.058504918s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288079506 (592h0m14.058720841s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288237839 (592h0m14.058879184s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288418594 (592h0m14.059059929s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288629087 (592h0m14.059270422s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289488541 (592h0m14.060129877s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289738971 (592h0m14.060380306s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289989741 (592h0m14.060631086s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.290216545 (592h0m14.06085788s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.297416701 (592h0m14.068058036s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:41.030217031 (592h0m15.800858356s) [upload=473459]: Next returned nil segment Jul 18 01:10:41.030234745 (592h0m15.80087608s) [upload=473459]: Waiting for error group managing segments... Jul 18 01:10:41.308832612 (592h0m16.079473947s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.358887261 (592h0m16.129528596s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.365986644 (592h0m16.136627979s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.375224368 (592h0m16.145865704s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.419739118 (592h0m16.190380453s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.430360108 (592h0m16.201001444s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.450861434 (592h0m16.221502769s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468109615 (592h0m16.23875095s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468727326 (592h0m16.239368661s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.516013518 (592h0m16.286654863s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.522587155 (592h0m16.29322849s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.553218502 (592h0m16.323859837s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:11:08.331622589 (592h0m43.102263924s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38902->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331668216 (592h0m43.102309551s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:08.331710026 (592h0m43.102351361s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:60806->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331772906 (592h0m43.102414241s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331804857 (592h0m43.102446182s) [seg_pos={0 0}, upload=473459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 01:11:08.598012820 (592h0m43.368654156s) [upload=473459]: Done uploading segments... Jul 18 01:11:08.598041896 (592h0m43.368683231s) [upload=473459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.02853882s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X.X......................] duration: 5.898247025s FAIL (2025-07-18 01:26:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== T467KJ2SHEV5EJW2LJCYJSSHAG2CJIU2A, seg_pos={0 0}, upload=473444]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: write tcp 188.166.225.170:43510->77.90.31.16:4733: use of closed network connection; context canceled Jul 18 01:09:36.439338648 (591h59m11.209979983s) [seg_pos={0 0}, upload=473444]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 01:09:36.439407128 (591h59m11.210048464s) [upload=473444]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 01:09:36.439411076 (591h59m11.210052411s) [upload=473444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 01:09:36.756883131 (591h59m11.527524466s) [upload=473444]: Done uploading segments... Jul 18 01:09:36.756897708 (591h59m11.527539044s) [upload=473444]: Upload finished. err: <nil> Jul 18 01:10:39.025598816 (592h0m13.796240151s) [upload=473459]: Starting upload Jul 18 01:10:39.025658440 (592h0m13.796299765s) [upload=473459]: Done starting upload Jul 18 01:10:39.025759353 (592h0m13.796400688s) [upload=473459]: Uploading segments... Jul 18 01:10:39.025907185 (592h0m13.79654852s) [upload=473459]: Got next segment. Inline: false Jul 18 01:10:39.025950799 (592h0m13.796592124s) [upload=473459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 01:10:39.025957632 (592h0m13.796598967s) [upload=473459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 01:10:39.286495526 (592h0m14.057136861s) [seg_pos={0 0}, upload=473459]: Begin upload segment... Jul 18 01:10:39.286547334 (592h0m14.057188699s) [seg_pos={0 0}, upload=473459]: Done begin upload segment. Jul 18 01:10:39.286596759 (592h0m14.057238094s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.286620685 (592h0m14.05726202s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287116924 (592h0m14.057758259s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287176458 (592h0m14.057817793s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287488926 (592h0m14.058130261s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287772308 (592h0m14.058413643s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287863583 (592h0m14.058504918s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288079506 (592h0m14.058720841s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288237839 (592h0m14.058879184s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288418594 (592h0m14.059059929s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288629087 (592h0m14.059270422s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289488541 (592h0m14.060129877s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289738971 (592h0m14.060380306s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289989741 (592h0m14.060631086s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.290216545 (592h0m14.06085788s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.297416701 (592h0m14.068058036s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:41.030217031 (592h0m15.800858356s) [upload=473459]: Next returned nil segment Jul 18 01:10:41.030234745 (592h0m15.80087608s) [upload=473459]: Waiting for error group managing segments... Jul 18 01:10:41.308832612 (592h0m16.079473947s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.358887261 (592h0m16.129528596s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.365986644 (592h0m16.136627979s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.375224368 (592h0m16.145865704s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.419739118 (592h0m16.190380453s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.430360108 (592h0m16.201001444s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.450861434 (592h0m16.221502769s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468109615 (592h0m16.23875095s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468727326 (592h0m16.239368661s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.516013518 (592h0m16.286654863s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.522587155 (592h0m16.29322849s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.553218502 (592h0m16.323859837s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:11:08.331622589 (592h0m43.102263924s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38902->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331668216 (592h0m43.102309551s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:08.331710026 (592h0m43.102351361s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:60806->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331772906 (592h0m43.102414241s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331804857 (592h0m43.102446182s) [seg_pos={0 0}, upload=473459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 01:11:08.598012820 (592h0m43.368654156s) [upload=473459]: Done uploading segments... Jul 18 01:11:08.598041896 (592h0m43.368683231s) [upload=473459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.898272284s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......................................XX....................] duration: 5.812523754s FAIL (2025-07-18 01:26:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ress=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683391198 (592h0m30.454032533s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683761526 (592h0m30.454402861s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683801033 (592h0m30.454442368s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684063464 (592h0m30.454704799s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684360503 (592h0m30.455001838s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684654846 (592h0m30.455296181s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685431742 (592h0m30.456073077s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685861425 (592h0m30.45650276s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686139176 (592h0m30.456780511s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686424722 (592h0m30.457066057s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686687836 (592h0m30.457329171s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686944527 (592h0m30.457585862s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.687208402 (592h0m30.457849737s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688069810 (592h0m30.458711145s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688391375 (592h0m30.45903271s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688755091 (592h0m30.459396426s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.689020598 (592h0m30.459661934s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:56.791786484 (592h0m31.562427839s) [upload=473462]: Next returned nil segment Jul 18 01:10:56.791810180 (592h0m31.562451535s) [upload=473462]: Waiting for error group managing segments... Jul 18 01:10:57.072508846 (592h0m31.843150181s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.084313581 (592h0m31.854954917s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.130829109 (592h0m31.901470444s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.380019692 (592h0m32.150661027s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.586116023 (592h0m32.356757358s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.590030633 (592h0m32.360671968s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.644593087 (592h0m32.415234412s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.711170438 (592h0m32.481811773s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.815350524 (592h0m32.585991869s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:11:54.506942141 (592h1m29.277583476s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: rpc: dial tcp 77.90.31.13:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.506990634 (592h1m29.277631959s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507021904 (592h1m29.277663239s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: rpc: dial tcp 77.90.31.19:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507055037 (592h1m29.277696372s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: rpc: dial tcp 77.90.31.18:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507085656 (592h1m29.277726992s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 188.166.225.170:42120->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507153856 (592h1m29.277795192s) [noise=true , address=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj, address:77.90.31.15:4704): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507228530 (592h1m29.277869865s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507325304 (592h1m29.277966639s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12eUC555Lzo37B5FafAFMfXjSkfm1zv1pRZmH71jf4xpdCWH6kG, address:77.90.31.21:4904): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507433642 (592h1m29.278074977s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507448881 (592h1m29.278090216s) [seg_pos={0 0}, upload=473462]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 18 01:11:54.756684719 (592h1m29.527326054s) [upload=473462]: Done uploading segments... Jul 18 01:11:54.756710668 (592h1m29.527352004s) [upload=473462]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 5.812548752s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X.X.....................] duration: 6.003467052s FAIL (2025-07-18 01:26:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== T467KJ2SHEV5EJW2LJCYJSSHAG2CJIU2A, seg_pos={0 0}, upload=473444]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: write tcp 188.166.225.170:43510->77.90.31.16:4733: use of closed network connection; context canceled Jul 18 01:09:36.439338648 (591h59m11.209979983s) [seg_pos={0 0}, upload=473444]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 01:09:36.439407128 (591h59m11.210048464s) [upload=473444]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 01:09:36.439411076 (591h59m11.210052411s) [upload=473444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 01:09:36.756883131 (591h59m11.527524466s) [upload=473444]: Done uploading segments... Jul 18 01:09:36.756897708 (591h59m11.527539044s) [upload=473444]: Upload finished. err: <nil> Jul 18 01:10:39.025598816 (592h0m13.796240151s) [upload=473459]: Starting upload Jul 18 01:10:39.025658440 (592h0m13.796299765s) [upload=473459]: Done starting upload Jul 18 01:10:39.025759353 (592h0m13.796400688s) [upload=473459]: Uploading segments... Jul 18 01:10:39.025907185 (592h0m13.79654852s) [upload=473459]: Got next segment. Inline: false Jul 18 01:10:39.025950799 (592h0m13.796592124s) [upload=473459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 01:10:39.025957632 (592h0m13.796598967s) [upload=473459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 01:10:39.286495526 (592h0m14.057136861s) [seg_pos={0 0}, upload=473459]: Begin upload segment... Jul 18 01:10:39.286547334 (592h0m14.057188699s) [seg_pos={0 0}, upload=473459]: Done begin upload segment. Jul 18 01:10:39.286596759 (592h0m14.057238094s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.286620685 (592h0m14.05726202s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287116924 (592h0m14.057758259s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287176458 (592h0m14.057817793s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287488926 (592h0m14.058130261s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287772308 (592h0m14.058413643s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287863583 (592h0m14.058504918s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288079506 (592h0m14.058720841s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288237839 (592h0m14.058879184s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288418594 (592h0m14.059059929s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288629087 (592h0m14.059270422s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289488541 (592h0m14.060129877s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289738971 (592h0m14.060380306s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289989741 (592h0m14.060631086s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.290216545 (592h0m14.06085788s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.297416701 (592h0m14.068058036s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:41.030217031 (592h0m15.800858356s) [upload=473459]: Next returned nil segment Jul 18 01:10:41.030234745 (592h0m15.80087608s) [upload=473459]: Waiting for error group managing segments... Jul 18 01:10:41.308832612 (592h0m16.079473947s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.358887261 (592h0m16.129528596s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.365986644 (592h0m16.136627979s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.375224368 (592h0m16.145865704s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.419739118 (592h0m16.190380453s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.430360108 (592h0m16.201001444s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.450861434 (592h0m16.221502769s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468109615 (592h0m16.23875095s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468727326 (592h0m16.239368661s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.516013518 (592h0m16.286654863s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.522587155 (592h0m16.29322849s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.553218502 (592h0m16.323859837s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:11:08.331622589 (592h0m43.102263924s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38902->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331668216 (592h0m43.102309551s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:08.331710026 (592h0m43.102351361s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:60806->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331772906 (592h0m43.102414241s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331804857 (592h0m43.102446182s) [seg_pos={0 0}, upload=473459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 01:11:08.598012820 (592h0m43.368654156s) [upload=473459]: Done uploading segments... Jul 18 01:11:08.598041896 (592h0m43.368683231s) [upload=473459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.003495886s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......................................XX...................] duration: 5.89224312s FAIL (2025-07-18 01:25:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ress=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683391198 (592h0m30.454032533s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683761526 (592h0m30.454402861s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683801033 (592h0m30.454442368s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684063464 (592h0m30.454704799s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684360503 (592h0m30.455001838s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684654846 (592h0m30.455296181s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685431742 (592h0m30.456073077s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685861425 (592h0m30.45650276s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686139176 (592h0m30.456780511s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686424722 (592h0m30.457066057s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686687836 (592h0m30.457329171s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686944527 (592h0m30.457585862s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.687208402 (592h0m30.457849737s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688069810 (592h0m30.458711145s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688391375 (592h0m30.45903271s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688755091 (592h0m30.459396426s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.689020598 (592h0m30.459661934s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:56.791786484 (592h0m31.562427839s) [upload=473462]: Next returned nil segment Jul 18 01:10:56.791810180 (592h0m31.562451535s) [upload=473462]: Waiting for error group managing segments... Jul 18 01:10:57.072508846 (592h0m31.843150181s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.084313581 (592h0m31.854954917s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.130829109 (592h0m31.901470444s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.380019692 (592h0m32.150661027s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.586116023 (592h0m32.356757358s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.590030633 (592h0m32.360671968s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.644593087 (592h0m32.415234412s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.711170438 (592h0m32.481811773s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.815350524 (592h0m32.585991869s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:11:54.506942141 (592h1m29.277583476s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: rpc: dial tcp 77.90.31.13:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.506990634 (592h1m29.277631959s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507021904 (592h1m29.277663239s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: rpc: dial tcp 77.90.31.19:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507055037 (592h1m29.277696372s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: rpc: dial tcp 77.90.31.18:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507085656 (592h1m29.277726992s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 188.166.225.170:42120->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507153856 (592h1m29.277795192s) [noise=true , address=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj, address:77.90.31.15:4704): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507228530 (592h1m29.277869865s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507325304 (592h1m29.277966639s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12eUC555Lzo37B5FafAFMfXjSkfm1zv1pRZmH71jf4xpdCWH6kG, address:77.90.31.21:4904): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507433642 (592h1m29.278074977s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507448881 (592h1m29.278090216s) [seg_pos={0 0}, upload=473462]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 18 01:11:54.756684719 (592h1m29.527326054s) [upload=473462]: Done uploading segments... Jul 18 01:11:54.756710668 (592h1m29.527352004s) [upload=473462]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 5.892272135s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X.X....................] duration: 5.916394888s FAIL (2025-07-18 01:25:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== T467KJ2SHEV5EJW2LJCYJSSHAG2CJIU2A, seg_pos={0 0}, upload=473444]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: write tcp 188.166.225.170:43510->77.90.31.16:4733: use of closed network connection; context canceled Jul 18 01:09:36.439338648 (591h59m11.209979983s) [seg_pos={0 0}, upload=473444]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 01:09:36.439407128 (591h59m11.210048464s) [upload=473444]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 01:09:36.439411076 (591h59m11.210052411s) [upload=473444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 01:09:36.756883131 (591h59m11.527524466s) [upload=473444]: Done uploading segments... Jul 18 01:09:36.756897708 (591h59m11.527539044s) [upload=473444]: Upload finished. err: <nil> Jul 18 01:10:39.025598816 (592h0m13.796240151s) [upload=473459]: Starting upload Jul 18 01:10:39.025658440 (592h0m13.796299765s) [upload=473459]: Done starting upload Jul 18 01:10:39.025759353 (592h0m13.796400688s) [upload=473459]: Uploading segments... Jul 18 01:10:39.025907185 (592h0m13.79654852s) [upload=473459]: Got next segment. Inline: false Jul 18 01:10:39.025950799 (592h0m13.796592124s) [upload=473459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 01:10:39.025957632 (592h0m13.796598967s) [upload=473459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 01:10:39.286495526 (592h0m14.057136861s) [seg_pos={0 0}, upload=473459]: Begin upload segment... Jul 18 01:10:39.286547334 (592h0m14.057188699s) [seg_pos={0 0}, upload=473459]: Done begin upload segment. Jul 18 01:10:39.286596759 (592h0m14.057238094s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.286620685 (592h0m14.05726202s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287116924 (592h0m14.057758259s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287176458 (592h0m14.057817793s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287488926 (592h0m14.058130261s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287772308 (592h0m14.058413643s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287863583 (592h0m14.058504918s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288079506 (592h0m14.058720841s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288237839 (592h0m14.058879184s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288418594 (592h0m14.059059929s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288629087 (592h0m14.059270422s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289488541 (592h0m14.060129877s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289738971 (592h0m14.060380306s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289989741 (592h0m14.060631086s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.290216545 (592h0m14.06085788s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.297416701 (592h0m14.068058036s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:41.030217031 (592h0m15.800858356s) [upload=473459]: Next returned nil segment Jul 18 01:10:41.030234745 (592h0m15.80087608s) [upload=473459]: Waiting for error group managing segments... Jul 18 01:10:41.308832612 (592h0m16.079473947s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.358887261 (592h0m16.129528596s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.365986644 (592h0m16.136627979s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.375224368 (592h0m16.145865704s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.419739118 (592h0m16.190380453s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.430360108 (592h0m16.201001444s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.450861434 (592h0m16.221502769s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468109615 (592h0m16.23875095s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468727326 (592h0m16.239368661s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.516013518 (592h0m16.286654863s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.522587155 (592h0m16.29322849s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.553218502 (592h0m16.323859837s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:11:08.331622589 (592h0m43.102263924s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38902->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331668216 (592h0m43.102309551s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:08.331710026 (592h0m43.102351361s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:60806->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331772906 (592h0m43.102414241s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331804857 (592h0m43.102446182s) [seg_pos={0 0}, upload=473459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 01:11:08.598012820 (592h0m43.368654156s) [upload=473459]: Done uploading segments... Jul 18 01:11:08.598041896 (592h0m43.368683231s) [upload=473459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.916420266s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................X.X...................] duration: 6.00161027s FAIL (2025-07-18 01:24:44 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== T467KJ2SHEV5EJW2LJCYJSSHAG2CJIU2A, seg_pos={0 0}, upload=473444]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: write tcp 188.166.225.170:43510->77.90.31.16:4733: use of closed network connection; context canceled Jul 18 01:09:36.439338648 (591h59m11.209979983s) [seg_pos={0 0}, upload=473444]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 01:09:36.439407128 (591h59m11.210048464s) [upload=473444]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 01:09:36.439411076 (591h59m11.210052411s) [upload=473444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 01:09:36.756883131 (591h59m11.527524466s) [upload=473444]: Done uploading segments... Jul 18 01:09:36.756897708 (591h59m11.527539044s) [upload=473444]: Upload finished. err: <nil> Jul 18 01:10:39.025598816 (592h0m13.796240151s) [upload=473459]: Starting upload Jul 18 01:10:39.025658440 (592h0m13.796299765s) [upload=473459]: Done starting upload Jul 18 01:10:39.025759353 (592h0m13.796400688s) [upload=473459]: Uploading segments... Jul 18 01:10:39.025907185 (592h0m13.79654852s) [upload=473459]: Got next segment. Inline: false Jul 18 01:10:39.025950799 (592h0m13.796592124s) [upload=473459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 01:10:39.025957632 (592h0m13.796598967s) [upload=473459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 01:10:39.286495526 (592h0m14.057136861s) [seg_pos={0 0}, upload=473459]: Begin upload segment... Jul 18 01:10:39.286547334 (592h0m14.057188699s) [seg_pos={0 0}, upload=473459]: Done begin upload segment. Jul 18 01:10:39.286596759 (592h0m14.057238094s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.286620685 (592h0m14.05726202s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287116924 (592h0m14.057758259s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287176458 (592h0m14.057817793s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287488926 (592h0m14.058130261s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287772308 (592h0m14.058413643s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287863583 (592h0m14.058504918s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288079506 (592h0m14.058720841s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288237839 (592h0m14.058879184s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288418594 (592h0m14.059059929s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288629087 (592h0m14.059270422s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289488541 (592h0m14.060129877s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289738971 (592h0m14.060380306s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289989741 (592h0m14.060631086s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.290216545 (592h0m14.06085788s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.297416701 (592h0m14.068058036s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:41.030217031 (592h0m15.800858356s) [upload=473459]: Next returned nil segment Jul 18 01:10:41.030234745 (592h0m15.80087608s) [upload=473459]: Waiting for error group managing segments... Jul 18 01:10:41.308832612 (592h0m16.079473947s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.358887261 (592h0m16.129528596s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.365986644 (592h0m16.136627979s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.375224368 (592h0m16.145865704s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.419739118 (592h0m16.190380453s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.430360108 (592h0m16.201001444s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.450861434 (592h0m16.221502769s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468109615 (592h0m16.23875095s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468727326 (592h0m16.239368661s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.516013518 (592h0m16.286654863s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.522587155 (592h0m16.29322849s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.553218502 (592h0m16.323859837s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:11:08.331622589 (592h0m43.102263924s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38902->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331668216 (592h0m43.102309551s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:08.331710026 (592h0m43.102351361s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:60806->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331772906 (592h0m43.102414241s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331804857 (592h0m43.102446182s) [seg_pos={0 0}, upload=473459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 01:11:08.598012820 (592h0m43.368654156s) [upload=473459]: Done uploading segments... Jul 18 01:11:08.598041896 (592h0m43.368683231s) [upload=473459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.001641631s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........................................XX..................] duration: 5.809220182s FAIL (2025-07-18 01:24:38 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ress=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683391198 (592h0m30.454032533s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683761526 (592h0m30.454402861s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683801033 (592h0m30.454442368s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684063464 (592h0m30.454704799s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684360503 (592h0m30.455001838s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684654846 (592h0m30.455296181s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685431742 (592h0m30.456073077s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685861425 (592h0m30.45650276s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686139176 (592h0m30.456780511s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686424722 (592h0m30.457066057s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686687836 (592h0m30.457329171s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686944527 (592h0m30.457585862s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.687208402 (592h0m30.457849737s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688069810 (592h0m30.458711145s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688391375 (592h0m30.45903271s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688755091 (592h0m30.459396426s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.689020598 (592h0m30.459661934s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:56.791786484 (592h0m31.562427839s) [upload=473462]: Next returned nil segment Jul 18 01:10:56.791810180 (592h0m31.562451535s) [upload=473462]: Waiting for error group managing segments... Jul 18 01:10:57.072508846 (592h0m31.843150181s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.084313581 (592h0m31.854954917s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.130829109 (592h0m31.901470444s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.380019692 (592h0m32.150661027s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.586116023 (592h0m32.356757358s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.590030633 (592h0m32.360671968s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.644593087 (592h0m32.415234412s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.711170438 (592h0m32.481811773s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.815350524 (592h0m32.585991869s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:11:54.506942141 (592h1m29.277583476s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: rpc: dial tcp 77.90.31.13:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.506990634 (592h1m29.277631959s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507021904 (592h1m29.277663239s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: rpc: dial tcp 77.90.31.19:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507055037 (592h1m29.277696372s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: rpc: dial tcp 77.90.31.18:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507085656 (592h1m29.277726992s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 188.166.225.170:42120->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507153856 (592h1m29.277795192s) [noise=true , address=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj, address:77.90.31.15:4704): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507228530 (592h1m29.277869865s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507325304 (592h1m29.277966639s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12eUC555Lzo37B5FafAFMfXjSkfm1zv1pRZmH71jf4xpdCWH6kG, address:77.90.31.21:4904): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507433642 (592h1m29.278074977s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507448881 (592h1m29.278090216s) [seg_pos={0 0}, upload=473462]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 18 01:11:54.756684719 (592h1m29.527326054s) [upload=473462]: Done uploading segments... Jul 18 01:11:54.756710668 (592h1m29.527352004s) [upload=473462]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 5.809247342s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X.X..................] duration: 5.904948458s FAIL (2025-07-18 01:24:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== T467KJ2SHEV5EJW2LJCYJSSHAG2CJIU2A, seg_pos={0 0}, upload=473444]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: write tcp 188.166.225.170:43510->77.90.31.16:4733: use of closed network connection; context canceled Jul 18 01:09:36.439338648 (591h59m11.209979983s) [seg_pos={0 0}, upload=473444]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 01:09:36.439407128 (591h59m11.210048464s) [upload=473444]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 01:09:36.439411076 (591h59m11.210052411s) [upload=473444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 01:09:36.756883131 (591h59m11.527524466s) [upload=473444]: Done uploading segments... Jul 18 01:09:36.756897708 (591h59m11.527539044s) [upload=473444]: Upload finished. err: <nil> Jul 18 01:10:39.025598816 (592h0m13.796240151s) [upload=473459]: Starting upload Jul 18 01:10:39.025658440 (592h0m13.796299765s) [upload=473459]: Done starting upload Jul 18 01:10:39.025759353 (592h0m13.796400688s) [upload=473459]: Uploading segments... Jul 18 01:10:39.025907185 (592h0m13.79654852s) [upload=473459]: Got next segment. Inline: false Jul 18 01:10:39.025950799 (592h0m13.796592124s) [upload=473459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 01:10:39.025957632 (592h0m13.796598967s) [upload=473459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 01:10:39.286495526 (592h0m14.057136861s) [seg_pos={0 0}, upload=473459]: Begin upload segment... Jul 18 01:10:39.286547334 (592h0m14.057188699s) [seg_pos={0 0}, upload=473459]: Done begin upload segment. Jul 18 01:10:39.286596759 (592h0m14.057238094s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.286620685 (592h0m14.05726202s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287116924 (592h0m14.057758259s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287176458 (592h0m14.057817793s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287488926 (592h0m14.058130261s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287772308 (592h0m14.058413643s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287863583 (592h0m14.058504918s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288079506 (592h0m14.058720841s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288237839 (592h0m14.058879184s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288418594 (592h0m14.059059929s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288629087 (592h0m14.059270422s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289488541 (592h0m14.060129877s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289738971 (592h0m14.060380306s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289989741 (592h0m14.060631086s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.290216545 (592h0m14.06085788s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.297416701 (592h0m14.068058036s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:41.030217031 (592h0m15.800858356s) [upload=473459]: Next returned nil segment Jul 18 01:10:41.030234745 (592h0m15.80087608s) [upload=473459]: Waiting for error group managing segments... Jul 18 01:10:41.308832612 (592h0m16.079473947s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.358887261 (592h0m16.129528596s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.365986644 (592h0m16.136627979s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.375224368 (592h0m16.145865704s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.419739118 (592h0m16.190380453s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.430360108 (592h0m16.201001444s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.450861434 (592h0m16.221502769s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468109615 (592h0m16.23875095s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468727326 (592h0m16.239368661s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.516013518 (592h0m16.286654863s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.522587155 (592h0m16.29322849s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.553218502 (592h0m16.323859837s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:11:08.331622589 (592h0m43.102263924s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38902->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331668216 (592h0m43.102309551s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:08.331710026 (592h0m43.102351361s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:60806->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331772906 (592h0m43.102414241s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331804857 (592h0m43.102446182s) [seg_pos={0 0}, upload=473459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 01:11:08.598012820 (592h0m43.368654156s) [upload=473459]: Done uploading segments... Jul 18 01:11:08.598041896 (592h0m43.368683231s) [upload=473459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.904977243s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........................................XX.................] duration: 5.982067797s FAIL (2025-07-18 01:24:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ress=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683391198 (592h0m30.454032533s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683761526 (592h0m30.454402861s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683801033 (592h0m30.454442368s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684063464 (592h0m30.454704799s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684360503 (592h0m30.455001838s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684654846 (592h0m30.455296181s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685431742 (592h0m30.456073077s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685861425 (592h0m30.45650276s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686139176 (592h0m30.456780511s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686424722 (592h0m30.457066057s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686687836 (592h0m30.457329171s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686944527 (592h0m30.457585862s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.687208402 (592h0m30.457849737s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688069810 (592h0m30.458711145s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688391375 (592h0m30.45903271s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688755091 (592h0m30.459396426s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.689020598 (592h0m30.459661934s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:56.791786484 (592h0m31.562427839s) [upload=473462]: Next returned nil segment Jul 18 01:10:56.791810180 (592h0m31.562451535s) [upload=473462]: Waiting for error group managing segments... Jul 18 01:10:57.072508846 (592h0m31.843150181s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.084313581 (592h0m31.854954917s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.130829109 (592h0m31.901470444s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.380019692 (592h0m32.150661027s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.586116023 (592h0m32.356757358s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.590030633 (592h0m32.360671968s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.644593087 (592h0m32.415234412s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.711170438 (592h0m32.481811773s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.815350524 (592h0m32.585991869s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:11:54.506942141 (592h1m29.277583476s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: rpc: dial tcp 77.90.31.13:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.506990634 (592h1m29.277631959s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507021904 (592h1m29.277663239s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: rpc: dial tcp 77.90.31.19:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507055037 (592h1m29.277696372s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: rpc: dial tcp 77.90.31.18:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507085656 (592h1m29.277726992s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 188.166.225.170:42120->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507153856 (592h1m29.277795192s) [noise=true , address=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj, address:77.90.31.15:4704): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507228530 (592h1m29.277869865s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507325304 (592h1m29.277966639s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12eUC555Lzo37B5FafAFMfXjSkfm1zv1pRZmH71jf4xpdCWH6kG, address:77.90.31.21:4904): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507433642 (592h1m29.278074977s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507448881 (592h1m29.278090216s) [seg_pos={0 0}, upload=473462]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 18 01:11:54.756684719 (592h1m29.527326054s) [upload=473462]: Done uploading segments... Jul 18 01:11:54.756710668 (592h1m29.527352004s) [upload=473462]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 5.982092736s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................X.X.................] duration: 6.136930728s FAIL (2025-07-18 01:23:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== T467KJ2SHEV5EJW2LJCYJSSHAG2CJIU2A, seg_pos={0 0}, upload=473444]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: write tcp 188.166.225.170:43510->77.90.31.16:4733: use of closed network connection; context canceled Jul 18 01:09:36.439338648 (591h59m11.209979983s) [seg_pos={0 0}, upload=473444]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 01:09:36.439407128 (591h59m11.210048464s) [upload=473444]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 01:09:36.439411076 (591h59m11.210052411s) [upload=473444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 01:09:36.756883131 (591h59m11.527524466s) [upload=473444]: Done uploading segments... Jul 18 01:09:36.756897708 (591h59m11.527539044s) [upload=473444]: Upload finished. err: <nil> Jul 18 01:10:39.025598816 (592h0m13.796240151s) [upload=473459]: Starting upload Jul 18 01:10:39.025658440 (592h0m13.796299765s) [upload=473459]: Done starting upload Jul 18 01:10:39.025759353 (592h0m13.796400688s) [upload=473459]: Uploading segments... Jul 18 01:10:39.025907185 (592h0m13.79654852s) [upload=473459]: Got next segment. Inline: false Jul 18 01:10:39.025950799 (592h0m13.796592124s) [upload=473459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 01:10:39.025957632 (592h0m13.796598967s) [upload=473459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 01:10:39.286495526 (592h0m14.057136861s) [seg_pos={0 0}, upload=473459]: Begin upload segment... Jul 18 01:10:39.286547334 (592h0m14.057188699s) [seg_pos={0 0}, upload=473459]: Done begin upload segment. Jul 18 01:10:39.286596759 (592h0m14.057238094s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.286620685 (592h0m14.05726202s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287116924 (592h0m14.057758259s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287176458 (592h0m14.057817793s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287488926 (592h0m14.058130261s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287772308 (592h0m14.058413643s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287863583 (592h0m14.058504918s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288079506 (592h0m14.058720841s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288237839 (592h0m14.058879184s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288418594 (592h0m14.059059929s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288629087 (592h0m14.059270422s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289488541 (592h0m14.060129877s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289738971 (592h0m14.060380306s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289989741 (592h0m14.060631086s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.290216545 (592h0m14.06085788s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.297416701 (592h0m14.068058036s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:41.030217031 (592h0m15.800858356s) [upload=473459]: Next returned nil segment Jul 18 01:10:41.030234745 (592h0m15.80087608s) [upload=473459]: Waiting for error group managing segments... Jul 18 01:10:41.308832612 (592h0m16.079473947s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.358887261 (592h0m16.129528596s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.365986644 (592h0m16.136627979s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.375224368 (592h0m16.145865704s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.419739118 (592h0m16.190380453s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.430360108 (592h0m16.201001444s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.450861434 (592h0m16.221502769s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468109615 (592h0m16.23875095s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468727326 (592h0m16.239368661s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.516013518 (592h0m16.286654863s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.522587155 (592h0m16.29322849s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.553218502 (592h0m16.323859837s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:11:08.331622589 (592h0m43.102263924s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38902->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331668216 (592h0m43.102309551s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:08.331710026 (592h0m43.102351361s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:60806->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331772906 (592h0m43.102414241s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331804857 (592h0m43.102446182s) [seg_pos={0 0}, upload=473459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 01:11:08.598012820 (592h0m43.368654156s) [upload=473459]: Done uploading segments... Jul 18 01:11:08.598041896 (592h0m43.368683231s) [upload=473459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.136961555s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........................................XX................] duration: 5.771540139s FAIL (2025-07-18 01:23:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ress=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683391198 (592h0m30.454032533s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683761526 (592h0m30.454402861s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683801033 (592h0m30.454442368s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684063464 (592h0m30.454704799s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684360503 (592h0m30.455001838s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684654846 (592h0m30.455296181s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685431742 (592h0m30.456073077s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685861425 (592h0m30.45650276s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686139176 (592h0m30.456780511s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686424722 (592h0m30.457066057s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686687836 (592h0m30.457329171s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686944527 (592h0m30.457585862s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.687208402 (592h0m30.457849737s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688069810 (592h0m30.458711145s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688391375 (592h0m30.45903271s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688755091 (592h0m30.459396426s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.689020598 (592h0m30.459661934s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:56.791786484 (592h0m31.562427839s) [upload=473462]: Next returned nil segment Jul 18 01:10:56.791810180 (592h0m31.562451535s) [upload=473462]: Waiting for error group managing segments... Jul 18 01:10:57.072508846 (592h0m31.843150181s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.084313581 (592h0m31.854954917s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.130829109 (592h0m31.901470444s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.380019692 (592h0m32.150661027s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.586116023 (592h0m32.356757358s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.590030633 (592h0m32.360671968s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.644593087 (592h0m32.415234412s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.711170438 (592h0m32.481811773s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.815350524 (592h0m32.585991869s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:11:54.506942141 (592h1m29.277583476s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: rpc: dial tcp 77.90.31.13:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.506990634 (592h1m29.277631959s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507021904 (592h1m29.277663239s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: rpc: dial tcp 77.90.31.19:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507055037 (592h1m29.277696372s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: rpc: dial tcp 77.90.31.18:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507085656 (592h1m29.277726992s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 188.166.225.170:42120->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507153856 (592h1m29.277795192s) [noise=true , address=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj, address:77.90.31.15:4704): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507228530 (592h1m29.277869865s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507325304 (592h1m29.277966639s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12eUC555Lzo37B5FafAFMfXjSkfm1zv1pRZmH71jf4xpdCWH6kG, address:77.90.31.21:4904): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507433642 (592h1m29.278074977s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507448881 (592h1m29.278090216s) [seg_pos={0 0}, upload=473462]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 18 01:11:54.756684719 (592h1m29.527326054s) [upload=473462]: Done uploading segments... Jul 18 01:11:54.756710668 (592h1m29.527352004s) [upload=473462]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 5.771570047s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X.X................] duration: 5.929757963s FAIL (2025-07-18 01:23:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== T467KJ2SHEV5EJW2LJCYJSSHAG2CJIU2A, seg_pos={0 0}, upload=473444]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: write tcp 188.166.225.170:43510->77.90.31.16:4733: use of closed network connection; context canceled Jul 18 01:09:36.439338648 (591h59m11.209979983s) [seg_pos={0 0}, upload=473444]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 01:09:36.439407128 (591h59m11.210048464s) [upload=473444]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 01:09:36.439411076 (591h59m11.210052411s) [upload=473444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 01:09:36.756883131 (591h59m11.527524466s) [upload=473444]: Done uploading segments... Jul 18 01:09:36.756897708 (591h59m11.527539044s) [upload=473444]: Upload finished. err: <nil> Jul 18 01:10:39.025598816 (592h0m13.796240151s) [upload=473459]: Starting upload Jul 18 01:10:39.025658440 (592h0m13.796299765s) [upload=473459]: Done starting upload Jul 18 01:10:39.025759353 (592h0m13.796400688s) [upload=473459]: Uploading segments... Jul 18 01:10:39.025907185 (592h0m13.79654852s) [upload=473459]: Got next segment. Inline: false Jul 18 01:10:39.025950799 (592h0m13.796592124s) [upload=473459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 01:10:39.025957632 (592h0m13.796598967s) [upload=473459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 01:10:39.286495526 (592h0m14.057136861s) [seg_pos={0 0}, upload=473459]: Begin upload segment... Jul 18 01:10:39.286547334 (592h0m14.057188699s) [seg_pos={0 0}, upload=473459]: Done begin upload segment. Jul 18 01:10:39.286596759 (592h0m14.057238094s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.286620685 (592h0m14.05726202s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287116924 (592h0m14.057758259s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287176458 (592h0m14.057817793s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287488926 (592h0m14.058130261s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287772308 (592h0m14.058413643s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287863583 (592h0m14.058504918s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288079506 (592h0m14.058720841s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288237839 (592h0m14.058879184s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288418594 (592h0m14.059059929s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288629087 (592h0m14.059270422s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289488541 (592h0m14.060129877s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289738971 (592h0m14.060380306s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289989741 (592h0m14.060631086s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.290216545 (592h0m14.06085788s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.297416701 (592h0m14.068058036s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:41.030217031 (592h0m15.800858356s) [upload=473459]: Next returned nil segment Jul 18 01:10:41.030234745 (592h0m15.80087608s) [upload=473459]: Waiting for error group managing segments... Jul 18 01:10:41.308832612 (592h0m16.079473947s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.358887261 (592h0m16.129528596s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.365986644 (592h0m16.136627979s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.375224368 (592h0m16.145865704s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.419739118 (592h0m16.190380453s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.430360108 (592h0m16.201001444s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.450861434 (592h0m16.221502769s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468109615 (592h0m16.23875095s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468727326 (592h0m16.239368661s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.516013518 (592h0m16.286654863s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.522587155 (592h0m16.29322849s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.553218502 (592h0m16.323859837s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:11:08.331622589 (592h0m43.102263924s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38902->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331668216 (592h0m43.102309551s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:08.331710026 (592h0m43.102351361s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:60806->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331772906 (592h0m43.102414241s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331804857 (592h0m43.102446182s) [seg_pos={0 0}, upload=473459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 01:11:08.598012820 (592h0m43.368654156s) [upload=473459]: Done uploading segments... Jul 18 01:11:08.598041896 (592h0m43.368683231s) [upload=473459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.92978759s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........................................XX...............] duration: 5.902658816s FAIL (2025-07-18 01:22:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ress=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683391198 (592h0m30.454032533s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683761526 (592h0m30.454402861s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683801033 (592h0m30.454442368s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684063464 (592h0m30.454704799s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684360503 (592h0m30.455001838s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684654846 (592h0m30.455296181s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685431742 (592h0m30.456073077s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685861425 (592h0m30.45650276s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686139176 (592h0m30.456780511s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686424722 (592h0m30.457066057s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686687836 (592h0m30.457329171s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686944527 (592h0m30.457585862s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.687208402 (592h0m30.457849737s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688069810 (592h0m30.458711145s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688391375 (592h0m30.45903271s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688755091 (592h0m30.459396426s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.689020598 (592h0m30.459661934s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:56.791786484 (592h0m31.562427839s) [upload=473462]: Next returned nil segment Jul 18 01:10:56.791810180 (592h0m31.562451535s) [upload=473462]: Waiting for error group managing segments... Jul 18 01:10:57.072508846 (592h0m31.843150181s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.084313581 (592h0m31.854954917s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.130829109 (592h0m31.901470444s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.380019692 (592h0m32.150661027s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.586116023 (592h0m32.356757358s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.590030633 (592h0m32.360671968s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.644593087 (592h0m32.415234412s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.711170438 (592h0m32.481811773s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.815350524 (592h0m32.585991869s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:11:54.506942141 (592h1m29.277583476s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: rpc: dial tcp 77.90.31.13:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.506990634 (592h1m29.277631959s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507021904 (592h1m29.277663239s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: rpc: dial tcp 77.90.31.19:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507055037 (592h1m29.277696372s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: rpc: dial tcp 77.90.31.18:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507085656 (592h1m29.277726992s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 188.166.225.170:42120->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507153856 (592h1m29.277795192s) [noise=true , address=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj, address:77.90.31.15:4704): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507228530 (592h1m29.277869865s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507325304 (592h1m29.277966639s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12eUC555Lzo37B5FafAFMfXjSkfm1zv1pRZmH71jf4xpdCWH6kG, address:77.90.31.21:4904): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507433642 (592h1m29.278074977s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507448881 (592h1m29.278090216s) [seg_pos={0 0}, upload=473462]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 18 01:11:54.756684719 (592h1m29.527326054s) [upload=473462]: Done uploading segments... Jul 18 01:11:54.756710668 (592h1m29.527352004s) [upload=473462]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 5.902688983s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X.X...............] duration: 5.82302882s FAIL (2025-07-18 01:22:06 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== T467KJ2SHEV5EJW2LJCYJSSHAG2CJIU2A, seg_pos={0 0}, upload=473444]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: write tcp 188.166.225.170:43510->77.90.31.16:4733: use of closed network connection; context canceled Jul 18 01:09:36.439338648 (591h59m11.209979983s) [seg_pos={0 0}, upload=473444]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 01:09:36.439407128 (591h59m11.210048464s) [upload=473444]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 01:09:36.439411076 (591h59m11.210052411s) [upload=473444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 01:09:36.756883131 (591h59m11.527524466s) [upload=473444]: Done uploading segments... Jul 18 01:09:36.756897708 (591h59m11.527539044s) [upload=473444]: Upload finished. err: <nil> Jul 18 01:10:39.025598816 (592h0m13.796240151s) [upload=473459]: Starting upload Jul 18 01:10:39.025658440 (592h0m13.796299765s) [upload=473459]: Done starting upload Jul 18 01:10:39.025759353 (592h0m13.796400688s) [upload=473459]: Uploading segments... Jul 18 01:10:39.025907185 (592h0m13.79654852s) [upload=473459]: Got next segment. Inline: false Jul 18 01:10:39.025950799 (592h0m13.796592124s) [upload=473459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 01:10:39.025957632 (592h0m13.796598967s) [upload=473459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 01:10:39.286495526 (592h0m14.057136861s) [seg_pos={0 0}, upload=473459]: Begin upload segment... Jul 18 01:10:39.286547334 (592h0m14.057188699s) [seg_pos={0 0}, upload=473459]: Done begin upload segment. Jul 18 01:10:39.286596759 (592h0m14.057238094s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.286620685 (592h0m14.05726202s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287116924 (592h0m14.057758259s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287176458 (592h0m14.057817793s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287488926 (592h0m14.058130261s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287772308 (592h0m14.058413643s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287863583 (592h0m14.058504918s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288079506 (592h0m14.058720841s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288237839 (592h0m14.058879184s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288418594 (592h0m14.059059929s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288629087 (592h0m14.059270422s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289488541 (592h0m14.060129877s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289738971 (592h0m14.060380306s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289989741 (592h0m14.060631086s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.290216545 (592h0m14.06085788s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.297416701 (592h0m14.068058036s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:41.030217031 (592h0m15.800858356s) [upload=473459]: Next returned nil segment Jul 18 01:10:41.030234745 (592h0m15.80087608s) [upload=473459]: Waiting for error group managing segments... Jul 18 01:10:41.308832612 (592h0m16.079473947s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.358887261 (592h0m16.129528596s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.365986644 (592h0m16.136627979s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.375224368 (592h0m16.145865704s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.419739118 (592h0m16.190380453s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.430360108 (592h0m16.201001444s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.450861434 (592h0m16.221502769s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468109615 (592h0m16.23875095s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468727326 (592h0m16.239368661s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.516013518 (592h0m16.286654863s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.522587155 (592h0m16.29322849s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.553218502 (592h0m16.323859837s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:11:08.331622589 (592h0m43.102263924s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38902->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331668216 (592h0m43.102309551s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:08.331710026 (592h0m43.102351361s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:60806->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331772906 (592h0m43.102414241s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331804857 (592h0m43.102446182s) [seg_pos={0 0}, upload=473459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 01:11:08.598012820 (592h0m43.368654156s) [upload=473459]: Done uploading segments... Jul 18 01:11:08.598041896 (592h0m43.368683231s) [upload=473459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.82304888s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [............................................XX..............] duration: 5.93849947s FAIL (2025-07-18 01:21:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ress=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683391198 (592h0m30.454032533s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683761526 (592h0m30.454402861s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683801033 (592h0m30.454442368s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684063464 (592h0m30.454704799s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684360503 (592h0m30.455001838s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684654846 (592h0m30.455296181s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685431742 (592h0m30.456073077s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685861425 (592h0m30.45650276s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686139176 (592h0m30.456780511s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686424722 (592h0m30.457066057s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686687836 (592h0m30.457329171s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686944527 (592h0m30.457585862s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.687208402 (592h0m30.457849737s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688069810 (592h0m30.458711145s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688391375 (592h0m30.45903271s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688755091 (592h0m30.459396426s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.689020598 (592h0m30.459661934s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:56.791786484 (592h0m31.562427839s) [upload=473462]: Next returned nil segment Jul 18 01:10:56.791810180 (592h0m31.562451535s) [upload=473462]: Waiting for error group managing segments... Jul 18 01:10:57.072508846 (592h0m31.843150181s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.084313581 (592h0m31.854954917s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.130829109 (592h0m31.901470444s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.380019692 (592h0m32.150661027s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.586116023 (592h0m32.356757358s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.590030633 (592h0m32.360671968s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.644593087 (592h0m32.415234412s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.711170438 (592h0m32.481811773s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.815350524 (592h0m32.585991869s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:11:54.506942141 (592h1m29.277583476s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: rpc: dial tcp 77.90.31.13:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.506990634 (592h1m29.277631959s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507021904 (592h1m29.277663239s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: rpc: dial tcp 77.90.31.19:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507055037 (592h1m29.277696372s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: rpc: dial tcp 77.90.31.18:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507085656 (592h1m29.277726992s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 188.166.225.170:42120->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507153856 (592h1m29.277795192s) [noise=true , address=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj, address:77.90.31.15:4704): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507228530 (592h1m29.277869865s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507325304 (592h1m29.277966639s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12eUC555Lzo37B5FafAFMfXjSkfm1zv1pRZmH71jf4xpdCWH6kG, address:77.90.31.21:4904): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507433642 (592h1m29.278074977s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507448881 (592h1m29.278090216s) [seg_pos={0 0}, upload=473462]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 18 01:11:54.756684719 (592h1m29.527326054s) [upload=473462]: Done uploading segments... Jul 18 01:11:54.756710668 (592h1m29.527352004s) [upload=473462]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 5.938537413s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X.X..............] duration: 6.059251173s FAIL (2025-07-18 01:20:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== T467KJ2SHEV5EJW2LJCYJSSHAG2CJIU2A, seg_pos={0 0}, upload=473444]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: write tcp 188.166.225.170:43510->77.90.31.16:4733: use of closed network connection; context canceled Jul 18 01:09:36.439338648 (591h59m11.209979983s) [seg_pos={0 0}, upload=473444]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 01:09:36.439407128 (591h59m11.210048464s) [upload=473444]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 01:09:36.439411076 (591h59m11.210052411s) [upload=473444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 01:09:36.756883131 (591h59m11.527524466s) [upload=473444]: Done uploading segments... Jul 18 01:09:36.756897708 (591h59m11.527539044s) [upload=473444]: Upload finished. err: <nil> Jul 18 01:10:39.025598816 (592h0m13.796240151s) [upload=473459]: Starting upload Jul 18 01:10:39.025658440 (592h0m13.796299765s) [upload=473459]: Done starting upload Jul 18 01:10:39.025759353 (592h0m13.796400688s) [upload=473459]: Uploading segments... Jul 18 01:10:39.025907185 (592h0m13.79654852s) [upload=473459]: Got next segment. Inline: false Jul 18 01:10:39.025950799 (592h0m13.796592124s) [upload=473459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 01:10:39.025957632 (592h0m13.796598967s) [upload=473459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 01:10:39.286495526 (592h0m14.057136861s) [seg_pos={0 0}, upload=473459]: Begin upload segment... Jul 18 01:10:39.286547334 (592h0m14.057188699s) [seg_pos={0 0}, upload=473459]: Done begin upload segment. Jul 18 01:10:39.286596759 (592h0m14.057238094s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.286620685 (592h0m14.05726202s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287116924 (592h0m14.057758259s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287176458 (592h0m14.057817793s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287488926 (592h0m14.058130261s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287772308 (592h0m14.058413643s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287863583 (592h0m14.058504918s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288079506 (592h0m14.058720841s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288237839 (592h0m14.058879184s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288418594 (592h0m14.059059929s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288629087 (592h0m14.059270422s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289488541 (592h0m14.060129877s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289738971 (592h0m14.060380306s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289989741 (592h0m14.060631086s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.290216545 (592h0m14.06085788s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.297416701 (592h0m14.068058036s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:41.030217031 (592h0m15.800858356s) [upload=473459]: Next returned nil segment Jul 18 01:10:41.030234745 (592h0m15.80087608s) [upload=473459]: Waiting for error group managing segments... Jul 18 01:10:41.308832612 (592h0m16.079473947s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.358887261 (592h0m16.129528596s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.365986644 (592h0m16.136627979s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.375224368 (592h0m16.145865704s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.419739118 (592h0m16.190380453s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.430360108 (592h0m16.201001444s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.450861434 (592h0m16.221502769s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468109615 (592h0m16.23875095s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468727326 (592h0m16.239368661s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.516013518 (592h0m16.286654863s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.522587155 (592h0m16.29322849s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.553218502 (592h0m16.323859837s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:11:08.331622589 (592h0m43.102263924s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38902->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331668216 (592h0m43.102309551s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:08.331710026 (592h0m43.102351361s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:60806->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331772906 (592h0m43.102414241s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331804857 (592h0m43.102446182s) [seg_pos={0 0}, upload=473459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 01:11:08.598012820 (592h0m43.368654156s) [upload=473459]: Done uploading segments... Jul 18 01:11:08.598041896 (592h0m43.368683231s) [upload=473459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.059297291s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.............................................XX.............] duration: 5.826302997s FAIL (2025-07-18 01:20:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ress=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683391198 (592h0m30.454032533s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683761526 (592h0m30.454402861s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683801033 (592h0m30.454442368s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684063464 (592h0m30.454704799s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684360503 (592h0m30.455001838s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684654846 (592h0m30.455296181s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685431742 (592h0m30.456073077s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685861425 (592h0m30.45650276s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686139176 (592h0m30.456780511s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686424722 (592h0m30.457066057s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686687836 (592h0m30.457329171s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686944527 (592h0m30.457585862s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.687208402 (592h0m30.457849737s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688069810 (592h0m30.458711145s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688391375 (592h0m30.45903271s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688755091 (592h0m30.459396426s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.689020598 (592h0m30.459661934s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:56.791786484 (592h0m31.562427839s) [upload=473462]: Next returned nil segment Jul 18 01:10:56.791810180 (592h0m31.562451535s) [upload=473462]: Waiting for error group managing segments... Jul 18 01:10:57.072508846 (592h0m31.843150181s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.084313581 (592h0m31.854954917s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.130829109 (592h0m31.901470444s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.380019692 (592h0m32.150661027s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.586116023 (592h0m32.356757358s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.590030633 (592h0m32.360671968s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.644593087 (592h0m32.415234412s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.711170438 (592h0m32.481811773s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.815350524 (592h0m32.585991869s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:11:54.506942141 (592h1m29.277583476s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: rpc: dial tcp 77.90.31.13:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.506990634 (592h1m29.277631959s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507021904 (592h1m29.277663239s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: rpc: dial tcp 77.90.31.19:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507055037 (592h1m29.277696372s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: rpc: dial tcp 77.90.31.18:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507085656 (592h1m29.277726992s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 188.166.225.170:42120->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507153856 (592h1m29.277795192s) [noise=true , address=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj, address:77.90.31.15:4704): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507228530 (592h1m29.277869865s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507325304 (592h1m29.277966639s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12eUC555Lzo37B5FafAFMfXjSkfm1zv1pRZmH71jf4xpdCWH6kG, address:77.90.31.21:4904): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507433642 (592h1m29.278074977s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507448881 (592h1m29.278090216s) [seg_pos={0 0}, upload=473462]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 18 01:11:54.756684719 (592h1m29.527326054s) [upload=473462]: Done uploading segments... Jul 18 01:11:54.756710668 (592h1m29.527352004s) [upload=473462]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 5.826329306s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................X.X.............] duration: 6.054996163s FAIL (2025-07-18 01:19:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== T467KJ2SHEV5EJW2LJCYJSSHAG2CJIU2A, seg_pos={0 0}, upload=473444]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: write tcp 188.166.225.170:43510->77.90.31.16:4733: use of closed network connection; context canceled Jul 18 01:09:36.439338648 (591h59m11.209979983s) [seg_pos={0 0}, upload=473444]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 01:09:36.439407128 (591h59m11.210048464s) [upload=473444]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 01:09:36.439411076 (591h59m11.210052411s) [upload=473444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 01:09:36.756883131 (591h59m11.527524466s) [upload=473444]: Done uploading segments... Jul 18 01:09:36.756897708 (591h59m11.527539044s) [upload=473444]: Upload finished. err: <nil> Jul 18 01:10:39.025598816 (592h0m13.796240151s) [upload=473459]: Starting upload Jul 18 01:10:39.025658440 (592h0m13.796299765s) [upload=473459]: Done starting upload Jul 18 01:10:39.025759353 (592h0m13.796400688s) [upload=473459]: Uploading segments... Jul 18 01:10:39.025907185 (592h0m13.79654852s) [upload=473459]: Got next segment. Inline: false Jul 18 01:10:39.025950799 (592h0m13.796592124s) [upload=473459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 01:10:39.025957632 (592h0m13.796598967s) [upload=473459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 01:10:39.286495526 (592h0m14.057136861s) [seg_pos={0 0}, upload=473459]: Begin upload segment... Jul 18 01:10:39.286547334 (592h0m14.057188699s) [seg_pos={0 0}, upload=473459]: Done begin upload segment. Jul 18 01:10:39.286596759 (592h0m14.057238094s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.286620685 (592h0m14.05726202s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287116924 (592h0m14.057758259s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287176458 (592h0m14.057817793s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287488926 (592h0m14.058130261s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287772308 (592h0m14.058413643s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287863583 (592h0m14.058504918s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288079506 (592h0m14.058720841s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288237839 (592h0m14.058879184s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288418594 (592h0m14.059059929s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288629087 (592h0m14.059270422s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289488541 (592h0m14.060129877s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289738971 (592h0m14.060380306s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289989741 (592h0m14.060631086s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.290216545 (592h0m14.06085788s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.297416701 (592h0m14.068058036s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:41.030217031 (592h0m15.800858356s) [upload=473459]: Next returned nil segment Jul 18 01:10:41.030234745 (592h0m15.80087608s) [upload=473459]: Waiting for error group managing segments... Jul 18 01:10:41.308832612 (592h0m16.079473947s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.358887261 (592h0m16.129528596s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.365986644 (592h0m16.136627979s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.375224368 (592h0m16.145865704s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.419739118 (592h0m16.190380453s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.430360108 (592h0m16.201001444s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.450861434 (592h0m16.221502769s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468109615 (592h0m16.23875095s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468727326 (592h0m16.239368661s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.516013518 (592h0m16.286654863s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.522587155 (592h0m16.29322849s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.553218502 (592h0m16.323859837s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:11:08.331622589 (592h0m43.102263924s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38902->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331668216 (592h0m43.102309551s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:08.331710026 (592h0m43.102351361s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:60806->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331772906 (592h0m43.102414241s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331804857 (592h0m43.102446182s) [seg_pos={0 0}, upload=473459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 01:11:08.598012820 (592h0m43.368654156s) [upload=473459]: Done uploading segments... Jul 18 01:11:08.598041896 (592h0m43.368683231s) [upload=473459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.055023456s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X.X............] duration: 5.966305752s FAIL (2025-07-18 01:19:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== T467KJ2SHEV5EJW2LJCYJSSHAG2CJIU2A, seg_pos={0 0}, upload=473444]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: write tcp 188.166.225.170:43510->77.90.31.16:4733: use of closed network connection; context canceled Jul 18 01:09:36.439338648 (591h59m11.209979983s) [seg_pos={0 0}, upload=473444]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 01:09:36.439407128 (591h59m11.210048464s) [upload=473444]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 01:09:36.439411076 (591h59m11.210052411s) [upload=473444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 01:09:36.756883131 (591h59m11.527524466s) [upload=473444]: Done uploading segments... Jul 18 01:09:36.756897708 (591h59m11.527539044s) [upload=473444]: Upload finished. err: <nil> Jul 18 01:10:39.025598816 (592h0m13.796240151s) [upload=473459]: Starting upload Jul 18 01:10:39.025658440 (592h0m13.796299765s) [upload=473459]: Done starting upload Jul 18 01:10:39.025759353 (592h0m13.796400688s) [upload=473459]: Uploading segments... Jul 18 01:10:39.025907185 (592h0m13.79654852s) [upload=473459]: Got next segment. Inline: false Jul 18 01:10:39.025950799 (592h0m13.796592124s) [upload=473459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 01:10:39.025957632 (592h0m13.796598967s) [upload=473459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 01:10:39.286495526 (592h0m14.057136861s) [seg_pos={0 0}, upload=473459]: Begin upload segment... Jul 18 01:10:39.286547334 (592h0m14.057188699s) [seg_pos={0 0}, upload=473459]: Done begin upload segment. Jul 18 01:10:39.286596759 (592h0m14.057238094s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.286620685 (592h0m14.05726202s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287116924 (592h0m14.057758259s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287176458 (592h0m14.057817793s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287488926 (592h0m14.058130261s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287772308 (592h0m14.058413643s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287863583 (592h0m14.058504918s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288079506 (592h0m14.058720841s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288237839 (592h0m14.058879184s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288418594 (592h0m14.059059929s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288629087 (592h0m14.059270422s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289488541 (592h0m14.060129877s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289738971 (592h0m14.060380306s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289989741 (592h0m14.060631086s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.290216545 (592h0m14.06085788s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.297416701 (592h0m14.068058036s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:41.030217031 (592h0m15.800858356s) [upload=473459]: Next returned nil segment Jul 18 01:10:41.030234745 (592h0m15.80087608s) [upload=473459]: Waiting for error group managing segments... Jul 18 01:10:41.308832612 (592h0m16.079473947s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.358887261 (592h0m16.129528596s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.365986644 (592h0m16.136627979s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.375224368 (592h0m16.145865704s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.419739118 (592h0m16.190380453s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.430360108 (592h0m16.201001444s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.450861434 (592h0m16.221502769s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468109615 (592h0m16.23875095s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468727326 (592h0m16.239368661s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.516013518 (592h0m16.286654863s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.522587155 (592h0m16.29322849s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.553218502 (592h0m16.323859837s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:11:08.331622589 (592h0m43.102263924s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38902->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331668216 (592h0m43.102309551s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:08.331710026 (592h0m43.102351361s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:60806->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331772906 (592h0m43.102414241s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331804857 (592h0m43.102446182s) [seg_pos={0 0}, upload=473459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 01:11:08.598012820 (592h0m43.368654156s) [upload=473459]: Done uploading segments... Jul 18 01:11:08.598041896 (592h0m43.368683231s) [upload=473459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.966329778s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..............................................XX............] duration: 5.623055254s FAIL (2025-07-18 01:19:38 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ress=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683391198 (592h0m30.454032533s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683761526 (592h0m30.454402861s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683801033 (592h0m30.454442368s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684063464 (592h0m30.454704799s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684360503 (592h0m30.455001838s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684654846 (592h0m30.455296181s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685431742 (592h0m30.456073077s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685861425 (592h0m30.45650276s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686139176 (592h0m30.456780511s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686424722 (592h0m30.457066057s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686687836 (592h0m30.457329171s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686944527 (592h0m30.457585862s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.687208402 (592h0m30.457849737s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688069810 (592h0m30.458711145s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688391375 (592h0m30.45903271s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688755091 (592h0m30.459396426s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.689020598 (592h0m30.459661934s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:56.791786484 (592h0m31.562427839s) [upload=473462]: Next returned nil segment Jul 18 01:10:56.791810180 (592h0m31.562451535s) [upload=473462]: Waiting for error group managing segments... Jul 18 01:10:57.072508846 (592h0m31.843150181s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.084313581 (592h0m31.854954917s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.130829109 (592h0m31.901470444s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.380019692 (592h0m32.150661027s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.586116023 (592h0m32.356757358s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.590030633 (592h0m32.360671968s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.644593087 (592h0m32.415234412s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.711170438 (592h0m32.481811773s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.815350524 (592h0m32.585991869s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:11:54.506942141 (592h1m29.277583476s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: rpc: dial tcp 77.90.31.13:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.506990634 (592h1m29.277631959s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507021904 (592h1m29.277663239s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: rpc: dial tcp 77.90.31.19:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507055037 (592h1m29.277696372s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: rpc: dial tcp 77.90.31.18:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507085656 (592h1m29.277726992s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 188.166.225.170:42120->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507153856 (592h1m29.277795192s) [noise=true , address=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj, address:77.90.31.15:4704): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507228530 (592h1m29.277869865s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507325304 (592h1m29.277966639s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12eUC555Lzo37B5FafAFMfXjSkfm1zv1pRZmH71jf4xpdCWH6kG, address:77.90.31.21:4904): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507433642 (592h1m29.278074977s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507448881 (592h1m29.278090216s) [seg_pos={0 0}, upload=473462]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 18 01:11:54.756684719 (592h1m29.527326054s) [upload=473462]: Done uploading segments... Jul 18 01:11:54.756710668 (592h1m29.527352004s) [upload=473462]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 5.623081744s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X.X...........] duration: 6.07860557s FAIL (2025-07-18 01:19:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== T467KJ2SHEV5EJW2LJCYJSSHAG2CJIU2A, seg_pos={0 0}, upload=473444]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: write tcp 188.166.225.170:43510->77.90.31.16:4733: use of closed network connection; context canceled Jul 18 01:09:36.439338648 (591h59m11.209979983s) [seg_pos={0 0}, upload=473444]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 01:09:36.439407128 (591h59m11.210048464s) [upload=473444]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 01:09:36.439411076 (591h59m11.210052411s) [upload=473444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 01:09:36.756883131 (591h59m11.527524466s) [upload=473444]: Done uploading segments... Jul 18 01:09:36.756897708 (591h59m11.527539044s) [upload=473444]: Upload finished. err: <nil> Jul 18 01:10:39.025598816 (592h0m13.796240151s) [upload=473459]: Starting upload Jul 18 01:10:39.025658440 (592h0m13.796299765s) [upload=473459]: Done starting upload Jul 18 01:10:39.025759353 (592h0m13.796400688s) [upload=473459]: Uploading segments... Jul 18 01:10:39.025907185 (592h0m13.79654852s) [upload=473459]: Got next segment. Inline: false Jul 18 01:10:39.025950799 (592h0m13.796592124s) [upload=473459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 01:10:39.025957632 (592h0m13.796598967s) [upload=473459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 01:10:39.286495526 (592h0m14.057136861s) [seg_pos={0 0}, upload=473459]: Begin upload segment... Jul 18 01:10:39.286547334 (592h0m14.057188699s) [seg_pos={0 0}, upload=473459]: Done begin upload segment. Jul 18 01:10:39.286596759 (592h0m14.057238094s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.286620685 (592h0m14.05726202s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287116924 (592h0m14.057758259s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287176458 (592h0m14.057817793s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287488926 (592h0m14.058130261s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287772308 (592h0m14.058413643s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287863583 (592h0m14.058504918s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288079506 (592h0m14.058720841s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288237839 (592h0m14.058879184s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288418594 (592h0m14.059059929s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288629087 (592h0m14.059270422s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289488541 (592h0m14.060129877s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289738971 (592h0m14.060380306s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289989741 (592h0m14.060631086s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.290216545 (592h0m14.06085788s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.297416701 (592h0m14.068058036s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:41.030217031 (592h0m15.800858356s) [upload=473459]: Next returned nil segment Jul 18 01:10:41.030234745 (592h0m15.80087608s) [upload=473459]: Waiting for error group managing segments... Jul 18 01:10:41.308832612 (592h0m16.079473947s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.358887261 (592h0m16.129528596s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.365986644 (592h0m16.136627979s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.375224368 (592h0m16.145865704s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.419739118 (592h0m16.190380453s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.430360108 (592h0m16.201001444s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.450861434 (592h0m16.221502769s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468109615 (592h0m16.23875095s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468727326 (592h0m16.239368661s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.516013518 (592h0m16.286654863s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.522587155 (592h0m16.29322849s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.553218502 (592h0m16.323859837s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:11:08.331622589 (592h0m43.102263924s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38902->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331668216 (592h0m43.102309551s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:08.331710026 (592h0m43.102351361s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:60806->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331772906 (592h0m43.102414241s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331804857 (592h0m43.102446182s) [seg_pos={0 0}, upload=473459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 01:11:08.598012820 (592h0m43.368654156s) [upload=473459]: Done uploading segments... Jul 18 01:11:08.598041896 (592h0m43.368683231s) [upload=473459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.078632141s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............................................XX...........] duration: 5.80165123s FAIL (2025-07-18 01:18:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ress=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683391198 (592h0m30.454032533s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683761526 (592h0m30.454402861s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683801033 (592h0m30.454442368s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684063464 (592h0m30.454704799s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684360503 (592h0m30.455001838s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684654846 (592h0m30.455296181s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685431742 (592h0m30.456073077s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685861425 (592h0m30.45650276s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686139176 (592h0m30.456780511s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686424722 (592h0m30.457066057s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686687836 (592h0m30.457329171s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686944527 (592h0m30.457585862s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.687208402 (592h0m30.457849737s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688069810 (592h0m30.458711145s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688391375 (592h0m30.45903271s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688755091 (592h0m30.459396426s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.689020598 (592h0m30.459661934s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:56.791786484 (592h0m31.562427839s) [upload=473462]: Next returned nil segment Jul 18 01:10:56.791810180 (592h0m31.562451535s) [upload=473462]: Waiting for error group managing segments... Jul 18 01:10:57.072508846 (592h0m31.843150181s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.084313581 (592h0m31.854954917s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.130829109 (592h0m31.901470444s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.380019692 (592h0m32.150661027s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.586116023 (592h0m32.356757358s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.590030633 (592h0m32.360671968s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.644593087 (592h0m32.415234412s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.711170438 (592h0m32.481811773s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.815350524 (592h0m32.585991869s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:11:54.506942141 (592h1m29.277583476s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: rpc: dial tcp 77.90.31.13:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.506990634 (592h1m29.277631959s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507021904 (592h1m29.277663239s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: rpc: dial tcp 77.90.31.19:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507055037 (592h1m29.277696372s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: rpc: dial tcp 77.90.31.18:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507085656 (592h1m29.277726992s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 188.166.225.170:42120->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507153856 (592h1m29.277795192s) [noise=true , address=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj, address:77.90.31.15:4704): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507228530 (592h1m29.277869865s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507325304 (592h1m29.277966639s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12eUC555Lzo37B5FafAFMfXjSkfm1zv1pRZmH71jf4xpdCWH6kG, address:77.90.31.21:4904): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507433642 (592h1m29.278074977s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507448881 (592h1m29.278090216s) [seg_pos={0 0}, upload=473462]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 18 01:11:54.756684719 (592h1m29.527326054s) [upload=473462]: Done uploading segments... Jul 18 01:11:54.756710668 (592h1m29.527352004s) [upload=473462]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 5.801682289s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X.X..........] duration: 5.936767377s FAIL (2025-07-18 01:18:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== T467KJ2SHEV5EJW2LJCYJSSHAG2CJIU2A, seg_pos={0 0}, upload=473444]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: write tcp 188.166.225.170:43510->77.90.31.16:4733: use of closed network connection; context canceled Jul 18 01:09:36.439338648 (591h59m11.209979983s) [seg_pos={0 0}, upload=473444]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 01:09:36.439407128 (591h59m11.210048464s) [upload=473444]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 01:09:36.439411076 (591h59m11.210052411s) [upload=473444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 01:09:36.756883131 (591h59m11.527524466s) [upload=473444]: Done uploading segments... Jul 18 01:09:36.756897708 (591h59m11.527539044s) [upload=473444]: Upload finished. err: <nil> Jul 18 01:10:39.025598816 (592h0m13.796240151s) [upload=473459]: Starting upload Jul 18 01:10:39.025658440 (592h0m13.796299765s) [upload=473459]: Done starting upload Jul 18 01:10:39.025759353 (592h0m13.796400688s) [upload=473459]: Uploading segments... Jul 18 01:10:39.025907185 (592h0m13.79654852s) [upload=473459]: Got next segment. Inline: false Jul 18 01:10:39.025950799 (592h0m13.796592124s) [upload=473459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 01:10:39.025957632 (592h0m13.796598967s) [upload=473459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 01:10:39.286495526 (592h0m14.057136861s) [seg_pos={0 0}, upload=473459]: Begin upload segment... Jul 18 01:10:39.286547334 (592h0m14.057188699s) [seg_pos={0 0}, upload=473459]: Done begin upload segment. Jul 18 01:10:39.286596759 (592h0m14.057238094s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.286620685 (592h0m14.05726202s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287116924 (592h0m14.057758259s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287176458 (592h0m14.057817793s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287488926 (592h0m14.058130261s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287772308 (592h0m14.058413643s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287863583 (592h0m14.058504918s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288079506 (592h0m14.058720841s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288237839 (592h0m14.058879184s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288418594 (592h0m14.059059929s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288629087 (592h0m14.059270422s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289488541 (592h0m14.060129877s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289738971 (592h0m14.060380306s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289989741 (592h0m14.060631086s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.290216545 (592h0m14.06085788s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.297416701 (592h0m14.068058036s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:41.030217031 (592h0m15.800858356s) [upload=473459]: Next returned nil segment Jul 18 01:10:41.030234745 (592h0m15.80087608s) [upload=473459]: Waiting for error group managing segments... Jul 18 01:10:41.308832612 (592h0m16.079473947s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.358887261 (592h0m16.129528596s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.365986644 (592h0m16.136627979s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.375224368 (592h0m16.145865704s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.419739118 (592h0m16.190380453s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.430360108 (592h0m16.201001444s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.450861434 (592h0m16.221502769s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468109615 (592h0m16.23875095s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468727326 (592h0m16.239368661s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.516013518 (592h0m16.286654863s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.522587155 (592h0m16.29322849s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.553218502 (592h0m16.323859837s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:11:08.331622589 (592h0m43.102263924s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38902->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331668216 (592h0m43.102309551s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:08.331710026 (592h0m43.102351361s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:60806->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331772906 (592h0m43.102414241s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331804857 (592h0m43.102446182s) [seg_pos={0 0}, upload=473459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 01:11:08.598012820 (592h0m43.368654156s) [upload=473459]: Done uploading segments... Jul 18 01:11:08.598041896 (592h0m43.368683231s) [upload=473459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.936797413s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................................................XX..........] duration: 5.734681125s FAIL (2025-07-18 01:18:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ress=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683391198 (592h0m30.454032533s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683761526 (592h0m30.454402861s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683801033 (592h0m30.454442368s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684063464 (592h0m30.454704799s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684360503 (592h0m30.455001838s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684654846 (592h0m30.455296181s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685431742 (592h0m30.456073077s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685861425 (592h0m30.45650276s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686139176 (592h0m30.456780511s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686424722 (592h0m30.457066057s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686687836 (592h0m30.457329171s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686944527 (592h0m30.457585862s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.687208402 (592h0m30.457849737s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688069810 (592h0m30.458711145s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688391375 (592h0m30.45903271s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688755091 (592h0m30.459396426s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.689020598 (592h0m30.459661934s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:56.791786484 (592h0m31.562427839s) [upload=473462]: Next returned nil segment Jul 18 01:10:56.791810180 (592h0m31.562451535s) [upload=473462]: Waiting for error group managing segments... Jul 18 01:10:57.072508846 (592h0m31.843150181s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.084313581 (592h0m31.854954917s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.130829109 (592h0m31.901470444s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.380019692 (592h0m32.150661027s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.586116023 (592h0m32.356757358s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.590030633 (592h0m32.360671968s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.644593087 (592h0m32.415234412s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.711170438 (592h0m32.481811773s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.815350524 (592h0m32.585991869s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:11:54.506942141 (592h1m29.277583476s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: rpc: dial tcp 77.90.31.13:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.506990634 (592h1m29.277631959s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507021904 (592h1m29.277663239s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: rpc: dial tcp 77.90.31.19:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507055037 (592h1m29.277696372s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: rpc: dial tcp 77.90.31.18:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507085656 (592h1m29.277726992s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 188.166.225.170:42120->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507153856 (592h1m29.277795192s) [noise=true , address=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj, address:77.90.31.15:4704): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507228530 (592h1m29.277869865s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507325304 (592h1m29.277966639s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12eUC555Lzo37B5FafAFMfXjSkfm1zv1pRZmH71jf4xpdCWH6kG, address:77.90.31.21:4904): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507433642 (592h1m29.278074977s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507448881 (592h1m29.278090216s) [seg_pos={0 0}, upload=473462]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 18 01:11:54.756684719 (592h1m29.527326054s) [upload=473462]: Done uploading segments... Jul 18 01:11:54.756710668 (592h1m29.527352004s) [upload=473462]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 5.734710522s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X.X.........] duration: 5.91483785s FAIL (2025-07-18 01:18:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== T467KJ2SHEV5EJW2LJCYJSSHAG2CJIU2A, seg_pos={0 0}, upload=473444]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: write tcp 188.166.225.170:43510->77.90.31.16:4733: use of closed network connection; context canceled Jul 18 01:09:36.439338648 (591h59m11.209979983s) [seg_pos={0 0}, upload=473444]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 01:09:36.439407128 (591h59m11.210048464s) [upload=473444]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 01:09:36.439411076 (591h59m11.210052411s) [upload=473444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 01:09:36.756883131 (591h59m11.527524466s) [upload=473444]: Done uploading segments... Jul 18 01:09:36.756897708 (591h59m11.527539044s) [upload=473444]: Upload finished. err: <nil> Jul 18 01:10:39.025598816 (592h0m13.796240151s) [upload=473459]: Starting upload Jul 18 01:10:39.025658440 (592h0m13.796299765s) [upload=473459]: Done starting upload Jul 18 01:10:39.025759353 (592h0m13.796400688s) [upload=473459]: Uploading segments... Jul 18 01:10:39.025907185 (592h0m13.79654852s) [upload=473459]: Got next segment. Inline: false Jul 18 01:10:39.025950799 (592h0m13.796592124s) [upload=473459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 01:10:39.025957632 (592h0m13.796598967s) [upload=473459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 01:10:39.286495526 (592h0m14.057136861s) [seg_pos={0 0}, upload=473459]: Begin upload segment... Jul 18 01:10:39.286547334 (592h0m14.057188699s) [seg_pos={0 0}, upload=473459]: Done begin upload segment. Jul 18 01:10:39.286596759 (592h0m14.057238094s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.286620685 (592h0m14.05726202s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287116924 (592h0m14.057758259s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287176458 (592h0m14.057817793s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287488926 (592h0m14.058130261s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287772308 (592h0m14.058413643s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287863583 (592h0m14.058504918s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288079506 (592h0m14.058720841s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288237839 (592h0m14.058879184s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288418594 (592h0m14.059059929s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288629087 (592h0m14.059270422s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289488541 (592h0m14.060129877s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289738971 (592h0m14.060380306s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289989741 (592h0m14.060631086s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.290216545 (592h0m14.06085788s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.297416701 (592h0m14.068058036s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:41.030217031 (592h0m15.800858356s) [upload=473459]: Next returned nil segment Jul 18 01:10:41.030234745 (592h0m15.80087608s) [upload=473459]: Waiting for error group managing segments... Jul 18 01:10:41.308832612 (592h0m16.079473947s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.358887261 (592h0m16.129528596s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.365986644 (592h0m16.136627979s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.375224368 (592h0m16.145865704s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.419739118 (592h0m16.190380453s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.430360108 (592h0m16.201001444s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.450861434 (592h0m16.221502769s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468109615 (592h0m16.23875095s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468727326 (592h0m16.239368661s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.516013518 (592h0m16.286654863s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.522587155 (592h0m16.29322849s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.553218502 (592h0m16.323859837s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:11:08.331622589 (592h0m43.102263924s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38902->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331668216 (592h0m43.102309551s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:08.331710026 (592h0m43.102351361s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:60806->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331772906 (592h0m43.102414241s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331804857 (592h0m43.102446182s) [seg_pos={0 0}, upload=473459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 01:11:08.598012820 (592h0m43.368654156s) [upload=473459]: Done uploading segments... Jul 18 01:11:08.598041896 (592h0m43.368683231s) [upload=473459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.914866574s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................................................XX.........] duration: 5.801835792s FAIL (2025-07-18 01:18:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ress=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683391198 (592h0m30.454032533s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683761526 (592h0m30.454402861s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683801033 (592h0m30.454442368s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684063464 (592h0m30.454704799s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684360503 (592h0m30.455001838s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684654846 (592h0m30.455296181s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685431742 (592h0m30.456073077s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685861425 (592h0m30.45650276s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686139176 (592h0m30.456780511s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686424722 (592h0m30.457066057s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686687836 (592h0m30.457329171s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686944527 (592h0m30.457585862s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.687208402 (592h0m30.457849737s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688069810 (592h0m30.458711145s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688391375 (592h0m30.45903271s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688755091 (592h0m30.459396426s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.689020598 (592h0m30.459661934s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:56.791786484 (592h0m31.562427839s) [upload=473462]: Next returned nil segment Jul 18 01:10:56.791810180 (592h0m31.562451535s) [upload=473462]: Waiting for error group managing segments... Jul 18 01:10:57.072508846 (592h0m31.843150181s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.084313581 (592h0m31.854954917s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.130829109 (592h0m31.901470444s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.380019692 (592h0m32.150661027s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.586116023 (592h0m32.356757358s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.590030633 (592h0m32.360671968s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.644593087 (592h0m32.415234412s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.711170438 (592h0m32.481811773s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.815350524 (592h0m32.585991869s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:11:54.506942141 (592h1m29.277583476s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: rpc: dial tcp 77.90.31.13:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.506990634 (592h1m29.277631959s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507021904 (592h1m29.277663239s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: rpc: dial tcp 77.90.31.19:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507055037 (592h1m29.277696372s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: rpc: dial tcp 77.90.31.18:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507085656 (592h1m29.277726992s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 188.166.225.170:42120->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507153856 (592h1m29.277795192s) [noise=true , address=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj, address:77.90.31.15:4704): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507228530 (592h1m29.277869865s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507325304 (592h1m29.277966639s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12eUC555Lzo37B5FafAFMfXjSkfm1zv1pRZmH71jf4xpdCWH6kG, address:77.90.31.21:4904): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507433642 (592h1m29.278074977s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507448881 (592h1m29.278090216s) [seg_pos={0 0}, upload=473462]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 18 01:11:54.756684719 (592h1m29.527326054s) [upload=473462]: Done uploading segments... Jul 18 01:11:54.756710668 (592h1m29.527352004s) [upload=473462]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 5.801862222s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X.X........] duration: 5.897336662s FAIL (2025-07-18 01:17:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== T467KJ2SHEV5EJW2LJCYJSSHAG2CJIU2A, seg_pos={0 0}, upload=473444]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: write tcp 188.166.225.170:43510->77.90.31.16:4733: use of closed network connection; context canceled Jul 18 01:09:36.439338648 (591h59m11.209979983s) [seg_pos={0 0}, upload=473444]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 01:09:36.439407128 (591h59m11.210048464s) [upload=473444]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 01:09:36.439411076 (591h59m11.210052411s) [upload=473444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 01:09:36.756883131 (591h59m11.527524466s) [upload=473444]: Done uploading segments... Jul 18 01:09:36.756897708 (591h59m11.527539044s) [upload=473444]: Upload finished. err: <nil> Jul 18 01:10:39.025598816 (592h0m13.796240151s) [upload=473459]: Starting upload Jul 18 01:10:39.025658440 (592h0m13.796299765s) [upload=473459]: Done starting upload Jul 18 01:10:39.025759353 (592h0m13.796400688s) [upload=473459]: Uploading segments... Jul 18 01:10:39.025907185 (592h0m13.79654852s) [upload=473459]: Got next segment. Inline: false Jul 18 01:10:39.025950799 (592h0m13.796592124s) [upload=473459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 01:10:39.025957632 (592h0m13.796598967s) [upload=473459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 01:10:39.286495526 (592h0m14.057136861s) [seg_pos={0 0}, upload=473459]: Begin upload segment... Jul 18 01:10:39.286547334 (592h0m14.057188699s) [seg_pos={0 0}, upload=473459]: Done begin upload segment. Jul 18 01:10:39.286596759 (592h0m14.057238094s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.286620685 (592h0m14.05726202s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287116924 (592h0m14.057758259s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287176458 (592h0m14.057817793s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287488926 (592h0m14.058130261s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287772308 (592h0m14.058413643s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287863583 (592h0m14.058504918s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288079506 (592h0m14.058720841s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288237839 (592h0m14.058879184s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288418594 (592h0m14.059059929s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288629087 (592h0m14.059270422s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289488541 (592h0m14.060129877s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289738971 (592h0m14.060380306s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289989741 (592h0m14.060631086s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.290216545 (592h0m14.06085788s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.297416701 (592h0m14.068058036s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:41.030217031 (592h0m15.800858356s) [upload=473459]: Next returned nil segment Jul 18 01:10:41.030234745 (592h0m15.80087608s) [upload=473459]: Waiting for error group managing segments... Jul 18 01:10:41.308832612 (592h0m16.079473947s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.358887261 (592h0m16.129528596s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.365986644 (592h0m16.136627979s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.375224368 (592h0m16.145865704s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.419739118 (592h0m16.190380453s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.430360108 (592h0m16.201001444s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.450861434 (592h0m16.221502769s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468109615 (592h0m16.23875095s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468727326 (592h0m16.239368661s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.516013518 (592h0m16.286654863s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.522587155 (592h0m16.29322849s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.553218502 (592h0m16.323859837s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:11:08.331622589 (592h0m43.102263924s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38902->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331668216 (592h0m43.102309551s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:08.331710026 (592h0m43.102351361s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:60806->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331772906 (592h0m43.102414241s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331804857 (592h0m43.102446182s) [seg_pos={0 0}, upload=473459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 01:11:08.598012820 (592h0m43.368654156s) [upload=473459]: Done uploading segments... Jul 18 01:11:08.598041896 (592h0m43.368683231s) [upload=473459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.897364636s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X.X.......] duration: 5.907982093s FAIL (2025-07-18 01:17:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== T467KJ2SHEV5EJW2LJCYJSSHAG2CJIU2A, seg_pos={0 0}, upload=473444]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: write tcp 188.166.225.170:43510->77.90.31.16:4733: use of closed network connection; context canceled Jul 18 01:09:36.439338648 (591h59m11.209979983s) [seg_pos={0 0}, upload=473444]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 01:09:36.439407128 (591h59m11.210048464s) [upload=473444]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 01:09:36.439411076 (591h59m11.210052411s) [upload=473444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 01:09:36.756883131 (591h59m11.527524466s) [upload=473444]: Done uploading segments... Jul 18 01:09:36.756897708 (591h59m11.527539044s) [upload=473444]: Upload finished. err: <nil> Jul 18 01:10:39.025598816 (592h0m13.796240151s) [upload=473459]: Starting upload Jul 18 01:10:39.025658440 (592h0m13.796299765s) [upload=473459]: Done starting upload Jul 18 01:10:39.025759353 (592h0m13.796400688s) [upload=473459]: Uploading segments... Jul 18 01:10:39.025907185 (592h0m13.79654852s) [upload=473459]: Got next segment. Inline: false Jul 18 01:10:39.025950799 (592h0m13.796592124s) [upload=473459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 01:10:39.025957632 (592h0m13.796598967s) [upload=473459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 01:10:39.286495526 (592h0m14.057136861s) [seg_pos={0 0}, upload=473459]: Begin upload segment... Jul 18 01:10:39.286547334 (592h0m14.057188699s) [seg_pos={0 0}, upload=473459]: Done begin upload segment. Jul 18 01:10:39.286596759 (592h0m14.057238094s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.286620685 (592h0m14.05726202s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287116924 (592h0m14.057758259s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287176458 (592h0m14.057817793s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287488926 (592h0m14.058130261s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287772308 (592h0m14.058413643s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287863583 (592h0m14.058504918s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288079506 (592h0m14.058720841s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288237839 (592h0m14.058879184s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288418594 (592h0m14.059059929s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288629087 (592h0m14.059270422s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289488541 (592h0m14.060129877s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289738971 (592h0m14.060380306s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289989741 (592h0m14.060631086s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.290216545 (592h0m14.06085788s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.297416701 (592h0m14.068058036s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:41.030217031 (592h0m15.800858356s) [upload=473459]: Next returned nil segment Jul 18 01:10:41.030234745 (592h0m15.80087608s) [upload=473459]: Waiting for error group managing segments... Jul 18 01:10:41.308832612 (592h0m16.079473947s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.358887261 (592h0m16.129528596s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.365986644 (592h0m16.136627979s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.375224368 (592h0m16.145865704s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.419739118 (592h0m16.190380453s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.430360108 (592h0m16.201001444s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.450861434 (592h0m16.221502769s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468109615 (592h0m16.23875095s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468727326 (592h0m16.239368661s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.516013518 (592h0m16.286654863s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.522587155 (592h0m16.29322849s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.553218502 (592h0m16.323859837s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:11:08.331622589 (592h0m43.102263924s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38902->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331668216 (592h0m43.102309551s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:08.331710026 (592h0m43.102351361s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:60806->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331772906 (592h0m43.102414241s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331804857 (592h0m43.102446182s) [seg_pos={0 0}, upload=473459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 01:11:08.598012820 (592h0m43.368654156s) [upload=473459]: Done uploading segments... Jul 18 01:11:08.598041896 (592h0m43.368683231s) [upload=473459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.908010929s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................................................XX........] duration: 5.907679423s FAIL (2025-07-18 01:17:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ress=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683391198 (592h0m30.454032533s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683761526 (592h0m30.454402861s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683801033 (592h0m30.454442368s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684063464 (592h0m30.454704799s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684360503 (592h0m30.455001838s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684654846 (592h0m30.455296181s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685431742 (592h0m30.456073077s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685861425 (592h0m30.45650276s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686139176 (592h0m30.456780511s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686424722 (592h0m30.457066057s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686687836 (592h0m30.457329171s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686944527 (592h0m30.457585862s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.687208402 (592h0m30.457849737s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688069810 (592h0m30.458711145s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688391375 (592h0m30.45903271s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688755091 (592h0m30.459396426s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.689020598 (592h0m30.459661934s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:56.791786484 (592h0m31.562427839s) [upload=473462]: Next returned nil segment Jul 18 01:10:56.791810180 (592h0m31.562451535s) [upload=473462]: Waiting for error group managing segments... Jul 18 01:10:57.072508846 (592h0m31.843150181s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.084313581 (592h0m31.854954917s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.130829109 (592h0m31.901470444s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.380019692 (592h0m32.150661027s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.586116023 (592h0m32.356757358s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.590030633 (592h0m32.360671968s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.644593087 (592h0m32.415234412s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.711170438 (592h0m32.481811773s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.815350524 (592h0m32.585991869s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:11:54.506942141 (592h1m29.277583476s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: rpc: dial tcp 77.90.31.13:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.506990634 (592h1m29.277631959s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507021904 (592h1m29.277663239s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: rpc: dial tcp 77.90.31.19:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507055037 (592h1m29.277696372s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: rpc: dial tcp 77.90.31.18:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507085656 (592h1m29.277726992s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 188.166.225.170:42120->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507153856 (592h1m29.277795192s) [noise=true , address=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj, address:77.90.31.15:4704): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507228530 (592h1m29.277869865s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507325304 (592h1m29.277966639s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12eUC555Lzo37B5FafAFMfXjSkfm1zv1pRZmH71jf4xpdCWH6kG, address:77.90.31.21:4904): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507433642 (592h1m29.278074977s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507448881 (592h1m29.278090216s) [seg_pos={0 0}, upload=473462]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 18 01:11:54.756684719 (592h1m29.527326054s) [upload=473462]: Done uploading segments... Jul 18 01:11:54.756710668 (592h1m29.527352004s) [upload=473462]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 5.907708058s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................................................XX.......] duration: 5.777976702s FAIL (2025-07-18 01:16:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ress=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683391198 (592h0m30.454032533s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683761526 (592h0m30.454402861s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683801033 (592h0m30.454442368s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684063464 (592h0m30.454704799s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684360503 (592h0m30.455001838s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684654846 (592h0m30.455296181s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685431742 (592h0m30.456073077s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685861425 (592h0m30.45650276s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686139176 (592h0m30.456780511s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686424722 (592h0m30.457066057s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686687836 (592h0m30.457329171s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686944527 (592h0m30.457585862s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.687208402 (592h0m30.457849737s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688069810 (592h0m30.458711145s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688391375 (592h0m30.45903271s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688755091 (592h0m30.459396426s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.689020598 (592h0m30.459661934s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:56.791786484 (592h0m31.562427839s) [upload=473462]: Next returned nil segment Jul 18 01:10:56.791810180 (592h0m31.562451535s) [upload=473462]: Waiting for error group managing segments... Jul 18 01:10:57.072508846 (592h0m31.843150181s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.084313581 (592h0m31.854954917s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.130829109 (592h0m31.901470444s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.380019692 (592h0m32.150661027s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.586116023 (592h0m32.356757358s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.590030633 (592h0m32.360671968s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.644593087 (592h0m32.415234412s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.711170438 (592h0m32.481811773s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.815350524 (592h0m32.585991869s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:11:54.506942141 (592h1m29.277583476s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: rpc: dial tcp 77.90.31.13:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.506990634 (592h1m29.277631959s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507021904 (592h1m29.277663239s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: rpc: dial tcp 77.90.31.19:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507055037 (592h1m29.277696372s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: rpc: dial tcp 77.90.31.18:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507085656 (592h1m29.277726992s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 188.166.225.170:42120->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507153856 (592h1m29.277795192s) [noise=true , address=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj, address:77.90.31.15:4704): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507228530 (592h1m29.277869865s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507325304 (592h1m29.277966639s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12eUC555Lzo37B5FafAFMfXjSkfm1zv1pRZmH71jf4xpdCWH6kG, address:77.90.31.21:4904): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507433642 (592h1m29.278074977s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507448881 (592h1m29.278090216s) [seg_pos={0 0}, upload=473462]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 18 01:11:54.756684719 (592h1m29.527326054s) [upload=473462]: Done uploading segments... Jul 18 01:11:54.756710668 (592h1m29.527352004s) [upload=473462]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 5.777999045s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X.X......] duration: 6.003198659s FAIL (2025-07-18 01:16:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== T467KJ2SHEV5EJW2LJCYJSSHAG2CJIU2A, seg_pos={0 0}, upload=473444]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: write tcp 188.166.225.170:43510->77.90.31.16:4733: use of closed network connection; context canceled Jul 18 01:09:36.439338648 (591h59m11.209979983s) [seg_pos={0 0}, upload=473444]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 01:09:36.439407128 (591h59m11.210048464s) [upload=473444]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 01:09:36.439411076 (591h59m11.210052411s) [upload=473444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 01:09:36.756883131 (591h59m11.527524466s) [upload=473444]: Done uploading segments... Jul 18 01:09:36.756897708 (591h59m11.527539044s) [upload=473444]: Upload finished. err: <nil> Jul 18 01:10:39.025598816 (592h0m13.796240151s) [upload=473459]: Starting upload Jul 18 01:10:39.025658440 (592h0m13.796299765s) [upload=473459]: Done starting upload Jul 18 01:10:39.025759353 (592h0m13.796400688s) [upload=473459]: Uploading segments... Jul 18 01:10:39.025907185 (592h0m13.79654852s) [upload=473459]: Got next segment. Inline: false Jul 18 01:10:39.025950799 (592h0m13.796592124s) [upload=473459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 01:10:39.025957632 (592h0m13.796598967s) [upload=473459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 01:10:39.286495526 (592h0m14.057136861s) [seg_pos={0 0}, upload=473459]: Begin upload segment... Jul 18 01:10:39.286547334 (592h0m14.057188699s) [seg_pos={0 0}, upload=473459]: Done begin upload segment. Jul 18 01:10:39.286596759 (592h0m14.057238094s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.286620685 (592h0m14.05726202s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287116924 (592h0m14.057758259s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287176458 (592h0m14.057817793s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287488926 (592h0m14.058130261s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287772308 (592h0m14.058413643s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287863583 (592h0m14.058504918s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288079506 (592h0m14.058720841s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288237839 (592h0m14.058879184s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288418594 (592h0m14.059059929s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288629087 (592h0m14.059270422s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289488541 (592h0m14.060129877s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289738971 (592h0m14.060380306s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289989741 (592h0m14.060631086s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.290216545 (592h0m14.06085788s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.297416701 (592h0m14.068058036s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:41.030217031 (592h0m15.800858356s) [upload=473459]: Next returned nil segment Jul 18 01:10:41.030234745 (592h0m15.80087608s) [upload=473459]: Waiting for error group managing segments... Jul 18 01:10:41.308832612 (592h0m16.079473947s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.358887261 (592h0m16.129528596s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.365986644 (592h0m16.136627979s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.375224368 (592h0m16.145865704s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.419739118 (592h0m16.190380453s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.430360108 (592h0m16.201001444s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.450861434 (592h0m16.221502769s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468109615 (592h0m16.23875095s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468727326 (592h0m16.239368661s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.516013518 (592h0m16.286654863s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.522587155 (592h0m16.29322849s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.553218502 (592h0m16.323859837s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:11:08.331622589 (592h0m43.102263924s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38902->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331668216 (592h0m43.102309551s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:08.331710026 (592h0m43.102351361s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:60806->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331772906 (592h0m43.102414241s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331804857 (592h0m43.102446182s) [seg_pos={0 0}, upload=473459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 01:11:08.598012820 (592h0m43.368654156s) [upload=473459]: Done uploading segments... Jul 18 01:11:08.598041896 (592h0m43.368683231s) [upload=473459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.003220972s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....................................................XX......] duration: 5.957092265s FAIL (2025-07-18 01:15:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ress=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683391198 (592h0m30.454032533s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683761526 (592h0m30.454402861s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683801033 (592h0m30.454442368s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684063464 (592h0m30.454704799s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684360503 (592h0m30.455001838s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684654846 (592h0m30.455296181s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685431742 (592h0m30.456073077s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685861425 (592h0m30.45650276s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686139176 (592h0m30.456780511s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686424722 (592h0m30.457066057s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686687836 (592h0m30.457329171s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686944527 (592h0m30.457585862s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.687208402 (592h0m30.457849737s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688069810 (592h0m30.458711145s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688391375 (592h0m30.45903271s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688755091 (592h0m30.459396426s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.689020598 (592h0m30.459661934s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:56.791786484 (592h0m31.562427839s) [upload=473462]: Next returned nil segment Jul 18 01:10:56.791810180 (592h0m31.562451535s) [upload=473462]: Waiting for error group managing segments... Jul 18 01:10:57.072508846 (592h0m31.843150181s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.084313581 (592h0m31.854954917s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.130829109 (592h0m31.901470444s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.380019692 (592h0m32.150661027s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.586116023 (592h0m32.356757358s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.590030633 (592h0m32.360671968s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.644593087 (592h0m32.415234412s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.711170438 (592h0m32.481811773s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.815350524 (592h0m32.585991869s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:11:54.506942141 (592h1m29.277583476s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: rpc: dial tcp 77.90.31.13:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.506990634 (592h1m29.277631959s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507021904 (592h1m29.277663239s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: rpc: dial tcp 77.90.31.19:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507055037 (592h1m29.277696372s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: rpc: dial tcp 77.90.31.18:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507085656 (592h1m29.277726992s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 188.166.225.170:42120->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507153856 (592h1m29.277795192s) [noise=true , address=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj, address:77.90.31.15:4704): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507228530 (592h1m29.277869865s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507325304 (592h1m29.277966639s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12eUC555Lzo37B5FafAFMfXjSkfm1zv1pRZmH71jf4xpdCWH6kG, address:77.90.31.21:4904): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507433642 (592h1m29.278074977s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507448881 (592h1m29.278090216s) [seg_pos={0 0}, upload=473462]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 18 01:11:54.756684719 (592h1m29.527326054s) [upload=473462]: Done uploading segments... Jul 18 01:11:54.756710668 (592h1m29.527352004s) [upload=473462]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 5.957116509s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................X.X.....] duration: 6.047865257s FAIL (2025-07-18 01:15:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== T467KJ2SHEV5EJW2LJCYJSSHAG2CJIU2A, seg_pos={0 0}, upload=473444]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: write tcp 188.166.225.170:43510->77.90.31.16:4733: use of closed network connection; context canceled Jul 18 01:09:36.439338648 (591h59m11.209979983s) [seg_pos={0 0}, upload=473444]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 01:09:36.439407128 (591h59m11.210048464s) [upload=473444]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 01:09:36.439411076 (591h59m11.210052411s) [upload=473444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 01:09:36.756883131 (591h59m11.527524466s) [upload=473444]: Done uploading segments... Jul 18 01:09:36.756897708 (591h59m11.527539044s) [upload=473444]: Upload finished. err: <nil> Jul 18 01:10:39.025598816 (592h0m13.796240151s) [upload=473459]: Starting upload Jul 18 01:10:39.025658440 (592h0m13.796299765s) [upload=473459]: Done starting upload Jul 18 01:10:39.025759353 (592h0m13.796400688s) [upload=473459]: Uploading segments... Jul 18 01:10:39.025907185 (592h0m13.79654852s) [upload=473459]: Got next segment. Inline: false Jul 18 01:10:39.025950799 (592h0m13.796592124s) [upload=473459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 01:10:39.025957632 (592h0m13.796598967s) [upload=473459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 01:10:39.286495526 (592h0m14.057136861s) [seg_pos={0 0}, upload=473459]: Begin upload segment... Jul 18 01:10:39.286547334 (592h0m14.057188699s) [seg_pos={0 0}, upload=473459]: Done begin upload segment. Jul 18 01:10:39.286596759 (592h0m14.057238094s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.286620685 (592h0m14.05726202s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287116924 (592h0m14.057758259s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287176458 (592h0m14.057817793s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287488926 (592h0m14.058130261s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287772308 (592h0m14.058413643s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287863583 (592h0m14.058504918s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288079506 (592h0m14.058720841s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288237839 (592h0m14.058879184s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288418594 (592h0m14.059059929s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288629087 (592h0m14.059270422s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289488541 (592h0m14.060129877s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289738971 (592h0m14.060380306s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289989741 (592h0m14.060631086s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.290216545 (592h0m14.06085788s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.297416701 (592h0m14.068058036s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:41.030217031 (592h0m15.800858356s) [upload=473459]: Next returned nil segment Jul 18 01:10:41.030234745 (592h0m15.80087608s) [upload=473459]: Waiting for error group managing segments... Jul 18 01:10:41.308832612 (592h0m16.079473947s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.358887261 (592h0m16.129528596s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.365986644 (592h0m16.136627979s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.375224368 (592h0m16.145865704s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.419739118 (592h0m16.190380453s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.430360108 (592h0m16.201001444s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.450861434 (592h0m16.221502769s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468109615 (592h0m16.23875095s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468727326 (592h0m16.239368661s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.516013518 (592h0m16.286654863s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.522587155 (592h0m16.29322849s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.553218502 (592h0m16.323859837s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:11:08.331622589 (592h0m43.102263924s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38902->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331668216 (592h0m43.102309551s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:08.331710026 (592h0m43.102351361s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:60806->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331772906 (592h0m43.102414241s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331804857 (592h0m43.102446182s) [seg_pos={0 0}, upload=473459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 01:11:08.598012820 (592h0m43.368654156s) [upload=473459]: Done uploading segments... Jul 18 01:11:08.598041896 (592h0m43.368683231s) [upload=473459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.047893804s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................XX.....] duration: 5.89276224s FAIL (2025-07-18 01:14:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ress=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683391198 (592h0m30.454032533s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683761526 (592h0m30.454402861s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683801033 (592h0m30.454442368s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684063464 (592h0m30.454704799s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684360503 (592h0m30.455001838s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684654846 (592h0m30.455296181s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685431742 (592h0m30.456073077s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685861425 (592h0m30.45650276s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686139176 (592h0m30.456780511s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686424722 (592h0m30.457066057s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686687836 (592h0m30.457329171s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686944527 (592h0m30.457585862s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.687208402 (592h0m30.457849737s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688069810 (592h0m30.458711145s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688391375 (592h0m30.45903271s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688755091 (592h0m30.459396426s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.689020598 (592h0m30.459661934s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:56.791786484 (592h0m31.562427839s) [upload=473462]: Next returned nil segment Jul 18 01:10:56.791810180 (592h0m31.562451535s) [upload=473462]: Waiting for error group managing segments... Jul 18 01:10:57.072508846 (592h0m31.843150181s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.084313581 (592h0m31.854954917s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.130829109 (592h0m31.901470444s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.380019692 (592h0m32.150661027s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.586116023 (592h0m32.356757358s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.590030633 (592h0m32.360671968s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.644593087 (592h0m32.415234412s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.711170438 (592h0m32.481811773s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.815350524 (592h0m32.585991869s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:11:54.506942141 (592h1m29.277583476s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: rpc: dial tcp 77.90.31.13:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.506990634 (592h1m29.277631959s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507021904 (592h1m29.277663239s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: rpc: dial tcp 77.90.31.19:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507055037 (592h1m29.277696372s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: rpc: dial tcp 77.90.31.18:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507085656 (592h1m29.277726992s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 188.166.225.170:42120->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507153856 (592h1m29.277795192s) [noise=true , address=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj, address:77.90.31.15:4704): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507228530 (592h1m29.277869865s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507325304 (592h1m29.277966639s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12eUC555Lzo37B5FafAFMfXjSkfm1zv1pRZmH71jf4xpdCWH6kG, address:77.90.31.21:4904): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507433642 (592h1m29.278074977s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507448881 (592h1m29.278090216s) [seg_pos={0 0}, upload=473462]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 18 01:11:54.756684719 (592h1m29.527326054s) [upload=473462]: Done uploading segments... Jul 18 01:11:54.756710668 (592h1m29.527352004s) [upload=473462]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 5.892790205s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................X.X....] duration: 6.039411263s FAIL (2025-07-18 01:14:06 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== T467KJ2SHEV5EJW2LJCYJSSHAG2CJIU2A, seg_pos={0 0}, upload=473444]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: write tcp 188.166.225.170:43510->77.90.31.16:4733: use of closed network connection; context canceled Jul 18 01:09:36.439338648 (591h59m11.209979983s) [seg_pos={0 0}, upload=473444]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 01:09:36.439407128 (591h59m11.210048464s) [upload=473444]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 01:09:36.439411076 (591h59m11.210052411s) [upload=473444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 01:09:36.756883131 (591h59m11.527524466s) [upload=473444]: Done uploading segments... Jul 18 01:09:36.756897708 (591h59m11.527539044s) [upload=473444]: Upload finished. err: <nil> Jul 18 01:10:39.025598816 (592h0m13.796240151s) [upload=473459]: Starting upload Jul 18 01:10:39.025658440 (592h0m13.796299765s) [upload=473459]: Done starting upload Jul 18 01:10:39.025759353 (592h0m13.796400688s) [upload=473459]: Uploading segments... Jul 18 01:10:39.025907185 (592h0m13.79654852s) [upload=473459]: Got next segment. Inline: false Jul 18 01:10:39.025950799 (592h0m13.796592124s) [upload=473459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 01:10:39.025957632 (592h0m13.796598967s) [upload=473459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 01:10:39.286495526 (592h0m14.057136861s) [seg_pos={0 0}, upload=473459]: Begin upload segment... Jul 18 01:10:39.286547334 (592h0m14.057188699s) [seg_pos={0 0}, upload=473459]: Done begin upload segment. Jul 18 01:10:39.286596759 (592h0m14.057238094s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.286620685 (592h0m14.05726202s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287116924 (592h0m14.057758259s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287176458 (592h0m14.057817793s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287488926 (592h0m14.058130261s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287772308 (592h0m14.058413643s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287863583 (592h0m14.058504918s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288079506 (592h0m14.058720841s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288237839 (592h0m14.058879184s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288418594 (592h0m14.059059929s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288629087 (592h0m14.059270422s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289488541 (592h0m14.060129877s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289738971 (592h0m14.060380306s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289989741 (592h0m14.060631086s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.290216545 (592h0m14.06085788s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.297416701 (592h0m14.068058036s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:41.030217031 (592h0m15.800858356s) [upload=473459]: Next returned nil segment Jul 18 01:10:41.030234745 (592h0m15.80087608s) [upload=473459]: Waiting for error group managing segments... Jul 18 01:10:41.308832612 (592h0m16.079473947s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.358887261 (592h0m16.129528596s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.365986644 (592h0m16.136627979s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.375224368 (592h0m16.145865704s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.419739118 (592h0m16.190380453s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.430360108 (592h0m16.201001444s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.450861434 (592h0m16.221502769s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468109615 (592h0m16.23875095s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468727326 (592h0m16.239368661s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.516013518 (592h0m16.286654863s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.522587155 (592h0m16.29322849s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.553218502 (592h0m16.323859837s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:11:08.331622589 (592h0m43.102263924s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38902->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331668216 (592h0m43.102309551s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:08.331710026 (592h0m43.102351361s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:60806->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331772906 (592h0m43.102414241s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331804857 (592h0m43.102446182s) [seg_pos={0 0}, upload=473459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 01:11:08.598012820 (592h0m43.368654156s) [upload=473459]: Done uploading segments... Jul 18 01:11:08.598041896 (592h0m43.368683231s) [upload=473459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.0394362s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................X.X...] duration: 6.045514541s FAIL (2025-07-18 01:13:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== T467KJ2SHEV5EJW2LJCYJSSHAG2CJIU2A, seg_pos={0 0}, upload=473444]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: write tcp 188.166.225.170:43510->77.90.31.16:4733: use of closed network connection; context canceled Jul 18 01:09:36.439338648 (591h59m11.209979983s) [seg_pos={0 0}, upload=473444]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 01:09:36.439407128 (591h59m11.210048464s) [upload=473444]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 01:09:36.439411076 (591h59m11.210052411s) [upload=473444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 01:09:36.756883131 (591h59m11.527524466s) [upload=473444]: Done uploading segments... Jul 18 01:09:36.756897708 (591h59m11.527539044s) [upload=473444]: Upload finished. err: <nil> Jul 18 01:10:39.025598816 (592h0m13.796240151s) [upload=473459]: Starting upload Jul 18 01:10:39.025658440 (592h0m13.796299765s) [upload=473459]: Done starting upload Jul 18 01:10:39.025759353 (592h0m13.796400688s) [upload=473459]: Uploading segments... Jul 18 01:10:39.025907185 (592h0m13.79654852s) [upload=473459]: Got next segment. Inline: false Jul 18 01:10:39.025950799 (592h0m13.796592124s) [upload=473459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 01:10:39.025957632 (592h0m13.796598967s) [upload=473459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 01:10:39.286495526 (592h0m14.057136861s) [seg_pos={0 0}, upload=473459]: Begin upload segment... Jul 18 01:10:39.286547334 (592h0m14.057188699s) [seg_pos={0 0}, upload=473459]: Done begin upload segment. Jul 18 01:10:39.286596759 (592h0m14.057238094s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.286620685 (592h0m14.05726202s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287116924 (592h0m14.057758259s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287176458 (592h0m14.057817793s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287488926 (592h0m14.058130261s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287772308 (592h0m14.058413643s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287863583 (592h0m14.058504918s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288079506 (592h0m14.058720841s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288237839 (592h0m14.058879184s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288418594 (592h0m14.059059929s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288629087 (592h0m14.059270422s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289488541 (592h0m14.060129877s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289738971 (592h0m14.060380306s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289989741 (592h0m14.060631086s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.290216545 (592h0m14.06085788s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.297416701 (592h0m14.068058036s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:41.030217031 (592h0m15.800858356s) [upload=473459]: Next returned nil segment Jul 18 01:10:41.030234745 (592h0m15.80087608s) [upload=473459]: Waiting for error group managing segments... Jul 18 01:10:41.308832612 (592h0m16.079473947s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.358887261 (592h0m16.129528596s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.365986644 (592h0m16.136627979s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.375224368 (592h0m16.145865704s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.419739118 (592h0m16.190380453s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.430360108 (592h0m16.201001444s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.450861434 (592h0m16.221502769s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468109615 (592h0m16.23875095s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468727326 (592h0m16.239368661s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.516013518 (592h0m16.286654863s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.522587155 (592h0m16.29322849s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.553218502 (592h0m16.323859837s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:11:08.331622589 (592h0m43.102263924s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38902->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331668216 (592h0m43.102309551s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:08.331710026 (592h0m43.102351361s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:60806->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331772906 (592h0m43.102414241s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331804857 (592h0m43.102446182s) [seg_pos={0 0}, upload=473459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 01:11:08.598012820 (592h0m43.368654156s) [upload=473459]: Done uploading segments... Jul 18 01:11:08.598041896 (592h0m43.368683231s) [upload=473459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.045551122s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......................................................XX....] duration: 6.096320387s FAIL (2025-07-18 01:13:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ress=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683391198 (592h0m30.454032533s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683761526 (592h0m30.454402861s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683801033 (592h0m30.454442368s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684063464 (592h0m30.454704799s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684360503 (592h0m30.455001838s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684654846 (592h0m30.455296181s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685431742 (592h0m30.456073077s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685861425 (592h0m30.45650276s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686139176 (592h0m30.456780511s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686424722 (592h0m30.457066057s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686687836 (592h0m30.457329171s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686944527 (592h0m30.457585862s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.687208402 (592h0m30.457849737s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688069810 (592h0m30.458711145s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688391375 (592h0m30.45903271s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688755091 (592h0m30.459396426s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.689020598 (592h0m30.459661934s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:56.791786484 (592h0m31.562427839s) [upload=473462]: Next returned nil segment Jul 18 01:10:56.791810180 (592h0m31.562451535s) [upload=473462]: Waiting for error group managing segments... Jul 18 01:10:57.072508846 (592h0m31.843150181s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.084313581 (592h0m31.854954917s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.130829109 (592h0m31.901470444s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.380019692 (592h0m32.150661027s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.586116023 (592h0m32.356757358s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.590030633 (592h0m32.360671968s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.644593087 (592h0m32.415234412s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.711170438 (592h0m32.481811773s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.815350524 (592h0m32.585991869s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:11:54.506942141 (592h1m29.277583476s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: rpc: dial tcp 77.90.31.13:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.506990634 (592h1m29.277631959s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507021904 (592h1m29.277663239s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: rpc: dial tcp 77.90.31.19:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507055037 (592h1m29.277696372s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: rpc: dial tcp 77.90.31.18:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507085656 (592h1m29.277726992s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 188.166.225.170:42120->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507153856 (592h1m29.277795192s) [noise=true , address=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj, address:77.90.31.15:4704): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507228530 (592h1m29.277869865s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507325304 (592h1m29.277966639s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12eUC555Lzo37B5FafAFMfXjSkfm1zv1pRZmH71jf4xpdCWH6kG, address:77.90.31.21:4904): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507433642 (592h1m29.278074977s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507448881 (592h1m29.278090216s) [seg_pos={0 0}, upload=473462]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 18 01:11:54.756684719 (592h1m29.527326054s) [upload=473462]: Done uploading segments... Jul 18 01:11:54.756710668 (592h1m29.527352004s) [upload=473462]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 6.096346827s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................XX...] duration: 5.860146877s FAIL (2025-07-18 01:12:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ress=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683391198 (592h0m30.454032533s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683761526 (592h0m30.454402861s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683801033 (592h0m30.454442368s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684063464 (592h0m30.454704799s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684360503 (592h0m30.455001838s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684654846 (592h0m30.455296181s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685431742 (592h0m30.456073077s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685861425 (592h0m30.45650276s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686139176 (592h0m30.456780511s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686424722 (592h0m30.457066057s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686687836 (592h0m30.457329171s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686944527 (592h0m30.457585862s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.687208402 (592h0m30.457849737s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688069810 (592h0m30.458711145s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688391375 (592h0m30.45903271s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688755091 (592h0m30.459396426s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.689020598 (592h0m30.459661934s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:56.791786484 (592h0m31.562427839s) [upload=473462]: Next returned nil segment Jul 18 01:10:56.791810180 (592h0m31.562451535s) [upload=473462]: Waiting for error group managing segments... Jul 18 01:10:57.072508846 (592h0m31.843150181s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.084313581 (592h0m31.854954917s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.130829109 (592h0m31.901470444s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.380019692 (592h0m32.150661027s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.586116023 (592h0m32.356757358s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.590030633 (592h0m32.360671968s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.644593087 (592h0m32.415234412s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.711170438 (592h0m32.481811773s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.815350524 (592h0m32.585991869s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:11:54.506942141 (592h1m29.277583476s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: rpc: dial tcp 77.90.31.13:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.506990634 (592h1m29.277631959s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507021904 (592h1m29.277663239s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: rpc: dial tcp 77.90.31.19:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507055037 (592h1m29.277696372s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: rpc: dial tcp 77.90.31.18:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507085656 (592h1m29.277726992s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 188.166.225.170:42120->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507153856 (592h1m29.277795192s) [noise=true , address=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj, address:77.90.31.15:4704): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507228530 (592h1m29.277869865s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507325304 (592h1m29.277966639s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12eUC555Lzo37B5FafAFMfXjSkfm1zv1pRZmH71jf4xpdCWH6kG, address:77.90.31.21:4904): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507433642 (592h1m29.278074977s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507448881 (592h1m29.278090216s) [seg_pos={0 0}, upload=473462]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 18 01:11:54.756684719 (592h1m29.527326054s) [upload=473462]: Done uploading segments... Jul 18 01:11:54.756710668 (592h1m29.527352004s) [upload=473462]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 5.86017525s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................X.X..] duration: 5.973067379s FAIL (2025-07-18 01:12:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== T467KJ2SHEV5EJW2LJCYJSSHAG2CJIU2A, seg_pos={0 0}, upload=473444]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: write tcp 188.166.225.170:43510->77.90.31.16:4733: use of closed network connection; context canceled Jul 18 01:09:36.439338648 (591h59m11.209979983s) [seg_pos={0 0}, upload=473444]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 01:09:36.439407128 (591h59m11.210048464s) [upload=473444]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 01:09:36.439411076 (591h59m11.210052411s) [upload=473444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 01:09:36.756883131 (591h59m11.527524466s) [upload=473444]: Done uploading segments... Jul 18 01:09:36.756897708 (591h59m11.527539044s) [upload=473444]: Upload finished. err: <nil> Jul 18 01:10:39.025598816 (592h0m13.796240151s) [upload=473459]: Starting upload Jul 18 01:10:39.025658440 (592h0m13.796299765s) [upload=473459]: Done starting upload Jul 18 01:10:39.025759353 (592h0m13.796400688s) [upload=473459]: Uploading segments... Jul 18 01:10:39.025907185 (592h0m13.79654852s) [upload=473459]: Got next segment. Inline: false Jul 18 01:10:39.025950799 (592h0m13.796592124s) [upload=473459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 01:10:39.025957632 (592h0m13.796598967s) [upload=473459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 01:10:39.286495526 (592h0m14.057136861s) [seg_pos={0 0}, upload=473459]: Begin upload segment... Jul 18 01:10:39.286547334 (592h0m14.057188699s) [seg_pos={0 0}, upload=473459]: Done begin upload segment. Jul 18 01:10:39.286596759 (592h0m14.057238094s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.286620685 (592h0m14.05726202s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287116924 (592h0m14.057758259s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287176458 (592h0m14.057817793s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287488926 (592h0m14.058130261s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287772308 (592h0m14.058413643s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.287863583 (592h0m14.058504918s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288079506 (592h0m14.058720841s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288237839 (592h0m14.058879184s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288418594 (592h0m14.059059929s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.288629087 (592h0m14.059270422s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289488541 (592h0m14.060129877s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289738971 (592h0m14.060380306s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.289989741 (592h0m14.060631086s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.290216545 (592h0m14.06085788s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:39.297416701 (592h0m14.068058036s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Uploading piece... Jul 18 01:10:41.030217031 (592h0m15.800858356s) [upload=473459]: Next returned nil segment Jul 18 01:10:41.030234745 (592h0m15.80087608s) [upload=473459]: Waiting for error group managing segments... Jul 18 01:10:41.308832612 (592h0m16.079473947s) [noise=true , address=45.91.116.8:4047 , piece_id=7E6K2AWD3YANEN2WYD5CPT3U3VBWA6VHNVDF6DC5NJUGQHP3A3NA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.358887261 (592h0m16.129528596s) [noise=true , address=104.36.82.16:4295 , piece_id=N2YQBIMKQ7DEOWFMXWTBCNUMWXYKO2E2BEAKNMZYGUZ5N4BRUNTA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.365986644 (592h0m16.136627979s) [noise=true , address=104.36.82.13:4185 , piece_id=OOVUZS6RGGFLT5RS47MLGNSI6O4QGTCAIFTVISY3YSRULSDDMDSQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.375224368 (592h0m16.145865704s) [noise=true , address=45.91.116.4:4937 , piece_id=MHBQFHGDBWQ4NGAQNIY5NVYAT2ELG2BAVIDSRWZRSORFR35WVUFQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.419739118 (592h0m16.190380453s) [noise=true , address=104.36.82.13:4187 , piece_id=7JSX6JTF7PFLA5EJJKP5ERYMZ3WUZMBXKP5SEIOJQEJ64YCAWLNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.430360108 (592h0m16.201001444s) [noise=true , address=45.91.116.9:4262 , piece_id=P52ZRQP7U767JA7SSYWS6WPSN77X2O3AC6MNKJCESIMDC5O2XJGA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.450861434 (592h0m16.221502769s) [noise=true , address=104.36.82.5:4840 , piece_id=E5XS5KTCG5S3ELBLWWY5MSHSEHI6YF3C45KFZASYI55U2MAIJQNQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468109615 (592h0m16.23875095s) [noise=true , address=45.91.116.10:4082 , piece_id=E5X3OLJ7LZXI77WBGWBRNMFRNS6NGX5PFVPUOWPBJVYTGWDWB4YA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.468727326 (592h0m16.239368661s) [noise=true , address=45.91.116.11:4118 , piece_id=VLA6S3CZOJZQSUYUIIE62FHEJDUEH35S2UP6CJOXFK7ATJCU2TVQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.516013518 (592h0m16.286654863s) [noise=true , address=104.36.82.15:4259 , piece_id=RC3W63JUQD22JS4YV3P3FX4THBBPTVN4KMPWMSOZSEMKBPVABC5Q, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.522587155 (592h0m16.29322849s) [noise=true , address=104.36.82.2:4730 , piece_id=IQPV5SIE4KNSQI4LFTT7IR3IPF5OLW5G67NUSMF23O2GNR5GAWUQ, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:10:41.553218502 (592h0m16.323859837s) [noise=true , address=45.91.116.12:4226 , piece_id=5ZXQMWANA43PWXPQWE4LL5OCBEKETXV2EZC4VPRPF63N52PG7X4A, seg_pos={0 0}, upload=473459]: Done uploading piece. err: <nil> Jul 18 01:11:08.331622589 (592h0m43.102263924s) [noise=true , address=77.90.31.15:4699 , piece_id=BFA5C3AXYQSLHJBMR4PSU3XNFBK3IJHADGJEHOWK34FCAUL67KFA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38902->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331668216 (592h0m43.102309551s) [noise=true , address=77.90.31.13:4627 , piece_id=DP5K3YEDPDYIFDVARNAW3FM2LNQNZBCMXHPSQIOGWYKXNFKD6DSA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:08.331710026 (592h0m43.102351361s) [noise=true , address=77.90.31.19:4841 , piece_id=VXP6EFC7M6GEHWE4556WCZ7UYNWIQOXLQ3DJR4YU66IRQY2MBAIA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:60806->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331772906 (592h0m43.102414241s) [noise=true , address=77.90.31.19:4842 , piece_id=AX6OKKHUNOLSUPVQE3JXQKSGEN54AGDIAR6JGAYOPMGE2NTEVXXA, seg_pos={0 0}, upload=473459]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:08.331804857 (592h0m43.102446182s) [seg_pos={0 0}, upload=473459]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 01:11:08.598012820 (592h0m43.368654156s) [upload=473459]: Done uploading segments... Jul 18 01:11:08.598041896 (592h0m43.368683231s) [upload=473459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.973094119s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........................................................XX..] duration: 5.895464853s FAIL (2025-07-18 01:12:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ress=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683391198 (592h0m30.454032533s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683761526 (592h0m30.454402861s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683801033 (592h0m30.454442368s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684063464 (592h0m30.454704799s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684360503 (592h0m30.455001838s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684654846 (592h0m30.455296181s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685431742 (592h0m30.456073077s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685861425 (592h0m30.45650276s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686139176 (592h0m30.456780511s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686424722 (592h0m30.457066057s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686687836 (592h0m30.457329171s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686944527 (592h0m30.457585862s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.687208402 (592h0m30.457849737s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688069810 (592h0m30.458711145s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688391375 (592h0m30.45903271s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688755091 (592h0m30.459396426s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.689020598 (592h0m30.459661934s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:56.791786484 (592h0m31.562427839s) [upload=473462]: Next returned nil segment Jul 18 01:10:56.791810180 (592h0m31.562451535s) [upload=473462]: Waiting for error group managing segments... Jul 18 01:10:57.072508846 (592h0m31.843150181s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.084313581 (592h0m31.854954917s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.130829109 (592h0m31.901470444s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.380019692 (592h0m32.150661027s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.586116023 (592h0m32.356757358s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.590030633 (592h0m32.360671968s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.644593087 (592h0m32.415234412s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.711170438 (592h0m32.481811773s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.815350524 (592h0m32.585991869s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:11:54.506942141 (592h1m29.277583476s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: rpc: dial tcp 77.90.31.13:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.506990634 (592h1m29.277631959s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507021904 (592h1m29.277663239s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: rpc: dial tcp 77.90.31.19:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507055037 (592h1m29.277696372s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: rpc: dial tcp 77.90.31.18:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507085656 (592h1m29.277726992s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 188.166.225.170:42120->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507153856 (592h1m29.277795192s) [noise=true , address=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj, address:77.90.31.15:4704): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507228530 (592h1m29.277869865s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507325304 (592h1m29.277966639s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12eUC555Lzo37B5FafAFMfXjSkfm1zv1pRZmH71jf4xpdCWH6kG, address:77.90.31.21:4904): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507433642 (592h1m29.278074977s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507448881 (592h1m29.278090216s) [seg_pos={0 0}, upload=473462]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 18 01:11:54.756684719 (592h1m29.527326054s) [upload=473462]: Done uploading segments... Jul 18 01:11:54.756710668 (592h1m29.527352004s) [upload=473462]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 5.895494379s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................XX.] duration: 5.750300111s FAIL (2025-07-18 01:12:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ress=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683391198 (592h0m30.454032533s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683761526 (592h0m30.454402861s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683801033 (592h0m30.454442368s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684063464 (592h0m30.454704799s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684360503 (592h0m30.455001838s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684654846 (592h0m30.455296181s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685431742 (592h0m30.456073077s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685861425 (592h0m30.45650276s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686139176 (592h0m30.456780511s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686424722 (592h0m30.457066057s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686687836 (592h0m30.457329171s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686944527 (592h0m30.457585862s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.687208402 (592h0m30.457849737s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688069810 (592h0m30.458711145s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688391375 (592h0m30.45903271s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688755091 (592h0m30.459396426s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.689020598 (592h0m30.459661934s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:56.791786484 (592h0m31.562427839s) [upload=473462]: Next returned nil segment Jul 18 01:10:56.791810180 (592h0m31.562451535s) [upload=473462]: Waiting for error group managing segments... Jul 18 01:10:57.072508846 (592h0m31.843150181s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.084313581 (592h0m31.854954917s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.130829109 (592h0m31.901470444s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.380019692 (592h0m32.150661027s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.586116023 (592h0m32.356757358s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.590030633 (592h0m32.360671968s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.644593087 (592h0m32.415234412s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.711170438 (592h0m32.481811773s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.815350524 (592h0m32.585991869s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:11:54.506942141 (592h1m29.277583476s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: rpc: dial tcp 77.90.31.13:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.506990634 (592h1m29.277631959s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507021904 (592h1m29.277663239s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: rpc: dial tcp 77.90.31.19:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507055037 (592h1m29.277696372s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: rpc: dial tcp 77.90.31.18:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507085656 (592h1m29.277726992s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 188.166.225.170:42120->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507153856 (592h1m29.277795192s) [noise=true , address=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj, address:77.90.31.15:4704): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507228530 (592h1m29.277869865s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507325304 (592h1m29.277966639s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12eUC555Lzo37B5FafAFMfXjSkfm1zv1pRZmH71jf4xpdCWH6kG, address:77.90.31.21:4904): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507433642 (592h1m29.278074977s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507448881 (592h1m29.278090216s) [seg_pos={0 0}, upload=473462]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 18 01:11:54.756684719 (592h1m29.527326054s) [upload=473462]: Done uploading segments... Jul 18 01:11:54.756710668 (592h1m29.527352004s) [upload=473462]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 5.750320961s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 1m0.252041492s FAIL (2025-07-18 01:11:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ress=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683391198 (592h0m30.454032533s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683761526 (592h0m30.454402861s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.683801033 (592h0m30.454442368s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684063464 (592h0m30.454704799s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684360503 (592h0m30.455001838s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.684654846 (592h0m30.455296181s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685431742 (592h0m30.456073077s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.685861425 (592h0m30.45650276s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686139176 (592h0m30.456780511s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686424722 (592h0m30.457066057s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686687836 (592h0m30.457329171s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.686944527 (592h0m30.457585862s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.687208402 (592h0m30.457849737s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688069810 (592h0m30.458711145s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688391375 (592h0m30.45903271s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.688755091 (592h0m30.459396426s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:55.689020598 (592h0m30.459661934s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Uploading piece... Jul 18 01:10:56.791786484 (592h0m31.562427839s) [upload=473462]: Next returned nil segment Jul 18 01:10:56.791810180 (592h0m31.562451535s) [upload=473462]: Waiting for error group managing segments... Jul 18 01:10:57.072508846 (592h0m31.843150181s) [noise=true , address=79.127.205.233:4641 , piece_id=ZITXZYCAPCMYSQZNGMMGS4K3OAOLYTILFH5QKLCXHW6NJIGPACLA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.084313581 (592h0m31.854954917s) [noise=true , address=79.127.205.245:4224 , piece_id=HAJ52S6XX66XD3C6ZMU56YP5OQEEYDPRTSTVDBGXOXHAHAUY3N3A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.130829109 (592h0m31.901470444s) [noise=true , address=79.127.205.234:4595 , piece_id=MK6APXSMEA2T2Q6BZQUXEAKETGRYSJCFYOSAR3WEOWWPBKCNVXHQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.380019692 (592h0m32.150661027s) [noise=true , address=79.127.220.101:4045 , piece_id=ZZOB4VJ2UG4SQ2H43EIZGO3FHIQOENHFWAPGLKFM3RTYLK2DEKPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.586116023 (592h0m32.356757358s) [noise=true , address=79.127.219.48:4107 , piece_id=UJMGZGLBBUHNAUSZORMGK37U2GMWDHDUE43YAOF2YMALCH6H5WEA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.590030633 (592h0m32.360671968s) [noise=true , address=79.127.201.215:4349 , piece_id=VRU2SIUMMBI5UGH77OEGC6NL4P3UBSKVAN7WH2TP5B5UQNBZMC5A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.644593087 (592h0m32.415234412s) [noise=true , address=45.91.95.4:4012 , piece_id=TYKHU53FWX7OATNYZ6SJZXI2HGNVCY777A7CVDFXJORVO5MNII4A, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.711170438 (592h0m32.481811773s) [noise=true , address=104.36.82.50:4019 , piece_id=SOWQ4LMLRKAKIMR7YJI62NEV5K3JU3EINVRRDBL7PBMQV4WE4CPQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:10:57.815350524 (592h0m32.585991869s) [noise=true , address=45.91.116.12:4254 , piece_id=QPEMFLQOTSSROFEKSDKARMJ7R6BZGY44TXAOLAP35ZH73TCHBNLQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: <nil> Jul 18 01:11:54.506942141 (592h1m29.277583476s) [noise=true , address=77.90.31.13:4635 , piece_id=LUOGOPKTRW3BJIWBVTVRGXRFMXMYASMOXCW254FZQPRJOCQPXNQA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: rpc: dial tcp 77.90.31.13:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.506990634 (592h1m29.277631959s) [noise=true , address=77.90.31.14:4689 , piece_id=YBQH3RBNJG3YACTBUTGRTVVXCOTQMIN7AM7H6Y6BEKK34QYKYAZQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507021904 (592h1m29.277663239s) [noise=true , address=77.90.31.19:4850 , piece_id=NNE6FK2G3YOIR5F6X426L5P6LQ6ZLHBJHCOMDKBVSXP2BICK4WFQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: rpc: dial tcp 77.90.31.19:4850: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507055037 (592h1m29.277696372s) [noise=true , address=77.90.31.18:4821 , piece_id=DAXP3Y4RD4TF34YCWYAAMUNXVA26LYDTNJ64YTQYJ5ZPQMPB5D5Q, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: rpc: dial tcp 77.90.31.18:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507085656 (592h1m29.277726992s) [noise=true , address=77.90.31.18:4830 , piece_id=6DACHMTF7HVRL66TNQW4E2FOYIGLXKIQCFPD5GWYPOJAMFPB32PA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 188.166.225.170:42120->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 01:11:54.507153856 (592h1m29.277795192s) [noise=true , address=77.90.31.15:4704 , piece_id=Q5OMCQTADPUW6BRQJXZ52PAQEPQ7ZNF3Q3PTQZ3WQGM55VOKMMXA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj, address:77.90.31.15:4704): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507228530 (592h1m29.277869865s) [noise=true , address=77.90.31.17:4783 , piece_id=67OLYYUNANWGBRK5TQFCSFYA3YR6BKAP2U5TCXQDZV2PVC5I2APA, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo, address:77.90.31.17:4783): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507325304 (592h1m29.277966639s) [noise=true , address=77.90.31.21:4904 , piece_id=X66X7H3DPI3WCAJUIGXNAA3VHAM3FUMP4IYPZRSK3ZBGUSJVDJGQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:12eUC555Lzo37B5FafAFMfXjSkfm1zv1pRZmH71jf4xpdCWH6kG, address:77.90.31.21:4904): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507433642 (592h1m29.278074977s) [noise=true , address=77.90.31.11:4562 , piece_id=V3BRXEN4M5QUC6G4S6GRU3JL4D7AGYPW3KQPOEVSYOXW2FQIRHRQ, seg_pos={0 0}, upload=473462]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded Jul 18 01:11:54.507448881 (592h1m29.278090216s) [seg_pos={0 0}, upload=473462]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 18 01:11:54.756684719 (592h1m29.527326054s) [upload=473462]: Done uploading segments... Jul 18 01:11:54.756710668 (592h1m29.527352004s) [upload=473462]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.252052314s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)