Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.......X....................XX.XX.X..X.X...XX.X......X..X...] duration: 9.886779345s trace: https://jaeger.dev.storj.io/trace/536c87b2b89ff91b FAIL (2025-12-26 21:33:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 21:30:51.550437270 (510h1m51.421520305s) [upload=447552]: Starting upload Dec 26 21:30:51.550493317 (510h1m51.421576352s) [upload=447552]: Done starting upload Dec 26 21:30:51.550654945 (510h1m51.42173799s) [upload=447552]: Uploading segments... Dec 26 21:30:51.550767770 (510h1m51.421850805s) [upload=447552]: Got next segment. Inline: false Dec 26 21:30:51.550804860 (510h1m51.421887905s) [upload=447552]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 21:30:51.550810190 (510h1m51.421893235s) [upload=447552]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 21:30:51.794426629 (510h1m51.665509674s) [seg_pos={0 0}, upload=447552]: Begin upload segment... Dec 26 21:30:51.794483798 (510h1m51.665566853s) [seg_pos={0 0}, upload=447552]: Done begin upload segment. Dec 26 21:30:51.794495490 (510h1m51.665578525s) [noise=true , address=45.91.116.9:4264 , piece_id=QXVYPKJUVVVMTXMIPFHME4QVZF5SZJ536W5ZT6CLG3Q7TDCRPW2Q, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.794525237 (510h1m51.665608282s) [noise=true , address=77.90.31.14:4661 , piece_id=XAOMABOK36EBDLUI5NGILIJP5HJ3YTRO7IQTRYBIW4M3WIEFRLIQ, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.795292047 (510h1m51.666375082s) [noise=true , address=145.79.227.17:4972 , piece_id=ILM4WRBZKYGUKJKRHC7H4ZE2JWXOZTBFPU4HLFQ6LKCDHFH2VJMQ, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.795781268 (510h1m51.666864303s) [noise=true , address=77.90.31.13:4627 , piece_id=DM2PWELMEZOYOKQJOI2LFYFQ72QULYJYMDSQL7RJKU4P2JLXUIZA, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.796130614 (510h1m51.667213649s) [noise=true , address=145.79.227.11:4732 , piece_id=PFFVB3WH4DBOANMRNBHIGYFLLL35RTMPAYPG7QU5D7VMUCEBD57A, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.796561053 (510h1m51.667644088s) [noise=true , address=145.79.227.40:4050 , piece_id=ZMHHZIYZXT4KII2EUY3NORH5H4YH6WQWLK4S2OHYG3LBMEQIRXUQ, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.796698735 (510h1m51.66778177s) [noise=true , address=145.79.227.52:4446 , piece_id=ZXOPINHEA6WMCX4GREWOAWTXLZAL4FUUATQRJTP62L2TRGWLPKJQ, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.797166186 (510h1m51.668249221s) [noise=true , address=145.79.227.43:4159 , piece_id=VLAIPO7BPFNE7DI2L4UKQYUANDHUQVWFH7GI4SXC6KVTN575QXVA, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.797517014 (510h1m51.668600049s) [noise=true , address=145.79.227.37:4565 , piece_id=2HZYAHWH3BPF4IOP5OHGUFJFK7JYY2RB5ZTJA537SGDABU3DMRBQ, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.797859055 (510h1m51.66894209s) [noise=true , address=145.79.227.47:4265 , piece_id=3PUY42FXCQK62GNKAWIJ57X2KECZU3G5NA56YWRL7C7YADIL662A, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.798190917 (510h1m51.669273952s) [noise=true , address=145.79.227.45:4229 , piece_id=MDBKNXVJMWZD7XIFK362N3KQ2ZKS6JXDN3ADAGPUGD6EA6JYOCKQ, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.798535252 (510h1m51.669618287s) [noise=true , address=145.79.227.46:4601 , piece_id=7YNQC7KH6KUHXRPODLYGY33YXOGGSZ6KAQUEMLSEJVBKNGJBI4VQ, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.798864399 (510h1m51.669947434s) [noise=true , address=45.91.116.13:4153 , piece_id=YWDYXI7HO4546TDIWFKEHL7VYQ5O7WZ3XHGNGOBM67EIWO6OTFXQ, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.799268939 (510h1m51.670351984s) [noise=true , address=145.79.227.9:4365 , piece_id=OBHOCV5YR2XY4EFVUTK3WXEVW7WZXDG3LTGEAWVAUVCPEGLF2FIA, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.800747806 (510h1m51.671830841s) [noise=true , address=45.91.116.4:4939 , piece_id=SAHLWHCM5FID76UL6B7NFLFFJVIS4K6X5T3RFUXTIFQHR4VKAKKQ, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.803663848 (510h1m51.674746883s) [noise=true , address=77.90.31.21:4880 , piece_id=L2SXABFCPAQUDQOV47AGQJFFVC3ZVFQIVRFP2BNWE4HUFZFA5EAA, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:53.429958811 (510h1m53.301041846s) [upload=447552]: Next returned nil segment Dec 26 21:30:53.429972518 (510h1m53.301055563s) [upload=447552]: Waiting for error group managing segments... Dec 26 21:30:53.814614754 (510h1m53.685697789s) [noise=true , address=145.79.227.46:4601 , piece_id=7YNQC7KH6KUHXRPODLYGY33YXOGGSZ6KAQUEMLSEJVBKNGJBI4VQ, seg_pos={0 0}, upload=447552]: Done uploading piece. err: <nil> Dec 26 21:30:53.816217045 (510h1m53.68730008s) [noise=true , address=145.79.227.52:4446 , piece_id=ZXOPINHEA6WMCX4GREWOAWTXLZAL4FUUATQRJTP62L2TRGWLPKJQ, seg_pos={0 0}, upload=447552]: Done uploading piece. err: <nil> Dec 26 21:30:53.825544096 (510h1m53.696627141s) [noise=true , address=145.79.227.37:4565 , piece_id=2HZYAHWH3BPF4IOP5OHGUFJFK7JYY2RB5ZTJA537SGDABU3DMRBQ, seg_pos={0 0}, upload=447552]: Done uploading piece. err: <nil> Dec 26 21:30:53.836843141 (510h1m53.707926187s) [noise=true , address=145.79.227.47:4265 , piece_id=3PUY42FXCQK62GNKAWIJ57X2KECZU3G5NA56YWRL7C7YADIL662A, seg_pos={0 0}, upload=447552]: Done uploading piece. err: <nil> Dec 26 21:30:53.837329888 (510h1m53.708412933s) [noise=true , address=145.79.227.9:4365 , piece_id=OBHOCV5YR2XY4EFVUTK3WXEVW7WZXDG3LTGEAWVAUVCPEGLF2FIA, seg_pos={0 0}, upload=447552]: Done uploading piece. err: <nil> Dec 26 21:30:53.838208611 (510h1m53.709291646s) [noise=true , address=145.79.227.11:4732 , piece_id=PFFVB3WH4DBOANMRNBHIGYFLLL35RTMPAYPG7QU5D7VMUCEBD57A, seg_pos={0 0}, upload=447552]: Done uploading piece. err: <nil> Dec 26 21:30:53.852382080 (510h1m53.723465125s) [noise=true , address=145.79.227.45:4229 , piece_id=MDBKNXVJMWZD7XIFK362N3KQ2ZKS6JXDN3ADAGPUGD6EA6JYOCKQ, seg_pos={0 0}, upload=447552]: Done uploading piece. err: <nil> Dec 26 21:30:53.958443018 (510h1m53.829526063s) [noise=true , address=145.79.227.17:4972 , piece_id=ILM4WRBZKYGUKJKRHC7H4ZE2JWXOZTBFPU4HLFQ6LKCDHFH2VJMQ, seg_pos={0 0}, upload=447552]: Done uploading piece. err: <nil> Dec 26 21:30:53.973845728 (510h1m53.844928763s) [noise=true , address=77.90.31.21:4880 , piece_id=L2SXABFCPAQUDQOV47AGQJFFVC3ZVFQIVRFP2BNWE4HUFZFA5EAA, seg_pos={0 0}, upload=447552]: Done uploading piece. err: <nil> Dec 26 21:30:54.019655869 (510h1m53.890738914s) [noise=true , address=77.90.31.13:4627 , piece_id=DM2PWELMEZOYOKQJOI2LFYFQ72QULYJYMDSQL7RJKU4P2JLXUIZA, seg_pos={0 0}, upload=447552]: Done uploading piece. err: <nil> Dec 26 21:30:54.023044822 (510h1m53.894127857s) [noise=true , address=77.90.31.14:4661 , piece_id=XAOMABOK36EBDLUI5NGILIJP5HJ3YTRO7IQTRYBIW4M3WIEFRLIQ, seg_pos={0 0}, upload=447552]: Done uploading piece. err: <nil> Dec 26 21:30:54.038229146 (510h1m53.909312181s) [noise=true , address=45.91.116.4:4939 , piece_id=SAHLWHCM5FID76UL6B7NFLFFJVIS4K6X5T3RFUXTIFQHR4VKAKKQ, seg_pos={0 0}, upload=447552]: Done uploading piece. err: <nil> Dec 26 21:30:54.043525501 (510h1m53.914608536s) [noise=true , address=45.91.116.9:4264 , piece_id=QXVYPKJUVVVMTXMIPFHME4QVZF5SZJ536W5ZT6CLG3Q7TDCRPW2Q, seg_pos={0 0}, upload=447552]: Done uploading piece. err: <nil> Dec 26 21:30:54.043543855 (510h1m53.91462689s) [seg_pos={0 0}, upload=447552]: Segment reached optimal threshold of 13 pieces. Dec 26 21:30:54.043760137 (510h1m53.914843183s) [noise=true , address=145.79.227.40:4050 , piece_id=ZMHHZIYZXT4KII2EUY3NORH5H4YH6WQWLK4S2OHYG3LBMEQIRXUQ, seg_pos={0 0}, upload=447552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 21:30:54.043769977 (510h1m53.914853022s) [noise=true , address=145.79.227.40:4050 , piece_id=ZMHHZIYZXT4KII2EUY3NORH5H4YH6WQWLK4S2OHYG3LBMEQIRXUQ, seg_pos={0 0}, upload=447552]: Long tail cancellation due to optimal threshold Dec 26 21:30:54.043897429 (510h1m53.914980464s) [noise=true , address=145.79.227.43:4159 , piece_id=VLAIPO7BPFNE7DI2L4UKQYUANDHUQVWFH7GI4SXC6KVTN575QXVA, seg_pos={0 0}, upload=447552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 26 21:30:54.043903341 (510h1m53.914986376s) [noise=true , address=145.79.227.43:4159 , piece_id=VLAIPO7BPFNE7DI2L4UKQYUANDHUQVWFH7GI4SXC6KVTN575QXVA, seg_pos={0 0}, upload=447552]: Long tail cancellation due to optimal threshold Dec 26 21:30:54.044298543 (510h1m53.915381578s) [noise=true , address=45.91.116.13:4153 , piece_id=YWDYXI7HO4546TDIWFKEHL7VYQ5O7WZ3XHGNGOBM67EIWO6OTFXQ, seg_pos={0 0}, upload=447552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 26 21:30:54.044318151 (510h1m53.915401186s) [noise=true , address=45.91.116.13:4153 , piece_id=YWDYXI7HO4546TDIWFKEHL7VYQ5O7WZ3XHGNGOBM67EIWO6OTFXQ, seg_pos={0 0}, upload=447552]: Long tail cancellation due to optimal threshold Dec 26 21:30:54.044347435 (510h1m53.915430471s) [seg_pos={0 0}, upload=447552]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 21:30:54.044473566 (510h1m53.915556611s) [upload=447552]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:30:54.044481541 (510h1m53.915564576s) [upload=447552]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:30:54.368560325 (510h1m54.23964336s) [upload=447552]: Done uploading segments... Dec 26 21:30:54.368579281 (510h1m54.239662316s) [upload=447552]: Upload finished. err: <nil> Duration: 9.886808301s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 15.590690876s trace: https://jaeger.dev.storj.io/trace/6d2950019117adaf FAIL (2025-12-26 21:33:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Rj4hUGWvUPuU49QDsHYhibiJhyY5KDGus1rzNeP7KusgyvTsU, Piece ID: NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A) context deadline exceeded; piecestore: (Node ID: 12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, Piece ID: N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ) context deadline exceeded; piecestore: (Node ID: 12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, Piece ID: IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ) context deadline exceeded ======LOGS====== ng piece... Dec 26 20:54:08.087336805 (509h25m7.95841984s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087649520 (509h25m7.958732565s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087988717 (509h25m7.959071752s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088323233 (509h25m7.959406268s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088877028 (509h25m7.959960063s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089247203 (509h25m7.960330238s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089543066 (509h25m7.960626101s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089914133 (509h25m7.960997168s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090327259 (509h25m7.961410294s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090645345 (509h25m7.96172838s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091004318 (509h25m7.962087353s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091338976 (509h25m7.962422021s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091732054 (509h25m7.962815099s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:09.565187143 (509h25m9.436270188s) [upload=447022]: Next returned nil segment Dec 26 20:54:09.565201321 (509h25m9.436284356s) [upload=447022]: Waiting for error group managing segments... Dec 26 20:54:10.137065972 (509h25m10.008149007s) [noise=true , address=77.90.31.10:4523 , piece_id=GQYT3Z53Y6HVTY3IBUNFCIWBEE7KVAUWEJYO7QJFRRAJCHTJMYFA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.137147197 (509h25m10.008230232s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.149325196 (509h25m10.020408242s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.159611673 (509h25m10.030694718s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.164411764 (509h25m10.035494809s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.190924278 (509h25m10.062007313s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.232931086 (509h25m10.104014131s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.241658245 (509h25m10.11274129s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.472574370 (509h25m10.343657405s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.866910566 (509h25m10.737993611s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.886583507 (509h25m10.757666542s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.127773215 (509h25m10.99885626s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.192133217 (509h25m11.063216263s) [noise=true , address=145.79.227.13:4819 , piece_id=BQDRL2QJZ75TAMUM2VSGTF4U3NEMZC4FFQLN6QV6UOAFHWB56MZA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256146219 (509h25m11.127229264s) [noise=true , address=145.79.227.40:4062 , piece_id=IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256176597 (509h25m11.127259632s) [seg_pos={0 0}, upload=447022]: Segment reached optimal threshold of 14 pieces. Dec 26 20:54:11.256428137 (509h25m11.127511192s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98): protocol: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection Dec 26 20:54:11.256439549 (509h25m11.127522584s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256458915 (509h25m11.12754198s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF): protocol: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection Dec 26 20:54:11.256490705 (509h25m11.127573741s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256698171 (509h25m11.127781216s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: rpc: dial tcp 145.79.227.30:4796: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:54:11.256706457 (509h25m11.127789502s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256917309 (509h25m11.128000364s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection Dec 26 20:54:11.256926666 (509h25m11.128009711s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256964869 (509h25m11.128047914s) [seg_pos={0 0}, upload=447022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:54:11.257064138 (509h25m11.128147173s) [upload=447022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:11.257070690 (509h25m11.128153725s) [upload=447022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:11.561962810 (509h25m11.433045875s) [upload=447022]: Done uploading segments... Dec 26 20:54:11.561987427 (509h25m11.433070472s) [upload=447022]: Upload finished. err: <nil> Duration: 15.590720241s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [........X....................XX.XX.X..X.X...XX.X......X..X..] duration: 7.023194525s trace: https://jaeger.dev.storj.io/trace/536c87b2b89ff91b FAIL (2025-12-26 21:32:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 21:30:51.550437270 (510h1m51.421520305s) [upload=447552]: Starting upload Dec 26 21:30:51.550493317 (510h1m51.421576352s) [upload=447552]: Done starting upload Dec 26 21:30:51.550654945 (510h1m51.42173799s) [upload=447552]: Uploading segments... Dec 26 21:30:51.550767770 (510h1m51.421850805s) [upload=447552]: Got next segment. Inline: false Dec 26 21:30:51.550804860 (510h1m51.421887905s) [upload=447552]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 21:30:51.550810190 (510h1m51.421893235s) [upload=447552]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 21:30:51.794426629 (510h1m51.665509674s) [seg_pos={0 0}, upload=447552]: Begin upload segment... Dec 26 21:30:51.794483798 (510h1m51.665566853s) [seg_pos={0 0}, upload=447552]: Done begin upload segment. Dec 26 21:30:51.794495490 (510h1m51.665578525s) [noise=true , address=45.91.116.9:4264 , piece_id=QXVYPKJUVVVMTXMIPFHME4QVZF5SZJ536W5ZT6CLG3Q7TDCRPW2Q, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.794525237 (510h1m51.665608282s) [noise=true , address=77.90.31.14:4661 , piece_id=XAOMABOK36EBDLUI5NGILIJP5HJ3YTRO7IQTRYBIW4M3WIEFRLIQ, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.795292047 (510h1m51.666375082s) [noise=true , address=145.79.227.17:4972 , piece_id=ILM4WRBZKYGUKJKRHC7H4ZE2JWXOZTBFPU4HLFQ6LKCDHFH2VJMQ, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.795781268 (510h1m51.666864303s) [noise=true , address=77.90.31.13:4627 , piece_id=DM2PWELMEZOYOKQJOI2LFYFQ72QULYJYMDSQL7RJKU4P2JLXUIZA, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.796130614 (510h1m51.667213649s) [noise=true , address=145.79.227.11:4732 , piece_id=PFFVB3WH4DBOANMRNBHIGYFLLL35RTMPAYPG7QU5D7VMUCEBD57A, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.796561053 (510h1m51.667644088s) [noise=true , address=145.79.227.40:4050 , piece_id=ZMHHZIYZXT4KII2EUY3NORH5H4YH6WQWLK4S2OHYG3LBMEQIRXUQ, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.796698735 (510h1m51.66778177s) [noise=true , address=145.79.227.52:4446 , piece_id=ZXOPINHEA6WMCX4GREWOAWTXLZAL4FUUATQRJTP62L2TRGWLPKJQ, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.797166186 (510h1m51.668249221s) [noise=true , address=145.79.227.43:4159 , piece_id=VLAIPO7BPFNE7DI2L4UKQYUANDHUQVWFH7GI4SXC6KVTN575QXVA, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.797517014 (510h1m51.668600049s) [noise=true , address=145.79.227.37:4565 , piece_id=2HZYAHWH3BPF4IOP5OHGUFJFK7JYY2RB5ZTJA537SGDABU3DMRBQ, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.797859055 (510h1m51.66894209s) [noise=true , address=145.79.227.47:4265 , piece_id=3PUY42FXCQK62GNKAWIJ57X2KECZU3G5NA56YWRL7C7YADIL662A, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.798190917 (510h1m51.669273952s) [noise=true , address=145.79.227.45:4229 , piece_id=MDBKNXVJMWZD7XIFK362N3KQ2ZKS6JXDN3ADAGPUGD6EA6JYOCKQ, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.798535252 (510h1m51.669618287s) [noise=true , address=145.79.227.46:4601 , piece_id=7YNQC7KH6KUHXRPODLYGY33YXOGGSZ6KAQUEMLSEJVBKNGJBI4VQ, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.798864399 (510h1m51.669947434s) [noise=true , address=45.91.116.13:4153 , piece_id=YWDYXI7HO4546TDIWFKEHL7VYQ5O7WZ3XHGNGOBM67EIWO6OTFXQ, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.799268939 (510h1m51.670351984s) [noise=true , address=145.79.227.9:4365 , piece_id=OBHOCV5YR2XY4EFVUTK3WXEVW7WZXDG3LTGEAWVAUVCPEGLF2FIA, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.800747806 (510h1m51.671830841s) [noise=true , address=45.91.116.4:4939 , piece_id=SAHLWHCM5FID76UL6B7NFLFFJVIS4K6X5T3RFUXTIFQHR4VKAKKQ, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.803663848 (510h1m51.674746883s) [noise=true , address=77.90.31.21:4880 , piece_id=L2SXABFCPAQUDQOV47AGQJFFVC3ZVFQIVRFP2BNWE4HUFZFA5EAA, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:53.429958811 (510h1m53.301041846s) [upload=447552]: Next returned nil segment Dec 26 21:30:53.429972518 (510h1m53.301055563s) [upload=447552]: Waiting for error group managing segments... Dec 26 21:30:53.814614754 (510h1m53.685697789s) [noise=true , address=145.79.227.46:4601 , piece_id=7YNQC7KH6KUHXRPODLYGY33YXOGGSZ6KAQUEMLSEJVBKNGJBI4VQ, seg_pos={0 0}, upload=447552]: Done uploading piece. err: <nil> Dec 26 21:30:53.816217045 (510h1m53.68730008s) [noise=true , address=145.79.227.52:4446 , piece_id=ZXOPINHEA6WMCX4GREWOAWTXLZAL4FUUATQRJTP62L2TRGWLPKJQ, seg_pos={0 0}, upload=447552]: Done uploading piece. err: <nil> Dec 26 21:30:53.825544096 (510h1m53.696627141s) [noise=true , address=145.79.227.37:4565 , piece_id=2HZYAHWH3BPF4IOP5OHGUFJFK7JYY2RB5ZTJA537SGDABU3DMRBQ, seg_pos={0 0}, upload=447552]: Done uploading piece. err: <nil> Dec 26 21:30:53.836843141 (510h1m53.707926187s) [noise=true , address=145.79.227.47:4265 , piece_id=3PUY42FXCQK62GNKAWIJ57X2KECZU3G5NA56YWRL7C7YADIL662A, seg_pos={0 0}, upload=447552]: Done uploading piece. err: <nil> Dec 26 21:30:53.837329888 (510h1m53.708412933s) [noise=true , address=145.79.227.9:4365 , piece_id=OBHOCV5YR2XY4EFVUTK3WXEVW7WZXDG3LTGEAWVAUVCPEGLF2FIA, seg_pos={0 0}, upload=447552]: Done uploading piece. err: <nil> Dec 26 21:30:53.838208611 (510h1m53.709291646s) [noise=true , address=145.79.227.11:4732 , piece_id=PFFVB3WH4DBOANMRNBHIGYFLLL35RTMPAYPG7QU5D7VMUCEBD57A, seg_pos={0 0}, upload=447552]: Done uploading piece. err: <nil> Dec 26 21:30:53.852382080 (510h1m53.723465125s) [noise=true , address=145.79.227.45:4229 , piece_id=MDBKNXVJMWZD7XIFK362N3KQ2ZKS6JXDN3ADAGPUGD6EA6JYOCKQ, seg_pos={0 0}, upload=447552]: Done uploading piece. err: <nil> Dec 26 21:30:53.958443018 (510h1m53.829526063s) [noise=true , address=145.79.227.17:4972 , piece_id=ILM4WRBZKYGUKJKRHC7H4ZE2JWXOZTBFPU4HLFQ6LKCDHFH2VJMQ, seg_pos={0 0}, upload=447552]: Done uploading piece. err: <nil> Dec 26 21:30:53.973845728 (510h1m53.844928763s) [noise=true , address=77.90.31.21:4880 , piece_id=L2SXABFCPAQUDQOV47AGQJFFVC3ZVFQIVRFP2BNWE4HUFZFA5EAA, seg_pos={0 0}, upload=447552]: Done uploading piece. err: <nil> Dec 26 21:30:54.019655869 (510h1m53.890738914s) [noise=true , address=77.90.31.13:4627 , piece_id=DM2PWELMEZOYOKQJOI2LFYFQ72QULYJYMDSQL7RJKU4P2JLXUIZA, seg_pos={0 0}, upload=447552]: Done uploading piece. err: <nil> Dec 26 21:30:54.023044822 (510h1m53.894127857s) [noise=true , address=77.90.31.14:4661 , piece_id=XAOMABOK36EBDLUI5NGILIJP5HJ3YTRO7IQTRYBIW4M3WIEFRLIQ, seg_pos={0 0}, upload=447552]: Done uploading piece. err: <nil> Dec 26 21:30:54.038229146 (510h1m53.909312181s) [noise=true , address=45.91.116.4:4939 , piece_id=SAHLWHCM5FID76UL6B7NFLFFJVIS4K6X5T3RFUXTIFQHR4VKAKKQ, seg_pos={0 0}, upload=447552]: Done uploading piece. err: <nil> Dec 26 21:30:54.043525501 (510h1m53.914608536s) [noise=true , address=45.91.116.9:4264 , piece_id=QXVYPKJUVVVMTXMIPFHME4QVZF5SZJ536W5ZT6CLG3Q7TDCRPW2Q, seg_pos={0 0}, upload=447552]: Done uploading piece. err: <nil> Dec 26 21:30:54.043543855 (510h1m53.91462689s) [seg_pos={0 0}, upload=447552]: Segment reached optimal threshold of 13 pieces. Dec 26 21:30:54.043760137 (510h1m53.914843183s) [noise=true , address=145.79.227.40:4050 , piece_id=ZMHHZIYZXT4KII2EUY3NORH5H4YH6WQWLK4S2OHYG3LBMEQIRXUQ, seg_pos={0 0}, upload=447552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 21:30:54.043769977 (510h1m53.914853022s) [noise=true , address=145.79.227.40:4050 , piece_id=ZMHHZIYZXT4KII2EUY3NORH5H4YH6WQWLK4S2OHYG3LBMEQIRXUQ, seg_pos={0 0}, upload=447552]: Long tail cancellation due to optimal threshold Dec 26 21:30:54.043897429 (510h1m53.914980464s) [noise=true , address=145.79.227.43:4159 , piece_id=VLAIPO7BPFNE7DI2L4UKQYUANDHUQVWFH7GI4SXC6KVTN575QXVA, seg_pos={0 0}, upload=447552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 26 21:30:54.043903341 (510h1m53.914986376s) [noise=true , address=145.79.227.43:4159 , piece_id=VLAIPO7BPFNE7DI2L4UKQYUANDHUQVWFH7GI4SXC6KVTN575QXVA, seg_pos={0 0}, upload=447552]: Long tail cancellation due to optimal threshold Dec 26 21:30:54.044298543 (510h1m53.915381578s) [noise=true , address=45.91.116.13:4153 , piece_id=YWDYXI7HO4546TDIWFKEHL7VYQ5O7WZ3XHGNGOBM67EIWO6OTFXQ, seg_pos={0 0}, upload=447552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 26 21:30:54.044318151 (510h1m53.915401186s) [noise=true , address=45.91.116.13:4153 , piece_id=YWDYXI7HO4546TDIWFKEHL7VYQ5O7WZ3XHGNGOBM67EIWO6OTFXQ, seg_pos={0 0}, upload=447552]: Long tail cancellation due to optimal threshold Dec 26 21:30:54.044347435 (510h1m53.915430471s) [seg_pos={0 0}, upload=447552]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 21:30:54.044473566 (510h1m53.915556611s) [upload=447552]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:30:54.044481541 (510h1m53.915564576s) [upload=447552]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:30:54.368560325 (510h1m54.23964336s) [upload=447552]: Done uploading segments... Dec 26 21:30:54.368579281 (510h1m54.239662316s) [upload=447552]: Upload finished. err: <nil> Duration: 7.023220674s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 28.018520113s trace: https://jaeger.dev.storj.io/trace/6d2950019117adaf FAIL (2025-12-26 21:32:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Rj4hUGWvUPuU49QDsHYhibiJhyY5KDGus1rzNeP7KusgyvTsU, Piece ID: NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A) context deadline exceeded; piecestore: (Node ID: 12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, Piece ID: N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ) context deadline exceeded; piecestore: (Node ID: 12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, Piece ID: IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ) context deadline exceeded ======LOGS====== ng piece... Dec 26 20:54:08.087336805 (509h25m7.95841984s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087649520 (509h25m7.958732565s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087988717 (509h25m7.959071752s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088323233 (509h25m7.959406268s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088877028 (509h25m7.959960063s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089247203 (509h25m7.960330238s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089543066 (509h25m7.960626101s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089914133 (509h25m7.960997168s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090327259 (509h25m7.961410294s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090645345 (509h25m7.96172838s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091004318 (509h25m7.962087353s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091338976 (509h25m7.962422021s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091732054 (509h25m7.962815099s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:09.565187143 (509h25m9.436270188s) [upload=447022]: Next returned nil segment Dec 26 20:54:09.565201321 (509h25m9.436284356s) [upload=447022]: Waiting for error group managing segments... Dec 26 20:54:10.137065972 (509h25m10.008149007s) [noise=true , address=77.90.31.10:4523 , piece_id=GQYT3Z53Y6HVTY3IBUNFCIWBEE7KVAUWEJYO7QJFRRAJCHTJMYFA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.137147197 (509h25m10.008230232s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.149325196 (509h25m10.020408242s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.159611673 (509h25m10.030694718s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.164411764 (509h25m10.035494809s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.190924278 (509h25m10.062007313s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.232931086 (509h25m10.104014131s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.241658245 (509h25m10.11274129s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.472574370 (509h25m10.343657405s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.866910566 (509h25m10.737993611s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.886583507 (509h25m10.757666542s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.127773215 (509h25m10.99885626s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.192133217 (509h25m11.063216263s) [noise=true , address=145.79.227.13:4819 , piece_id=BQDRL2QJZ75TAMUM2VSGTF4U3NEMZC4FFQLN6QV6UOAFHWB56MZA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256146219 (509h25m11.127229264s) [noise=true , address=145.79.227.40:4062 , piece_id=IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256176597 (509h25m11.127259632s) [seg_pos={0 0}, upload=447022]: Segment reached optimal threshold of 14 pieces. Dec 26 20:54:11.256428137 (509h25m11.127511192s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98): protocol: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection Dec 26 20:54:11.256439549 (509h25m11.127522584s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256458915 (509h25m11.12754198s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF): protocol: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection Dec 26 20:54:11.256490705 (509h25m11.127573741s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256698171 (509h25m11.127781216s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: rpc: dial tcp 145.79.227.30:4796: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:54:11.256706457 (509h25m11.127789502s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256917309 (509h25m11.128000364s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection Dec 26 20:54:11.256926666 (509h25m11.128009711s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256964869 (509h25m11.128047914s) [seg_pos={0 0}, upload=447022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:54:11.257064138 (509h25m11.128147173s) [upload=447022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:11.257070690 (509h25m11.128153725s) [upload=447022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:11.561962810 (509h25m11.433045875s) [upload=447022]: Done uploading segments... Dec 26 20:54:11.561987427 (509h25m11.433070472s) [upload=447022]: Upload finished. err: <nil> Duration: 28.018549408s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.........X....................XX.XX.X..X.X...XX.X......X..X.] duration: 24.82422193s trace: https://jaeger.dev.storj.io/trace/536c87b2b89ff91b FAIL (2025-12-26 21:31:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 21:30:51.550437270 (510h1m51.421520305s) [upload=447552]: Starting upload Dec 26 21:30:51.550493317 (510h1m51.421576352s) [upload=447552]: Done starting upload Dec 26 21:30:51.550654945 (510h1m51.42173799s) [upload=447552]: Uploading segments... Dec 26 21:30:51.550767770 (510h1m51.421850805s) [upload=447552]: Got next segment. Inline: false Dec 26 21:30:51.550804860 (510h1m51.421887905s) [upload=447552]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 21:30:51.550810190 (510h1m51.421893235s) [upload=447552]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 21:30:51.794426629 (510h1m51.665509674s) [seg_pos={0 0}, upload=447552]: Begin upload segment... Dec 26 21:30:51.794483798 (510h1m51.665566853s) [seg_pos={0 0}, upload=447552]: Done begin upload segment. Dec 26 21:30:51.794495490 (510h1m51.665578525s) [noise=true , address=45.91.116.9:4264 , piece_id=QXVYPKJUVVVMTXMIPFHME4QVZF5SZJ536W5ZT6CLG3Q7TDCRPW2Q, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.794525237 (510h1m51.665608282s) [noise=true , address=77.90.31.14:4661 , piece_id=XAOMABOK36EBDLUI5NGILIJP5HJ3YTRO7IQTRYBIW4M3WIEFRLIQ, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.795292047 (510h1m51.666375082s) [noise=true , address=145.79.227.17:4972 , piece_id=ILM4WRBZKYGUKJKRHC7H4ZE2JWXOZTBFPU4HLFQ6LKCDHFH2VJMQ, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.795781268 (510h1m51.666864303s) [noise=true , address=77.90.31.13:4627 , piece_id=DM2PWELMEZOYOKQJOI2LFYFQ72QULYJYMDSQL7RJKU4P2JLXUIZA, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.796130614 (510h1m51.667213649s) [noise=true , address=145.79.227.11:4732 , piece_id=PFFVB3WH4DBOANMRNBHIGYFLLL35RTMPAYPG7QU5D7VMUCEBD57A, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.796561053 (510h1m51.667644088s) [noise=true , address=145.79.227.40:4050 , piece_id=ZMHHZIYZXT4KII2EUY3NORH5H4YH6WQWLK4S2OHYG3LBMEQIRXUQ, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.796698735 (510h1m51.66778177s) [noise=true , address=145.79.227.52:4446 , piece_id=ZXOPINHEA6WMCX4GREWOAWTXLZAL4FUUATQRJTP62L2TRGWLPKJQ, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.797166186 (510h1m51.668249221s) [noise=true , address=145.79.227.43:4159 , piece_id=VLAIPO7BPFNE7DI2L4UKQYUANDHUQVWFH7GI4SXC6KVTN575QXVA, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.797517014 (510h1m51.668600049s) [noise=true , address=145.79.227.37:4565 , piece_id=2HZYAHWH3BPF4IOP5OHGUFJFK7JYY2RB5ZTJA537SGDABU3DMRBQ, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.797859055 (510h1m51.66894209s) [noise=true , address=145.79.227.47:4265 , piece_id=3PUY42FXCQK62GNKAWIJ57X2KECZU3G5NA56YWRL7C7YADIL662A, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.798190917 (510h1m51.669273952s) [noise=true , address=145.79.227.45:4229 , piece_id=MDBKNXVJMWZD7XIFK362N3KQ2ZKS6JXDN3ADAGPUGD6EA6JYOCKQ, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.798535252 (510h1m51.669618287s) [noise=true , address=145.79.227.46:4601 , piece_id=7YNQC7KH6KUHXRPODLYGY33YXOGGSZ6KAQUEMLSEJVBKNGJBI4VQ, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.798864399 (510h1m51.669947434s) [noise=true , address=45.91.116.13:4153 , piece_id=YWDYXI7HO4546TDIWFKEHL7VYQ5O7WZ3XHGNGOBM67EIWO6OTFXQ, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.799268939 (510h1m51.670351984s) [noise=true , address=145.79.227.9:4365 , piece_id=OBHOCV5YR2XY4EFVUTK3WXEVW7WZXDG3LTGEAWVAUVCPEGLF2FIA, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.800747806 (510h1m51.671830841s) [noise=true , address=45.91.116.4:4939 , piece_id=SAHLWHCM5FID76UL6B7NFLFFJVIS4K6X5T3RFUXTIFQHR4VKAKKQ, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.803663848 (510h1m51.674746883s) [noise=true , address=77.90.31.21:4880 , piece_id=L2SXABFCPAQUDQOV47AGQJFFVC3ZVFQIVRFP2BNWE4HUFZFA5EAA, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:53.429958811 (510h1m53.301041846s) [upload=447552]: Next returned nil segment Dec 26 21:30:53.429972518 (510h1m53.301055563s) [upload=447552]: Waiting for error group managing segments... Dec 26 21:30:53.814614754 (510h1m53.685697789s) [noise=true , address=145.79.227.46:4601 , piece_id=7YNQC7KH6KUHXRPODLYGY33YXOGGSZ6KAQUEMLSEJVBKNGJBI4VQ, seg_pos={0 0}, upload=447552]: Done uploading piece. err: <nil> Dec 26 21:30:53.816217045 (510h1m53.68730008s) [noise=true , address=145.79.227.52:4446 , piece_id=ZXOPINHEA6WMCX4GREWOAWTXLZAL4FUUATQRJTP62L2TRGWLPKJQ, seg_pos={0 0}, upload=447552]: Done uploading piece. err: <nil> Dec 26 21:30:53.825544096 (510h1m53.696627141s) [noise=true , address=145.79.227.37:4565 , piece_id=2HZYAHWH3BPF4IOP5OHGUFJFK7JYY2RB5ZTJA537SGDABU3DMRBQ, seg_pos={0 0}, upload=447552]: Done uploading piece. err: <nil> Dec 26 21:30:53.836843141 (510h1m53.707926187s) [noise=true , address=145.79.227.47:4265 , piece_id=3PUY42FXCQK62GNKAWIJ57X2KECZU3G5NA56YWRL7C7YADIL662A, seg_pos={0 0}, upload=447552]: Done uploading piece. err: <nil> Dec 26 21:30:53.837329888 (510h1m53.708412933s) [noise=true , address=145.79.227.9:4365 , piece_id=OBHOCV5YR2XY4EFVUTK3WXEVW7WZXDG3LTGEAWVAUVCPEGLF2FIA, seg_pos={0 0}, upload=447552]: Done uploading piece. err: <nil> Dec 26 21:30:53.838208611 (510h1m53.709291646s) [noise=true , address=145.79.227.11:4732 , piece_id=PFFVB3WH4DBOANMRNBHIGYFLLL35RTMPAYPG7QU5D7VMUCEBD57A, seg_pos={0 0}, upload=447552]: Done uploading piece. err: <nil> Dec 26 21:30:53.852382080 (510h1m53.723465125s) [noise=true , address=145.79.227.45:4229 , piece_id=MDBKNXVJMWZD7XIFK362N3KQ2ZKS6JXDN3ADAGPUGD6EA6JYOCKQ, seg_pos={0 0}, upload=447552]: Done uploading piece. err: <nil> Dec 26 21:30:53.958443018 (510h1m53.829526063s) [noise=true , address=145.79.227.17:4972 , piece_id=ILM4WRBZKYGUKJKRHC7H4ZE2JWXOZTBFPU4HLFQ6LKCDHFH2VJMQ, seg_pos={0 0}, upload=447552]: Done uploading piece. err: <nil> Dec 26 21:30:53.973845728 (510h1m53.844928763s) [noise=true , address=77.90.31.21:4880 , piece_id=L2SXABFCPAQUDQOV47AGQJFFVC3ZVFQIVRFP2BNWE4HUFZFA5EAA, seg_pos={0 0}, upload=447552]: Done uploading piece. err: <nil> Dec 26 21:30:54.019655869 (510h1m53.890738914s) [noise=true , address=77.90.31.13:4627 , piece_id=DM2PWELMEZOYOKQJOI2LFYFQ72QULYJYMDSQL7RJKU4P2JLXUIZA, seg_pos={0 0}, upload=447552]: Done uploading piece. err: <nil> Dec 26 21:30:54.023044822 (510h1m53.894127857s) [noise=true , address=77.90.31.14:4661 , piece_id=XAOMABOK36EBDLUI5NGILIJP5HJ3YTRO7IQTRYBIW4M3WIEFRLIQ, seg_pos={0 0}, upload=447552]: Done uploading piece. err: <nil> Dec 26 21:30:54.038229146 (510h1m53.909312181s) [noise=true , address=45.91.116.4:4939 , piece_id=SAHLWHCM5FID76UL6B7NFLFFJVIS4K6X5T3RFUXTIFQHR4VKAKKQ, seg_pos={0 0}, upload=447552]: Done uploading piece. err: <nil> Dec 26 21:30:54.043525501 (510h1m53.914608536s) [noise=true , address=45.91.116.9:4264 , piece_id=QXVYPKJUVVVMTXMIPFHME4QVZF5SZJ536W5ZT6CLG3Q7TDCRPW2Q, seg_pos={0 0}, upload=447552]: Done uploading piece. err: <nil> Dec 26 21:30:54.043543855 (510h1m53.91462689s) [seg_pos={0 0}, upload=447552]: Segment reached optimal threshold of 13 pieces. Dec 26 21:30:54.043760137 (510h1m53.914843183s) [noise=true , address=145.79.227.40:4050 , piece_id=ZMHHZIYZXT4KII2EUY3NORH5H4YH6WQWLK4S2OHYG3LBMEQIRXUQ, seg_pos={0 0}, upload=447552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 21:30:54.043769977 (510h1m53.914853022s) [noise=true , address=145.79.227.40:4050 , piece_id=ZMHHZIYZXT4KII2EUY3NORH5H4YH6WQWLK4S2OHYG3LBMEQIRXUQ, seg_pos={0 0}, upload=447552]: Long tail cancellation due to optimal threshold Dec 26 21:30:54.043897429 (510h1m53.914980464s) [noise=true , address=145.79.227.43:4159 , piece_id=VLAIPO7BPFNE7DI2L4UKQYUANDHUQVWFH7GI4SXC6KVTN575QXVA, seg_pos={0 0}, upload=447552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 26 21:30:54.043903341 (510h1m53.914986376s) [noise=true , address=145.79.227.43:4159 , piece_id=VLAIPO7BPFNE7DI2L4UKQYUANDHUQVWFH7GI4SXC6KVTN575QXVA, seg_pos={0 0}, upload=447552]: Long tail cancellation due to optimal threshold Dec 26 21:30:54.044298543 (510h1m53.915381578s) [noise=true , address=45.91.116.13:4153 , piece_id=YWDYXI7HO4546TDIWFKEHL7VYQ5O7WZ3XHGNGOBM67EIWO6OTFXQ, seg_pos={0 0}, upload=447552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 26 21:30:54.044318151 (510h1m53.915401186s) [noise=true , address=45.91.116.13:4153 , piece_id=YWDYXI7HO4546TDIWFKEHL7VYQ5O7WZ3XHGNGOBM67EIWO6OTFXQ, seg_pos={0 0}, upload=447552]: Long tail cancellation due to optimal threshold Dec 26 21:30:54.044347435 (510h1m53.915430471s) [seg_pos={0 0}, upload=447552]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 21:30:54.044473566 (510h1m53.915556611s) [upload=447552]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:30:54.044481541 (510h1m53.915564576s) [upload=447552]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:30:54.368560325 (510h1m54.23964336s) [upload=447552]: Done uploading segments... Dec 26 21:30:54.368579281 (510h1m54.239662316s) [upload=447552]: Upload finished. err: <nil> Duration: 24.824267196s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 15.300428975s trace: https://jaeger.dev.storj.io/trace/6d2950019117adaf FAIL (2025-12-26 21:31:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Rj4hUGWvUPuU49QDsHYhibiJhyY5KDGus1rzNeP7KusgyvTsU, Piece ID: NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A) context deadline exceeded; piecestore: (Node ID: 12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, Piece ID: N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ) context deadline exceeded; piecestore: (Node ID: 12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, Piece ID: IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ) context deadline exceeded ======LOGS====== ng piece... Dec 26 20:54:08.087336805 (509h25m7.95841984s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087649520 (509h25m7.958732565s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087988717 (509h25m7.959071752s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088323233 (509h25m7.959406268s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088877028 (509h25m7.959960063s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089247203 (509h25m7.960330238s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089543066 (509h25m7.960626101s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089914133 (509h25m7.960997168s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090327259 (509h25m7.961410294s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090645345 (509h25m7.96172838s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091004318 (509h25m7.962087353s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091338976 (509h25m7.962422021s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091732054 (509h25m7.962815099s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:09.565187143 (509h25m9.436270188s) [upload=447022]: Next returned nil segment Dec 26 20:54:09.565201321 (509h25m9.436284356s) [upload=447022]: Waiting for error group managing segments... Dec 26 20:54:10.137065972 (509h25m10.008149007s) [noise=true , address=77.90.31.10:4523 , piece_id=GQYT3Z53Y6HVTY3IBUNFCIWBEE7KVAUWEJYO7QJFRRAJCHTJMYFA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.137147197 (509h25m10.008230232s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.149325196 (509h25m10.020408242s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.159611673 (509h25m10.030694718s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.164411764 (509h25m10.035494809s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.190924278 (509h25m10.062007313s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.232931086 (509h25m10.104014131s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.241658245 (509h25m10.11274129s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.472574370 (509h25m10.343657405s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.866910566 (509h25m10.737993611s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.886583507 (509h25m10.757666542s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.127773215 (509h25m10.99885626s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.192133217 (509h25m11.063216263s) [noise=true , address=145.79.227.13:4819 , piece_id=BQDRL2QJZ75TAMUM2VSGTF4U3NEMZC4FFQLN6QV6UOAFHWB56MZA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256146219 (509h25m11.127229264s) [noise=true , address=145.79.227.40:4062 , piece_id=IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256176597 (509h25m11.127259632s) [seg_pos={0 0}, upload=447022]: Segment reached optimal threshold of 14 pieces. Dec 26 20:54:11.256428137 (509h25m11.127511192s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98): protocol: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection Dec 26 20:54:11.256439549 (509h25m11.127522584s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256458915 (509h25m11.12754198s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF): protocol: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection Dec 26 20:54:11.256490705 (509h25m11.127573741s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256698171 (509h25m11.127781216s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: rpc: dial tcp 145.79.227.30:4796: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:54:11.256706457 (509h25m11.127789502s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256917309 (509h25m11.128000364s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection Dec 26 20:54:11.256926666 (509h25m11.128009711s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256964869 (509h25m11.128047914s) [seg_pos={0 0}, upload=447022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:54:11.257064138 (509h25m11.128147173s) [upload=447022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:11.257070690 (509h25m11.128153725s) [upload=447022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:11.561962810 (509h25m11.433045875s) [upload=447022]: Done uploading segments... Dec 26 20:54:11.561987427 (509h25m11.433070472s) [upload=447022]: Upload finished. err: <nil> Duration: 15.300444704s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 31.850105123s trace: https://jaeger.dev.storj.io/trace/6d2950019117adaf FAIL (2025-12-26 21:31:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Rj4hUGWvUPuU49QDsHYhibiJhyY5KDGus1rzNeP7KusgyvTsU, Piece ID: NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A) context deadline exceeded; piecestore: (Node ID: 12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, Piece ID: N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ) context deadline exceeded; piecestore: (Node ID: 12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, Piece ID: IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ) context deadline exceeded ======LOGS====== ng piece... Dec 26 20:54:08.087336805 (509h25m7.95841984s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087649520 (509h25m7.958732565s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087988717 (509h25m7.959071752s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088323233 (509h25m7.959406268s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088877028 (509h25m7.959960063s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089247203 (509h25m7.960330238s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089543066 (509h25m7.960626101s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089914133 (509h25m7.960997168s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090327259 (509h25m7.961410294s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090645345 (509h25m7.96172838s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091004318 (509h25m7.962087353s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091338976 (509h25m7.962422021s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091732054 (509h25m7.962815099s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:09.565187143 (509h25m9.436270188s) [upload=447022]: Next returned nil segment Dec 26 20:54:09.565201321 (509h25m9.436284356s) [upload=447022]: Waiting for error group managing segments... Dec 26 20:54:10.137065972 (509h25m10.008149007s) [noise=true , address=77.90.31.10:4523 , piece_id=GQYT3Z53Y6HVTY3IBUNFCIWBEE7KVAUWEJYO7QJFRRAJCHTJMYFA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.137147197 (509h25m10.008230232s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.149325196 (509h25m10.020408242s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.159611673 (509h25m10.030694718s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.164411764 (509h25m10.035494809s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.190924278 (509h25m10.062007313s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.232931086 (509h25m10.104014131s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.241658245 (509h25m10.11274129s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.472574370 (509h25m10.343657405s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.866910566 (509h25m10.737993611s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.886583507 (509h25m10.757666542s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.127773215 (509h25m10.99885626s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.192133217 (509h25m11.063216263s) [noise=true , address=145.79.227.13:4819 , piece_id=BQDRL2QJZ75TAMUM2VSGTF4U3NEMZC4FFQLN6QV6UOAFHWB56MZA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256146219 (509h25m11.127229264s) [noise=true , address=145.79.227.40:4062 , piece_id=IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256176597 (509h25m11.127259632s) [seg_pos={0 0}, upload=447022]: Segment reached optimal threshold of 14 pieces. Dec 26 20:54:11.256428137 (509h25m11.127511192s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98): protocol: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection Dec 26 20:54:11.256439549 (509h25m11.127522584s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256458915 (509h25m11.12754198s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF): protocol: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection Dec 26 20:54:11.256490705 (509h25m11.127573741s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256698171 (509h25m11.127781216s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: rpc: dial tcp 145.79.227.30:4796: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:54:11.256706457 (509h25m11.127789502s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256917309 (509h25m11.128000364s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection Dec 26 20:54:11.256926666 (509h25m11.128009711s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256964869 (509h25m11.128047914s) [seg_pos={0 0}, upload=447022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:54:11.257064138 (509h25m11.128147173s) [upload=447022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:11.257070690 (509h25m11.128153725s) [upload=447022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:11.561962810 (509h25m11.433045875s) [upload=447022]: Done uploading segments... Dec 26 20:54:11.561987427 (509h25m11.433070472s) [upload=447022]: Upload finished. err: <nil> Duration: 31.850130322s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..........X....................XX.XX.X..X.X...XX.X......X..X] duration: 30.002551173s trace: https://jaeger.dev.storj.io/trace/536c87b2b89ff91b FAIL (2025-12-26 21:31:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 21:30:51.550437270 (510h1m51.421520305s) [upload=447552]: Starting upload Dec 26 21:30:51.550493317 (510h1m51.421576352s) [upload=447552]: Done starting upload Dec 26 21:30:51.550654945 (510h1m51.42173799s) [upload=447552]: Uploading segments... Dec 26 21:30:51.550767770 (510h1m51.421850805s) [upload=447552]: Got next segment. Inline: false Dec 26 21:30:51.550804860 (510h1m51.421887905s) [upload=447552]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 21:30:51.550810190 (510h1m51.421893235s) [upload=447552]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 21:30:51.794426629 (510h1m51.665509674s) [seg_pos={0 0}, upload=447552]: Begin upload segment... Dec 26 21:30:51.794483798 (510h1m51.665566853s) [seg_pos={0 0}, upload=447552]: Done begin upload segment. Dec 26 21:30:51.794495490 (510h1m51.665578525s) [noise=true , address=45.91.116.9:4264 , piece_id=QXVYPKJUVVVMTXMIPFHME4QVZF5SZJ536W5ZT6CLG3Q7TDCRPW2Q, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.794525237 (510h1m51.665608282s) [noise=true , address=77.90.31.14:4661 , piece_id=XAOMABOK36EBDLUI5NGILIJP5HJ3YTRO7IQTRYBIW4M3WIEFRLIQ, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.795292047 (510h1m51.666375082s) [noise=true , address=145.79.227.17:4972 , piece_id=ILM4WRBZKYGUKJKRHC7H4ZE2JWXOZTBFPU4HLFQ6LKCDHFH2VJMQ, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.795781268 (510h1m51.666864303s) [noise=true , address=77.90.31.13:4627 , piece_id=DM2PWELMEZOYOKQJOI2LFYFQ72QULYJYMDSQL7RJKU4P2JLXUIZA, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.796130614 (510h1m51.667213649s) [noise=true , address=145.79.227.11:4732 , piece_id=PFFVB3WH4DBOANMRNBHIGYFLLL35RTMPAYPG7QU5D7VMUCEBD57A, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.796561053 (510h1m51.667644088s) [noise=true , address=145.79.227.40:4050 , piece_id=ZMHHZIYZXT4KII2EUY3NORH5H4YH6WQWLK4S2OHYG3LBMEQIRXUQ, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.796698735 (510h1m51.66778177s) [noise=true , address=145.79.227.52:4446 , piece_id=ZXOPINHEA6WMCX4GREWOAWTXLZAL4FUUATQRJTP62L2TRGWLPKJQ, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.797166186 (510h1m51.668249221s) [noise=true , address=145.79.227.43:4159 , piece_id=VLAIPO7BPFNE7DI2L4UKQYUANDHUQVWFH7GI4SXC6KVTN575QXVA, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.797517014 (510h1m51.668600049s) [noise=true , address=145.79.227.37:4565 , piece_id=2HZYAHWH3BPF4IOP5OHGUFJFK7JYY2RB5ZTJA537SGDABU3DMRBQ, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.797859055 (510h1m51.66894209s) [noise=true , address=145.79.227.47:4265 , piece_id=3PUY42FXCQK62GNKAWIJ57X2KECZU3G5NA56YWRL7C7YADIL662A, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.798190917 (510h1m51.669273952s) [noise=true , address=145.79.227.45:4229 , piece_id=MDBKNXVJMWZD7XIFK362N3KQ2ZKS6JXDN3ADAGPUGD6EA6JYOCKQ, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.798535252 (510h1m51.669618287s) [noise=true , address=145.79.227.46:4601 , piece_id=7YNQC7KH6KUHXRPODLYGY33YXOGGSZ6KAQUEMLSEJVBKNGJBI4VQ, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.798864399 (510h1m51.669947434s) [noise=true , address=45.91.116.13:4153 , piece_id=YWDYXI7HO4546TDIWFKEHL7VYQ5O7WZ3XHGNGOBM67EIWO6OTFXQ, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.799268939 (510h1m51.670351984s) [noise=true , address=145.79.227.9:4365 , piece_id=OBHOCV5YR2XY4EFVUTK3WXEVW7WZXDG3LTGEAWVAUVCPEGLF2FIA, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.800747806 (510h1m51.671830841s) [noise=true , address=45.91.116.4:4939 , piece_id=SAHLWHCM5FID76UL6B7NFLFFJVIS4K6X5T3RFUXTIFQHR4VKAKKQ, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:51.803663848 (510h1m51.674746883s) [noise=true , address=77.90.31.21:4880 , piece_id=L2SXABFCPAQUDQOV47AGQJFFVC3ZVFQIVRFP2BNWE4HUFZFA5EAA, seg_pos={0 0}, upload=447552]: Uploading piece... Dec 26 21:30:53.429958811 (510h1m53.301041846s) [upload=447552]: Next returned nil segment Dec 26 21:30:53.429972518 (510h1m53.301055563s) [upload=447552]: Waiting for error group managing segments... Dec 26 21:30:53.814614754 (510h1m53.685697789s) [noise=true , address=145.79.227.46:4601 , piece_id=7YNQC7KH6KUHXRPODLYGY33YXOGGSZ6KAQUEMLSEJVBKNGJBI4VQ, seg_pos={0 0}, upload=447552]: Done uploading piece. err: <nil> Dec 26 21:30:53.816217045 (510h1m53.68730008s) [noise=true , address=145.79.227.52:4446 , piece_id=ZXOPINHEA6WMCX4GREWOAWTXLZAL4FUUATQRJTP62L2TRGWLPKJQ, seg_pos={0 0}, upload=447552]: Done uploading piece. err: <nil> Dec 26 21:30:53.825544096 (510h1m53.696627141s) [noise=true , address=145.79.227.37:4565 , piece_id=2HZYAHWH3BPF4IOP5OHGUFJFK7JYY2RB5ZTJA537SGDABU3DMRBQ, seg_pos={0 0}, upload=447552]: Done uploading piece. err: <nil> Dec 26 21:30:53.836843141 (510h1m53.707926187s) [noise=true , address=145.79.227.47:4265 , piece_id=3PUY42FXCQK62GNKAWIJ57X2KECZU3G5NA56YWRL7C7YADIL662A, seg_pos={0 0}, upload=447552]: Done uploading piece. err: <nil> Dec 26 21:30:53.837329888 (510h1m53.708412933s) [noise=true , address=145.79.227.9:4365 , piece_id=OBHOCV5YR2XY4EFVUTK3WXEVW7WZXDG3LTGEAWVAUVCPEGLF2FIA, seg_pos={0 0}, upload=447552]: Done uploading piece. err: <nil> Dec 26 21:30:53.838208611 (510h1m53.709291646s) [noise=true , address=145.79.227.11:4732 , piece_id=PFFVB3WH4DBOANMRNBHIGYFLLL35RTMPAYPG7QU5D7VMUCEBD57A, seg_pos={0 0}, upload=447552]: Done uploading piece. err: <nil> Dec 26 21:30:53.852382080 (510h1m53.723465125s) [noise=true , address=145.79.227.45:4229 , piece_id=MDBKNXVJMWZD7XIFK362N3KQ2ZKS6JXDN3ADAGPUGD6EA6JYOCKQ, seg_pos={0 0}, upload=447552]: Done uploading piece. err: <nil> Dec 26 21:30:53.958443018 (510h1m53.829526063s) [noise=true , address=145.79.227.17:4972 , piece_id=ILM4WRBZKYGUKJKRHC7H4ZE2JWXOZTBFPU4HLFQ6LKCDHFH2VJMQ, seg_pos={0 0}, upload=447552]: Done uploading piece. err: <nil> Dec 26 21:30:53.973845728 (510h1m53.844928763s) [noise=true , address=77.90.31.21:4880 , piece_id=L2SXABFCPAQUDQOV47AGQJFFVC3ZVFQIVRFP2BNWE4HUFZFA5EAA, seg_pos={0 0}, upload=447552]: Done uploading piece. err: <nil> Dec 26 21:30:54.019655869 (510h1m53.890738914s) [noise=true , address=77.90.31.13:4627 , piece_id=DM2PWELMEZOYOKQJOI2LFYFQ72QULYJYMDSQL7RJKU4P2JLXUIZA, seg_pos={0 0}, upload=447552]: Done uploading piece. err: <nil> Dec 26 21:30:54.023044822 (510h1m53.894127857s) [noise=true , address=77.90.31.14:4661 , piece_id=XAOMABOK36EBDLUI5NGILIJP5HJ3YTRO7IQTRYBIW4M3WIEFRLIQ, seg_pos={0 0}, upload=447552]: Done uploading piece. err: <nil> Dec 26 21:30:54.038229146 (510h1m53.909312181s) [noise=true , address=45.91.116.4:4939 , piece_id=SAHLWHCM5FID76UL6B7NFLFFJVIS4K6X5T3RFUXTIFQHR4VKAKKQ, seg_pos={0 0}, upload=447552]: Done uploading piece. err: <nil> Dec 26 21:30:54.043525501 (510h1m53.914608536s) [noise=true , address=45.91.116.9:4264 , piece_id=QXVYPKJUVVVMTXMIPFHME4QVZF5SZJ536W5ZT6CLG3Q7TDCRPW2Q, seg_pos={0 0}, upload=447552]: Done uploading piece. err: <nil> Dec 26 21:30:54.043543855 (510h1m53.91462689s) [seg_pos={0 0}, upload=447552]: Segment reached optimal threshold of 13 pieces. Dec 26 21:30:54.043760137 (510h1m53.914843183s) [noise=true , address=145.79.227.40:4050 , piece_id=ZMHHZIYZXT4KII2EUY3NORH5H4YH6WQWLK4S2OHYG3LBMEQIRXUQ, seg_pos={0 0}, upload=447552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 21:30:54.043769977 (510h1m53.914853022s) [noise=true , address=145.79.227.40:4050 , piece_id=ZMHHZIYZXT4KII2EUY3NORH5H4YH6WQWLK4S2OHYG3LBMEQIRXUQ, seg_pos={0 0}, upload=447552]: Long tail cancellation due to optimal threshold Dec 26 21:30:54.043897429 (510h1m53.914980464s) [noise=true , address=145.79.227.43:4159 , piece_id=VLAIPO7BPFNE7DI2L4UKQYUANDHUQVWFH7GI4SXC6KVTN575QXVA, seg_pos={0 0}, upload=447552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 26 21:30:54.043903341 (510h1m53.914986376s) [noise=true , address=145.79.227.43:4159 , piece_id=VLAIPO7BPFNE7DI2L4UKQYUANDHUQVWFH7GI4SXC6KVTN575QXVA, seg_pos={0 0}, upload=447552]: Long tail cancellation due to optimal threshold Dec 26 21:30:54.044298543 (510h1m53.915381578s) [noise=true , address=45.91.116.13:4153 , piece_id=YWDYXI7HO4546TDIWFKEHL7VYQ5O7WZ3XHGNGOBM67EIWO6OTFXQ, seg_pos={0 0}, upload=447552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 26 21:30:54.044318151 (510h1m53.915401186s) [noise=true , address=45.91.116.13:4153 , piece_id=YWDYXI7HO4546TDIWFKEHL7VYQ5O7WZ3XHGNGOBM67EIWO6OTFXQ, seg_pos={0 0}, upload=447552]: Long tail cancellation due to optimal threshold Dec 26 21:30:54.044347435 (510h1m53.915430471s) [seg_pos={0 0}, upload=447552]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 21:30:54.044473566 (510h1m53.915556611s) [upload=447552]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:30:54.044481541 (510h1m53.915564576s) [upload=447552]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:30:54.368560325 (510h1m54.23964336s) [upload=447552]: Done uploading segments... Dec 26 21:30:54.368579281 (510h1m54.239662316s) [upload=447552]: Upload finished. err: <nil> Duration: 30.002560761s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...........X....................XX.XX.X..X.X...XX.X......X..] duration: 23.74699479s trace: https://jaeger.dev.storj.io/trace/1371bfedf867c5c8 FAIL (2025-12-26 21:30:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 21:28:45.912165011 (509h59m45.783248047s) [upload=447525]: Starting upload Dec 26 21:28:45.912224214 (509h59m45.78330725s) [upload=447525]: Done starting upload Dec 26 21:28:45.912284570 (509h59m45.783367605s) [upload=447525]: Uploading segments... Dec 26 21:28:45.912390121 (509h59m45.783473156s) [upload=447525]: Got next segment. Inline: false Dec 26 21:28:45.912432351 (509h59m45.783515386s) [upload=447525]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 21:28:45.912436579 (509h59m45.783519614s) [upload=447525]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 21:28:46.163151965 (509h59m46.03423501s) [seg_pos={0 0}, upload=447525]: Begin upload segment... Dec 26 21:28:46.163207551 (509h59m46.034290586s) [seg_pos={0 0}, upload=447525]: Done begin upload segment. Dec 26 21:28:46.163225184 (509h59m46.034308229s) [noise=true , address=45.91.116.13:4154 , piece_id=BRZSDMSY2OCG6VJOFS7IVI5Q33SSBZY4UCQ3MCYUTD2IP5SKWASA, seg_pos={0 0}, upload=447525]: Uploading piece... Dec 26 21:28:46.163239803 (509h59m46.034322838s) [noise=true , address=145.79.227.42:4123 , piece_id=EVF5G7DNTXLUTEC72FOJLPGALY5326FCLRLPPGHZG4R46IIXU23A, seg_pos={0 0}, upload=447525]: Uploading piece... Dec 26 21:28:46.163929415 (509h59m46.03501245s) [noise=true , address=77.90.31.19:4841 , piece_id=WGQGWOTNOKUBC7TEU2SW57TA4RN24KNLYER3WKLSJXIHV6L4M63Q, seg_pos={0 0}, upload=447525]: Uploading piece... Dec 26 21:28:46.164245497 (509h59m46.035328562s) [noise=true , address=77.90.31.10:4517 , piece_id=QXRMF4J75EN5T7LKOOTU33LWVHWF23VLAKJKLG2F4RFRKTT4KM4Q, seg_pos={0 0}, upload=447525]: Uploading piece... Dec 26 21:28:46.164474774 (509h59m46.035557849s) [noise=true , address=77.90.31.14:4663 , piece_id=FL3UQHVB5JTJKUBPY2MG5HC2PQKWT5EUC6G5XQUQHIA747LCKFKA, seg_pos={0 0}, upload=447525]: Uploading piece... Dec 26 21:28:46.164758724 (509h59m46.035841779s) [noise=true , address=145.79.227.52:4447 , piece_id=D5X4HVB3HCYGEIKE3IOHWGSMVE3Y7LMAB4RLAMY7HR7UHQFGSU6A, seg_pos={0 0}, upload=447525]: Uploading piece... Dec 26 21:28:46.164866469 (509h59m46.035949504s) [noise=true , address=77.90.31.16:4736 , piece_id=XTZQTZ26DWVAAXEGAZLHCQNM3PXSLS4QRQ76WQOIP6PUVHCZZU4A, seg_pos={0 0}, upload=447525]: Uploading piece... Dec 26 21:28:46.165140752 (509h59m46.036223797s) [noise=true , address=145.79.227.33:4870 , piece_id=ODPA3MIRMISJCPZ56RD6VPAQK4ZU7TBHJRIBZR35BKU2FTJX3YMQ, seg_pos={0 0}, upload=447525]: Uploading piece... Dec 26 21:28:46.165244830 (509h59m46.036327865s) [noise=true , address=145.79.227.25:4618 , piece_id=FU2N7FCWS75R2JUOWQU2MT7XG6IY2SKILNBFYQPJK5MBGNCIMASQ, seg_pos={0 0}, upload=447525]: Uploading piece... Dec 26 21:28:46.165539952 (509h59m46.036622987s) [noise=true , address=45.91.116.8:4048 , piece_id=ESMVYMNHDLKY6YM4I62OQQDTKMKB55Z6O76DQSGQX2FEKBYNXERA, seg_pos={0 0}, upload=447525]: Uploading piece... Dec 26 21:28:46.166064431 (509h59m46.037147466s) [noise=true , address=45.91.116.10:4084 , piece_id=LN5MCR7GXIEB623YLZNZ7RCXKETWR267BRDV4ZQIUEUGOJ3SWQJA, seg_pos={0 0}, upload=447525]: Uploading piece... Dec 26 21:28:46.166477347 (509h59m46.037560393s) [noise=true , address=77.90.31.11:4554 , piece_id=2EREUSZAHES7NP2BX4GM6HROGC5W27AFEPPF3PFHY7PJN5GTIQQA, seg_pos={0 0}, upload=447525]: Uploading piece... Dec 26 21:28:46.166897427 (509h59m46.037980462s) [noise=true , address=77.90.31.17:4770 , piece_id=MISQMFA33LIH7S3NV7QUKDZPIVUENJGHROFY7Q65ZRD5ZUDIFFUQ, seg_pos={0 0}, upload=447525]: Uploading piece... Dec 26 21:28:46.167304042 (509h59m46.038387087s) [noise=true , address=145.79.227.50:4373 , piece_id=PIWKIB7DGB3IMK4OFTSCHKQW2ZA3QRGPROMEQORL4KOOIAQO5AGQ, seg_pos={0 0}, upload=447525]: Uploading piece... Dec 26 21:28:46.167747576 (509h59m46.038830611s) [noise=true , address=145.79.227.6:4259 , piece_id=KHJJPJ4NVSALFB7HM2QTVMRVOX54BKX2EVOE2NIASCN4HDEV3U7Q, seg_pos={0 0}, upload=447525]: Uploading piece... Dec 26 21:28:46.168182354 (509h59m46.039265389s) [noise=true , address=145.79.227.37:4567 , piece_id=KWNASWGS3ZGUPP2FAUMGBC2Y7BRMASSDKES6S7DBE4MEWU7WTGOA, seg_pos={0 0}, upload=447525]: Uploading piece... Dec 26 21:28:47.874472141 (509h59m47.745555176s) [upload=447525]: Next returned nil segment Dec 26 21:28:47.874505685 (509h59m47.745588931s) [upload=447525]: Waiting for error group managing segments... Dec 26 21:28:48.157912623 (509h59m48.028995658s) [noise=true , address=145.79.227.50:4373 , piece_id=PIWKIB7DGB3IMK4OFTSCHKQW2ZA3QRGPROMEQORL4KOOIAQO5AGQ, seg_pos={0 0}, upload=447525]: Done uploading piece. err: <nil> Dec 26 21:28:48.162722100 (509h59m48.033805135s) [noise=true , address=145.79.227.37:4567 , piece_id=KWNASWGS3ZGUPP2FAUMGBC2Y7BRMASSDKES6S7DBE4MEWU7WTGOA, seg_pos={0 0}, upload=447525]: Done uploading piece. err: <nil> Dec 26 21:28:48.171436054 (509h59m48.042519099s) [noise=true , address=145.79.227.52:4447 , piece_id=D5X4HVB3HCYGEIKE3IOHWGSMVE3Y7LMAB4RLAMY7HR7UHQFGSU6A, seg_pos={0 0}, upload=447525]: Done uploading piece. err: <nil> Dec 26 21:28:48.182436491 (509h59m48.053519526s) [noise=true , address=145.79.227.42:4123 , piece_id=EVF5G7DNTXLUTEC72FOJLPGALY5326FCLRLPPGHZG4R46IIXU23A, seg_pos={0 0}, upload=447525]: Done uploading piece. err: <nil> Dec 26 21:28:48.186573789 (509h59m48.057656824s) [noise=true , address=145.79.227.25:4618 , piece_id=FU2N7FCWS75R2JUOWQU2MT7XG6IY2SKILNBFYQPJK5MBGNCIMASQ, seg_pos={0 0}, upload=447525]: Done uploading piece. err: <nil> Dec 26 21:28:48.313126337 (509h59m48.184209372s) [noise=true , address=145.79.227.33:4870 , piece_id=ODPA3MIRMISJCPZ56RD6VPAQK4ZU7TBHJRIBZR35BKU2FTJX3YMQ, seg_pos={0 0}, upload=447525]: Done uploading piece. err: <nil> Dec 26 21:28:48.399794218 (509h59m48.270877253s) [noise=true , address=77.90.31.11:4554 , piece_id=2EREUSZAHES7NP2BX4GM6HROGC5W27AFEPPF3PFHY7PJN5GTIQQA, seg_pos={0 0}, upload=447525]: Done uploading piece. err: <nil> Dec 26 21:28:48.411200788 (509h59m48.282283823s) [noise=true , address=45.91.116.8:4048 , piece_id=ESMVYMNHDLKY6YM4I62OQQDTKMKB55Z6O76DQSGQX2FEKBYNXERA, seg_pos={0 0}, upload=447525]: Done uploading piece. err: <nil> Dec 26 21:28:48.426320379 (509h59m48.297403414s) [noise=true , address=77.90.31.14:4663 , piece_id=FL3UQHVB5JTJKUBPY2MG5HC2PQKWT5EUC6G5XQUQHIA747LCKFKA, seg_pos={0 0}, upload=447525]: Done uploading piece. err: <nil> Dec 26 21:28:48.430827300 (509h59m48.301910335s) [noise=true , address=77.90.31.16:4736 , piece_id=XTZQTZ26DWVAAXEGAZLHCQNM3PXSLS4QRQ76WQOIP6PUVHCZZU4A, seg_pos={0 0}, upload=447525]: Done uploading piece. err: <nil> Dec 26 21:28:48.434335410 (509h59m48.305418446s) [noise=true , address=45.91.116.10:4084 , piece_id=LN5MCR7GXIEB623YLZNZ7RCXKETWR267BRDV4ZQIUEUGOJ3SWQJA, seg_pos={0 0}, upload=447525]: Done uploading piece. err: <nil> Dec 26 21:28:48.437381841 (509h59m48.308464876s) [noise=true , address=45.91.116.13:4154 , piece_id=BRZSDMSY2OCG6VJOFS7IVI5Q33SSBZY4UCQ3MCYUTD2IP5SKWASA, seg_pos={0 0}, upload=447525]: Done uploading piece. err: <nil> Dec 26 21:28:48.485937820 (509h59m48.357020855s) [noise=true , address=77.90.31.17:4770 , piece_id=MISQMFA33LIH7S3NV7QUKDZPIVUENJGHROFY7Q65ZRD5ZUDIFFUQ, seg_pos={0 0}, upload=447525]: Done uploading piece. err: <nil> Dec 26 21:28:48.485956907 (509h59m48.357039942s) [seg_pos={0 0}, upload=447525]: Segment reached optimal threshold of 13 pieces. Dec 26 21:28:48.486203576 (509h59m48.357286611s) [noise=true , address=77.90.31.10:4517 , piece_id=QXRMF4J75EN5T7LKOOTU33LWVHWF23VLAKJKLG2F4RFRKTT4KM4Q, seg_pos={0 0}, upload=447525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 26 21:28:48.486213555 (509h59m48.35729659s) [noise=true , address=77.90.31.10:4517 , piece_id=QXRMF4J75EN5T7LKOOTU33LWVHWF23VLAKJKLG2F4RFRKTT4KM4Q, seg_pos={0 0}, upload=447525]: Long tail cancellation due to optimal threshold Dec 26 21:28:48.486246007 (509h59m48.357329042s) [noise=true , address=145.79.227.6:4259 , piece_id=KHJJPJ4NVSALFB7HM2QTVMRVOX54BKX2EVOE2NIASCN4HDEV3U7Q, seg_pos={0 0}, upload=447525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:34518->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:34518->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34518->145.79.227.6:4259: use of closed network connection Dec 26 21:28:48.486262058 (509h59m48.357345103s) [noise=true , address=145.79.227.6:4259 , piece_id=KHJJPJ4NVSALFB7HM2QTVMRVOX54BKX2EVOE2NIASCN4HDEV3U7Q, seg_pos={0 0}, upload=447525]: Long tail cancellation due to optimal threshold Dec 26 21:28:48.486427512 (509h59m48.357510547s) [noise=true , address=77.90.31.19:4841 , piece_id=WGQGWOTNOKUBC7TEU2SW57TA4RN24KNLYER3WKLSJXIHV6L4M63Q, seg_pos={0 0}, upload=447525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF Dec 26 21:28:48.486434336 (509h59m48.357517371s) [noise=true , address=77.90.31.19:4841 , piece_id=WGQGWOTNOKUBC7TEU2SW57TA4RN24KNLYER3WKLSJXIHV6L4M63Q, seg_pos={0 0}, upload=447525]: Long tail cancellation due to optimal threshold Dec 26 21:28:48.486452500 (509h59m48.357535535s) [seg_pos={0 0}, upload=447525]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 21:28:48.486513867 (509h59m48.357596912s) [upload=447525]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:28:48.486518436 (509h59m48.357601471s) [upload=447525]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:28:48.798537907 (509h59m48.669620952s) [upload=447525]: Done uploading segments... Dec 26 21:28:48.798560891 (509h59m48.669643926s) [upload=447525]: Upload finished. err: <nil> Duration: 23.747022523s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [............X....................XX.XX.X..X.X...XX.X......X.] duration: 29.084683013s trace: https://jaeger.dev.storj.io/trace/1371bfedf867c5c8 FAIL (2025-12-26 21:30:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 21:28:45.912165011 (509h59m45.783248047s) [upload=447525]: Starting upload Dec 26 21:28:45.912224214 (509h59m45.78330725s) [upload=447525]: Done starting upload Dec 26 21:28:45.912284570 (509h59m45.783367605s) [upload=447525]: Uploading segments... Dec 26 21:28:45.912390121 (509h59m45.783473156s) [upload=447525]: Got next segment. Inline: false Dec 26 21:28:45.912432351 (509h59m45.783515386s) [upload=447525]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 21:28:45.912436579 (509h59m45.783519614s) [upload=447525]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 21:28:46.163151965 (509h59m46.03423501s) [seg_pos={0 0}, upload=447525]: Begin upload segment... Dec 26 21:28:46.163207551 (509h59m46.034290586s) [seg_pos={0 0}, upload=447525]: Done begin upload segment. Dec 26 21:28:46.163225184 (509h59m46.034308229s) [noise=true , address=45.91.116.13:4154 , piece_id=BRZSDMSY2OCG6VJOFS7IVI5Q33SSBZY4UCQ3MCYUTD2IP5SKWASA, seg_pos={0 0}, upload=447525]: Uploading piece... Dec 26 21:28:46.163239803 (509h59m46.034322838s) [noise=true , address=145.79.227.42:4123 , piece_id=EVF5G7DNTXLUTEC72FOJLPGALY5326FCLRLPPGHZG4R46IIXU23A, seg_pos={0 0}, upload=447525]: Uploading piece... Dec 26 21:28:46.163929415 (509h59m46.03501245s) [noise=true , address=77.90.31.19:4841 , piece_id=WGQGWOTNOKUBC7TEU2SW57TA4RN24KNLYER3WKLSJXIHV6L4M63Q, seg_pos={0 0}, upload=447525]: Uploading piece... Dec 26 21:28:46.164245497 (509h59m46.035328562s) [noise=true , address=77.90.31.10:4517 , piece_id=QXRMF4J75EN5T7LKOOTU33LWVHWF23VLAKJKLG2F4RFRKTT4KM4Q, seg_pos={0 0}, upload=447525]: Uploading piece... Dec 26 21:28:46.164474774 (509h59m46.035557849s) [noise=true , address=77.90.31.14:4663 , piece_id=FL3UQHVB5JTJKUBPY2MG5HC2PQKWT5EUC6G5XQUQHIA747LCKFKA, seg_pos={0 0}, upload=447525]: Uploading piece... Dec 26 21:28:46.164758724 (509h59m46.035841779s) [noise=true , address=145.79.227.52:4447 , piece_id=D5X4HVB3HCYGEIKE3IOHWGSMVE3Y7LMAB4RLAMY7HR7UHQFGSU6A, seg_pos={0 0}, upload=447525]: Uploading piece... Dec 26 21:28:46.164866469 (509h59m46.035949504s) [noise=true , address=77.90.31.16:4736 , piece_id=XTZQTZ26DWVAAXEGAZLHCQNM3PXSLS4QRQ76WQOIP6PUVHCZZU4A, seg_pos={0 0}, upload=447525]: Uploading piece... Dec 26 21:28:46.165140752 (509h59m46.036223797s) [noise=true , address=145.79.227.33:4870 , piece_id=ODPA3MIRMISJCPZ56RD6VPAQK4ZU7TBHJRIBZR35BKU2FTJX3YMQ, seg_pos={0 0}, upload=447525]: Uploading piece... Dec 26 21:28:46.165244830 (509h59m46.036327865s) [noise=true , address=145.79.227.25:4618 , piece_id=FU2N7FCWS75R2JUOWQU2MT7XG6IY2SKILNBFYQPJK5MBGNCIMASQ, seg_pos={0 0}, upload=447525]: Uploading piece... Dec 26 21:28:46.165539952 (509h59m46.036622987s) [noise=true , address=45.91.116.8:4048 , piece_id=ESMVYMNHDLKY6YM4I62OQQDTKMKB55Z6O76DQSGQX2FEKBYNXERA, seg_pos={0 0}, upload=447525]: Uploading piece... Dec 26 21:28:46.166064431 (509h59m46.037147466s) [noise=true , address=45.91.116.10:4084 , piece_id=LN5MCR7GXIEB623YLZNZ7RCXKETWR267BRDV4ZQIUEUGOJ3SWQJA, seg_pos={0 0}, upload=447525]: Uploading piece... Dec 26 21:28:46.166477347 (509h59m46.037560393s) [noise=true , address=77.90.31.11:4554 , piece_id=2EREUSZAHES7NP2BX4GM6HROGC5W27AFEPPF3PFHY7PJN5GTIQQA, seg_pos={0 0}, upload=447525]: Uploading piece... Dec 26 21:28:46.166897427 (509h59m46.037980462s) [noise=true , address=77.90.31.17:4770 , piece_id=MISQMFA33LIH7S3NV7QUKDZPIVUENJGHROFY7Q65ZRD5ZUDIFFUQ, seg_pos={0 0}, upload=447525]: Uploading piece... Dec 26 21:28:46.167304042 (509h59m46.038387087s) [noise=true , address=145.79.227.50:4373 , piece_id=PIWKIB7DGB3IMK4OFTSCHKQW2ZA3QRGPROMEQORL4KOOIAQO5AGQ, seg_pos={0 0}, upload=447525]: Uploading piece... Dec 26 21:28:46.167747576 (509h59m46.038830611s) [noise=true , address=145.79.227.6:4259 , piece_id=KHJJPJ4NVSALFB7HM2QTVMRVOX54BKX2EVOE2NIASCN4HDEV3U7Q, seg_pos={0 0}, upload=447525]: Uploading piece... Dec 26 21:28:46.168182354 (509h59m46.039265389s) [noise=true , address=145.79.227.37:4567 , piece_id=KWNASWGS3ZGUPP2FAUMGBC2Y7BRMASSDKES6S7DBE4MEWU7WTGOA, seg_pos={0 0}, upload=447525]: Uploading piece... Dec 26 21:28:47.874472141 (509h59m47.745555176s) [upload=447525]: Next returned nil segment Dec 26 21:28:47.874505685 (509h59m47.745588931s) [upload=447525]: Waiting for error group managing segments... Dec 26 21:28:48.157912623 (509h59m48.028995658s) [noise=true , address=145.79.227.50:4373 , piece_id=PIWKIB7DGB3IMK4OFTSCHKQW2ZA3QRGPROMEQORL4KOOIAQO5AGQ, seg_pos={0 0}, upload=447525]: Done uploading piece. err: <nil> Dec 26 21:28:48.162722100 (509h59m48.033805135s) [noise=true , address=145.79.227.37:4567 , piece_id=KWNASWGS3ZGUPP2FAUMGBC2Y7BRMASSDKES6S7DBE4MEWU7WTGOA, seg_pos={0 0}, upload=447525]: Done uploading piece. err: <nil> Dec 26 21:28:48.171436054 (509h59m48.042519099s) [noise=true , address=145.79.227.52:4447 , piece_id=D5X4HVB3HCYGEIKE3IOHWGSMVE3Y7LMAB4RLAMY7HR7UHQFGSU6A, seg_pos={0 0}, upload=447525]: Done uploading piece. err: <nil> Dec 26 21:28:48.182436491 (509h59m48.053519526s) [noise=true , address=145.79.227.42:4123 , piece_id=EVF5G7DNTXLUTEC72FOJLPGALY5326FCLRLPPGHZG4R46IIXU23A, seg_pos={0 0}, upload=447525]: Done uploading piece. err: <nil> Dec 26 21:28:48.186573789 (509h59m48.057656824s) [noise=true , address=145.79.227.25:4618 , piece_id=FU2N7FCWS75R2JUOWQU2MT7XG6IY2SKILNBFYQPJK5MBGNCIMASQ, seg_pos={0 0}, upload=447525]: Done uploading piece. err: <nil> Dec 26 21:28:48.313126337 (509h59m48.184209372s) [noise=true , address=145.79.227.33:4870 , piece_id=ODPA3MIRMISJCPZ56RD6VPAQK4ZU7TBHJRIBZR35BKU2FTJX3YMQ, seg_pos={0 0}, upload=447525]: Done uploading piece. err: <nil> Dec 26 21:28:48.399794218 (509h59m48.270877253s) [noise=true , address=77.90.31.11:4554 , piece_id=2EREUSZAHES7NP2BX4GM6HROGC5W27AFEPPF3PFHY7PJN5GTIQQA, seg_pos={0 0}, upload=447525]: Done uploading piece. err: <nil> Dec 26 21:28:48.411200788 (509h59m48.282283823s) [noise=true , address=45.91.116.8:4048 , piece_id=ESMVYMNHDLKY6YM4I62OQQDTKMKB55Z6O76DQSGQX2FEKBYNXERA, seg_pos={0 0}, upload=447525]: Done uploading piece. err: <nil> Dec 26 21:28:48.426320379 (509h59m48.297403414s) [noise=true , address=77.90.31.14:4663 , piece_id=FL3UQHVB5JTJKUBPY2MG5HC2PQKWT5EUC6G5XQUQHIA747LCKFKA, seg_pos={0 0}, upload=447525]: Done uploading piece. err: <nil> Dec 26 21:28:48.430827300 (509h59m48.301910335s) [noise=true , address=77.90.31.16:4736 , piece_id=XTZQTZ26DWVAAXEGAZLHCQNM3PXSLS4QRQ76WQOIP6PUVHCZZU4A, seg_pos={0 0}, upload=447525]: Done uploading piece. err: <nil> Dec 26 21:28:48.434335410 (509h59m48.305418446s) [noise=true , address=45.91.116.10:4084 , piece_id=LN5MCR7GXIEB623YLZNZ7RCXKETWR267BRDV4ZQIUEUGOJ3SWQJA, seg_pos={0 0}, upload=447525]: Done uploading piece. err: <nil> Dec 26 21:28:48.437381841 (509h59m48.308464876s) [noise=true , address=45.91.116.13:4154 , piece_id=BRZSDMSY2OCG6VJOFS7IVI5Q33SSBZY4UCQ3MCYUTD2IP5SKWASA, seg_pos={0 0}, upload=447525]: Done uploading piece. err: <nil> Dec 26 21:28:48.485937820 (509h59m48.357020855s) [noise=true , address=77.90.31.17:4770 , piece_id=MISQMFA33LIH7S3NV7QUKDZPIVUENJGHROFY7Q65ZRD5ZUDIFFUQ, seg_pos={0 0}, upload=447525]: Done uploading piece. err: <nil> Dec 26 21:28:48.485956907 (509h59m48.357039942s) [seg_pos={0 0}, upload=447525]: Segment reached optimal threshold of 13 pieces. Dec 26 21:28:48.486203576 (509h59m48.357286611s) [noise=true , address=77.90.31.10:4517 , piece_id=QXRMF4J75EN5T7LKOOTU33LWVHWF23VLAKJKLG2F4RFRKTT4KM4Q, seg_pos={0 0}, upload=447525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 26 21:28:48.486213555 (509h59m48.35729659s) [noise=true , address=77.90.31.10:4517 , piece_id=QXRMF4J75EN5T7LKOOTU33LWVHWF23VLAKJKLG2F4RFRKTT4KM4Q, seg_pos={0 0}, upload=447525]: Long tail cancellation due to optimal threshold Dec 26 21:28:48.486246007 (509h59m48.357329042s) [noise=true , address=145.79.227.6:4259 , piece_id=KHJJPJ4NVSALFB7HM2QTVMRVOX54BKX2EVOE2NIASCN4HDEV3U7Q, seg_pos={0 0}, upload=447525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:34518->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:34518->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34518->145.79.227.6:4259: use of closed network connection Dec 26 21:28:48.486262058 (509h59m48.357345103s) [noise=true , address=145.79.227.6:4259 , piece_id=KHJJPJ4NVSALFB7HM2QTVMRVOX54BKX2EVOE2NIASCN4HDEV3U7Q, seg_pos={0 0}, upload=447525]: Long tail cancellation due to optimal threshold Dec 26 21:28:48.486427512 (509h59m48.357510547s) [noise=true , address=77.90.31.19:4841 , piece_id=WGQGWOTNOKUBC7TEU2SW57TA4RN24KNLYER3WKLSJXIHV6L4M63Q, seg_pos={0 0}, upload=447525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF Dec 26 21:28:48.486434336 (509h59m48.357517371s) [noise=true , address=77.90.31.19:4841 , piece_id=WGQGWOTNOKUBC7TEU2SW57TA4RN24KNLYER3WKLSJXIHV6L4M63Q, seg_pos={0 0}, upload=447525]: Long tail cancellation due to optimal threshold Dec 26 21:28:48.486452500 (509h59m48.357535535s) [seg_pos={0 0}, upload=447525]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 21:28:48.486513867 (509h59m48.357596912s) [upload=447525]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:28:48.486518436 (509h59m48.357601471s) [upload=447525]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:28:48.798537907 (509h59m48.669620952s) [upload=447525]: Done uploading segments... Dec 26 21:28:48.798560891 (509h59m48.669643926s) [upload=447525]: Upload finished. err: <nil> Duration: 29.084708944s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 10.572350587s trace: https://jaeger.dev.storj.io/trace/6d2950019117adaf FAIL (2025-12-26 21:30:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Rj4hUGWvUPuU49QDsHYhibiJhyY5KDGus1rzNeP7KusgyvTsU, Piece ID: NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A) context deadline exceeded; piecestore: (Node ID: 12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, Piece ID: N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ) context deadline exceeded; piecestore: (Node ID: 12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, Piece ID: IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ) context deadline exceeded ======LOGS====== ng piece... Dec 26 20:54:08.087336805 (509h25m7.95841984s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087649520 (509h25m7.958732565s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087988717 (509h25m7.959071752s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088323233 (509h25m7.959406268s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088877028 (509h25m7.959960063s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089247203 (509h25m7.960330238s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089543066 (509h25m7.960626101s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089914133 (509h25m7.960997168s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090327259 (509h25m7.961410294s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090645345 (509h25m7.96172838s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091004318 (509h25m7.962087353s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091338976 (509h25m7.962422021s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091732054 (509h25m7.962815099s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:09.565187143 (509h25m9.436270188s) [upload=447022]: Next returned nil segment Dec 26 20:54:09.565201321 (509h25m9.436284356s) [upload=447022]: Waiting for error group managing segments... Dec 26 20:54:10.137065972 (509h25m10.008149007s) [noise=true , address=77.90.31.10:4523 , piece_id=GQYT3Z53Y6HVTY3IBUNFCIWBEE7KVAUWEJYO7QJFRRAJCHTJMYFA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.137147197 (509h25m10.008230232s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.149325196 (509h25m10.020408242s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.159611673 (509h25m10.030694718s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.164411764 (509h25m10.035494809s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.190924278 (509h25m10.062007313s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.232931086 (509h25m10.104014131s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.241658245 (509h25m10.11274129s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.472574370 (509h25m10.343657405s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.866910566 (509h25m10.737993611s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.886583507 (509h25m10.757666542s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.127773215 (509h25m10.99885626s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.192133217 (509h25m11.063216263s) [noise=true , address=145.79.227.13:4819 , piece_id=BQDRL2QJZ75TAMUM2VSGTF4U3NEMZC4FFQLN6QV6UOAFHWB56MZA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256146219 (509h25m11.127229264s) [noise=true , address=145.79.227.40:4062 , piece_id=IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256176597 (509h25m11.127259632s) [seg_pos={0 0}, upload=447022]: Segment reached optimal threshold of 14 pieces. Dec 26 20:54:11.256428137 (509h25m11.127511192s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98): protocol: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection Dec 26 20:54:11.256439549 (509h25m11.127522584s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256458915 (509h25m11.12754198s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF): protocol: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection Dec 26 20:54:11.256490705 (509h25m11.127573741s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256698171 (509h25m11.127781216s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: rpc: dial tcp 145.79.227.30:4796: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:54:11.256706457 (509h25m11.127789502s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256917309 (509h25m11.128000364s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection Dec 26 20:54:11.256926666 (509h25m11.128009711s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256964869 (509h25m11.128047914s) [seg_pos={0 0}, upload=447022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:54:11.257064138 (509h25m11.128147173s) [upload=447022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:11.257070690 (509h25m11.128153725s) [upload=447022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:11.561962810 (509h25m11.433045875s) [upload=447022]: Done uploading segments... Dec 26 20:54:11.561987427 (509h25m11.433070472s) [upload=447022]: Upload finished. err: <nil> Duration: 10.57238392s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 15.386778159s trace: https://jaeger.dev.storj.io/trace/6d2950019117adaf FAIL (2025-12-26 21:29:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Rj4hUGWvUPuU49QDsHYhibiJhyY5KDGus1rzNeP7KusgyvTsU, Piece ID: NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A) context deadline exceeded; piecestore: (Node ID: 12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, Piece ID: N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ) context deadline exceeded; piecestore: (Node ID: 12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, Piece ID: IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ) context deadline exceeded ======LOGS====== ng piece... Dec 26 20:54:08.087336805 (509h25m7.95841984s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087649520 (509h25m7.958732565s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087988717 (509h25m7.959071752s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088323233 (509h25m7.959406268s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088877028 (509h25m7.959960063s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089247203 (509h25m7.960330238s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089543066 (509h25m7.960626101s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089914133 (509h25m7.960997168s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090327259 (509h25m7.961410294s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090645345 (509h25m7.96172838s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091004318 (509h25m7.962087353s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091338976 (509h25m7.962422021s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091732054 (509h25m7.962815099s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:09.565187143 (509h25m9.436270188s) [upload=447022]: Next returned nil segment Dec 26 20:54:09.565201321 (509h25m9.436284356s) [upload=447022]: Waiting for error group managing segments... Dec 26 20:54:10.137065972 (509h25m10.008149007s) [noise=true , address=77.90.31.10:4523 , piece_id=GQYT3Z53Y6HVTY3IBUNFCIWBEE7KVAUWEJYO7QJFRRAJCHTJMYFA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.137147197 (509h25m10.008230232s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.149325196 (509h25m10.020408242s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.159611673 (509h25m10.030694718s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.164411764 (509h25m10.035494809s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.190924278 (509h25m10.062007313s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.232931086 (509h25m10.104014131s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.241658245 (509h25m10.11274129s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.472574370 (509h25m10.343657405s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.866910566 (509h25m10.737993611s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.886583507 (509h25m10.757666542s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.127773215 (509h25m10.99885626s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.192133217 (509h25m11.063216263s) [noise=true , address=145.79.227.13:4819 , piece_id=BQDRL2QJZ75TAMUM2VSGTF4U3NEMZC4FFQLN6QV6UOAFHWB56MZA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256146219 (509h25m11.127229264s) [noise=true , address=145.79.227.40:4062 , piece_id=IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256176597 (509h25m11.127259632s) [seg_pos={0 0}, upload=447022]: Segment reached optimal threshold of 14 pieces. Dec 26 20:54:11.256428137 (509h25m11.127511192s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98): protocol: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection Dec 26 20:54:11.256439549 (509h25m11.127522584s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256458915 (509h25m11.12754198s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF): protocol: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection Dec 26 20:54:11.256490705 (509h25m11.127573741s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256698171 (509h25m11.127781216s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: rpc: dial tcp 145.79.227.30:4796: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:54:11.256706457 (509h25m11.127789502s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256917309 (509h25m11.128000364s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection Dec 26 20:54:11.256926666 (509h25m11.128009711s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256964869 (509h25m11.128047914s) [seg_pos={0 0}, upload=447022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:54:11.257064138 (509h25m11.128147173s) [upload=447022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:11.257070690 (509h25m11.128153725s) [upload=447022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:11.561962810 (509h25m11.433045875s) [upload=447022]: Done uploading segments... Dec 26 20:54:11.561987427 (509h25m11.433070472s) [upload=447022]: Upload finished. err: <nil> Duration: 15.38680984s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.............X....................XX.XX.X..X.X...XX.X......X] duration: 30.002407388s trace: https://jaeger.dev.storj.io/trace/1371bfedf867c5c8 FAIL (2025-12-26 21:29:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 21:28:45.912165011 (509h59m45.783248047s) [upload=447525]: Starting upload Dec 26 21:28:45.912224214 (509h59m45.78330725s) [upload=447525]: Done starting upload Dec 26 21:28:45.912284570 (509h59m45.783367605s) [upload=447525]: Uploading segments... Dec 26 21:28:45.912390121 (509h59m45.783473156s) [upload=447525]: Got next segment. Inline: false Dec 26 21:28:45.912432351 (509h59m45.783515386s) [upload=447525]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 21:28:45.912436579 (509h59m45.783519614s) [upload=447525]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 21:28:46.163151965 (509h59m46.03423501s) [seg_pos={0 0}, upload=447525]: Begin upload segment... Dec 26 21:28:46.163207551 (509h59m46.034290586s) [seg_pos={0 0}, upload=447525]: Done begin upload segment. Dec 26 21:28:46.163225184 (509h59m46.034308229s) [noise=true , address=45.91.116.13:4154 , piece_id=BRZSDMSY2OCG6VJOFS7IVI5Q33SSBZY4UCQ3MCYUTD2IP5SKWASA, seg_pos={0 0}, upload=447525]: Uploading piece... Dec 26 21:28:46.163239803 (509h59m46.034322838s) [noise=true , address=145.79.227.42:4123 , piece_id=EVF5G7DNTXLUTEC72FOJLPGALY5326FCLRLPPGHZG4R46IIXU23A, seg_pos={0 0}, upload=447525]: Uploading piece... Dec 26 21:28:46.163929415 (509h59m46.03501245s) [noise=true , address=77.90.31.19:4841 , piece_id=WGQGWOTNOKUBC7TEU2SW57TA4RN24KNLYER3WKLSJXIHV6L4M63Q, seg_pos={0 0}, upload=447525]: Uploading piece... Dec 26 21:28:46.164245497 (509h59m46.035328562s) [noise=true , address=77.90.31.10:4517 , piece_id=QXRMF4J75EN5T7LKOOTU33LWVHWF23VLAKJKLG2F4RFRKTT4KM4Q, seg_pos={0 0}, upload=447525]: Uploading piece... Dec 26 21:28:46.164474774 (509h59m46.035557849s) [noise=true , address=77.90.31.14:4663 , piece_id=FL3UQHVB5JTJKUBPY2MG5HC2PQKWT5EUC6G5XQUQHIA747LCKFKA, seg_pos={0 0}, upload=447525]: Uploading piece... Dec 26 21:28:46.164758724 (509h59m46.035841779s) [noise=true , address=145.79.227.52:4447 , piece_id=D5X4HVB3HCYGEIKE3IOHWGSMVE3Y7LMAB4RLAMY7HR7UHQFGSU6A, seg_pos={0 0}, upload=447525]: Uploading piece... Dec 26 21:28:46.164866469 (509h59m46.035949504s) [noise=true , address=77.90.31.16:4736 , piece_id=XTZQTZ26DWVAAXEGAZLHCQNM3PXSLS4QRQ76WQOIP6PUVHCZZU4A, seg_pos={0 0}, upload=447525]: Uploading piece... Dec 26 21:28:46.165140752 (509h59m46.036223797s) [noise=true , address=145.79.227.33:4870 , piece_id=ODPA3MIRMISJCPZ56RD6VPAQK4ZU7TBHJRIBZR35BKU2FTJX3YMQ, seg_pos={0 0}, upload=447525]: Uploading piece... Dec 26 21:28:46.165244830 (509h59m46.036327865s) [noise=true , address=145.79.227.25:4618 , piece_id=FU2N7FCWS75R2JUOWQU2MT7XG6IY2SKILNBFYQPJK5MBGNCIMASQ, seg_pos={0 0}, upload=447525]: Uploading piece... Dec 26 21:28:46.165539952 (509h59m46.036622987s) [noise=true , address=45.91.116.8:4048 , piece_id=ESMVYMNHDLKY6YM4I62OQQDTKMKB55Z6O76DQSGQX2FEKBYNXERA, seg_pos={0 0}, upload=447525]: Uploading piece... Dec 26 21:28:46.166064431 (509h59m46.037147466s) [noise=true , address=45.91.116.10:4084 , piece_id=LN5MCR7GXIEB623YLZNZ7RCXKETWR267BRDV4ZQIUEUGOJ3SWQJA, seg_pos={0 0}, upload=447525]: Uploading piece... Dec 26 21:28:46.166477347 (509h59m46.037560393s) [noise=true , address=77.90.31.11:4554 , piece_id=2EREUSZAHES7NP2BX4GM6HROGC5W27AFEPPF3PFHY7PJN5GTIQQA, seg_pos={0 0}, upload=447525]: Uploading piece... Dec 26 21:28:46.166897427 (509h59m46.037980462s) [noise=true , address=77.90.31.17:4770 , piece_id=MISQMFA33LIH7S3NV7QUKDZPIVUENJGHROFY7Q65ZRD5ZUDIFFUQ, seg_pos={0 0}, upload=447525]: Uploading piece... Dec 26 21:28:46.167304042 (509h59m46.038387087s) [noise=true , address=145.79.227.50:4373 , piece_id=PIWKIB7DGB3IMK4OFTSCHKQW2ZA3QRGPROMEQORL4KOOIAQO5AGQ, seg_pos={0 0}, upload=447525]: Uploading piece... Dec 26 21:28:46.167747576 (509h59m46.038830611s) [noise=true , address=145.79.227.6:4259 , piece_id=KHJJPJ4NVSALFB7HM2QTVMRVOX54BKX2EVOE2NIASCN4HDEV3U7Q, seg_pos={0 0}, upload=447525]: Uploading piece... Dec 26 21:28:46.168182354 (509h59m46.039265389s) [noise=true , address=145.79.227.37:4567 , piece_id=KWNASWGS3ZGUPP2FAUMGBC2Y7BRMASSDKES6S7DBE4MEWU7WTGOA, seg_pos={0 0}, upload=447525]: Uploading piece... Dec 26 21:28:47.874472141 (509h59m47.745555176s) [upload=447525]: Next returned nil segment Dec 26 21:28:47.874505685 (509h59m47.745588931s) [upload=447525]: Waiting for error group managing segments... Dec 26 21:28:48.157912623 (509h59m48.028995658s) [noise=true , address=145.79.227.50:4373 , piece_id=PIWKIB7DGB3IMK4OFTSCHKQW2ZA3QRGPROMEQORL4KOOIAQO5AGQ, seg_pos={0 0}, upload=447525]: Done uploading piece. err: <nil> Dec 26 21:28:48.162722100 (509h59m48.033805135s) [noise=true , address=145.79.227.37:4567 , piece_id=KWNASWGS3ZGUPP2FAUMGBC2Y7BRMASSDKES6S7DBE4MEWU7WTGOA, seg_pos={0 0}, upload=447525]: Done uploading piece. err: <nil> Dec 26 21:28:48.171436054 (509h59m48.042519099s) [noise=true , address=145.79.227.52:4447 , piece_id=D5X4HVB3HCYGEIKE3IOHWGSMVE3Y7LMAB4RLAMY7HR7UHQFGSU6A, seg_pos={0 0}, upload=447525]: Done uploading piece. err: <nil> Dec 26 21:28:48.182436491 (509h59m48.053519526s) [noise=true , address=145.79.227.42:4123 , piece_id=EVF5G7DNTXLUTEC72FOJLPGALY5326FCLRLPPGHZG4R46IIXU23A, seg_pos={0 0}, upload=447525]: Done uploading piece. err: <nil> Dec 26 21:28:48.186573789 (509h59m48.057656824s) [noise=true , address=145.79.227.25:4618 , piece_id=FU2N7FCWS75R2JUOWQU2MT7XG6IY2SKILNBFYQPJK5MBGNCIMASQ, seg_pos={0 0}, upload=447525]: Done uploading piece. err: <nil> Dec 26 21:28:48.313126337 (509h59m48.184209372s) [noise=true , address=145.79.227.33:4870 , piece_id=ODPA3MIRMISJCPZ56RD6VPAQK4ZU7TBHJRIBZR35BKU2FTJX3YMQ, seg_pos={0 0}, upload=447525]: Done uploading piece. err: <nil> Dec 26 21:28:48.399794218 (509h59m48.270877253s) [noise=true , address=77.90.31.11:4554 , piece_id=2EREUSZAHES7NP2BX4GM6HROGC5W27AFEPPF3PFHY7PJN5GTIQQA, seg_pos={0 0}, upload=447525]: Done uploading piece. err: <nil> Dec 26 21:28:48.411200788 (509h59m48.282283823s) [noise=true , address=45.91.116.8:4048 , piece_id=ESMVYMNHDLKY6YM4I62OQQDTKMKB55Z6O76DQSGQX2FEKBYNXERA, seg_pos={0 0}, upload=447525]: Done uploading piece. err: <nil> Dec 26 21:28:48.426320379 (509h59m48.297403414s) [noise=true , address=77.90.31.14:4663 , piece_id=FL3UQHVB5JTJKUBPY2MG5HC2PQKWT5EUC6G5XQUQHIA747LCKFKA, seg_pos={0 0}, upload=447525]: Done uploading piece. err: <nil> Dec 26 21:28:48.430827300 (509h59m48.301910335s) [noise=true , address=77.90.31.16:4736 , piece_id=XTZQTZ26DWVAAXEGAZLHCQNM3PXSLS4QRQ76WQOIP6PUVHCZZU4A, seg_pos={0 0}, upload=447525]: Done uploading piece. err: <nil> Dec 26 21:28:48.434335410 (509h59m48.305418446s) [noise=true , address=45.91.116.10:4084 , piece_id=LN5MCR7GXIEB623YLZNZ7RCXKETWR267BRDV4ZQIUEUGOJ3SWQJA, seg_pos={0 0}, upload=447525]: Done uploading piece. err: <nil> Dec 26 21:28:48.437381841 (509h59m48.308464876s) [noise=true , address=45.91.116.13:4154 , piece_id=BRZSDMSY2OCG6VJOFS7IVI5Q33SSBZY4UCQ3MCYUTD2IP5SKWASA, seg_pos={0 0}, upload=447525]: Done uploading piece. err: <nil> Dec 26 21:28:48.485937820 (509h59m48.357020855s) [noise=true , address=77.90.31.17:4770 , piece_id=MISQMFA33LIH7S3NV7QUKDZPIVUENJGHROFY7Q65ZRD5ZUDIFFUQ, seg_pos={0 0}, upload=447525]: Done uploading piece. err: <nil> Dec 26 21:28:48.485956907 (509h59m48.357039942s) [seg_pos={0 0}, upload=447525]: Segment reached optimal threshold of 13 pieces. Dec 26 21:28:48.486203576 (509h59m48.357286611s) [noise=true , address=77.90.31.10:4517 , piece_id=QXRMF4J75EN5T7LKOOTU33LWVHWF23VLAKJKLG2F4RFRKTT4KM4Q, seg_pos={0 0}, upload=447525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 26 21:28:48.486213555 (509h59m48.35729659s) [noise=true , address=77.90.31.10:4517 , piece_id=QXRMF4J75EN5T7LKOOTU33LWVHWF23VLAKJKLG2F4RFRKTT4KM4Q, seg_pos={0 0}, upload=447525]: Long tail cancellation due to optimal threshold Dec 26 21:28:48.486246007 (509h59m48.357329042s) [noise=true , address=145.79.227.6:4259 , piece_id=KHJJPJ4NVSALFB7HM2QTVMRVOX54BKX2EVOE2NIASCN4HDEV3U7Q, seg_pos={0 0}, upload=447525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:34518->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:34518->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34518->145.79.227.6:4259: use of closed network connection Dec 26 21:28:48.486262058 (509h59m48.357345103s) [noise=true , address=145.79.227.6:4259 , piece_id=KHJJPJ4NVSALFB7HM2QTVMRVOX54BKX2EVOE2NIASCN4HDEV3U7Q, seg_pos={0 0}, upload=447525]: Long tail cancellation due to optimal threshold Dec 26 21:28:48.486427512 (509h59m48.357510547s) [noise=true , address=77.90.31.19:4841 , piece_id=WGQGWOTNOKUBC7TEU2SW57TA4RN24KNLYER3WKLSJXIHV6L4M63Q, seg_pos={0 0}, upload=447525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF Dec 26 21:28:48.486434336 (509h59m48.357517371s) [noise=true , address=77.90.31.19:4841 , piece_id=WGQGWOTNOKUBC7TEU2SW57TA4RN24KNLYER3WKLSJXIHV6L4M63Q, seg_pos={0 0}, upload=447525]: Long tail cancellation due to optimal threshold Dec 26 21:28:48.486452500 (509h59m48.357535535s) [seg_pos={0 0}, upload=447525]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 21:28:48.486513867 (509h59m48.357596912s) [upload=447525]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:28:48.486518436 (509h59m48.357601471s) [upload=447525]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:28:48.798537907 (509h59m48.669620952s) [upload=447525]: Done uploading segments... Dec 26 21:28:48.798560891 (509h59m48.669643926s) [upload=447525]: Upload finished. err: <nil> Duration: 30.002429019s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 15.407059677s trace: https://jaeger.dev.storj.io/trace/6d2950019117adaf FAIL (2025-12-26 21:28:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Rj4hUGWvUPuU49QDsHYhibiJhyY5KDGus1rzNeP7KusgyvTsU, Piece ID: NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A) context deadline exceeded; piecestore: (Node ID: 12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, Piece ID: N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ) context deadline exceeded; piecestore: (Node ID: 12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, Piece ID: IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ) context deadline exceeded ======LOGS====== ng piece... Dec 26 20:54:08.087336805 (509h25m7.95841984s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087649520 (509h25m7.958732565s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087988717 (509h25m7.959071752s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088323233 (509h25m7.959406268s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088877028 (509h25m7.959960063s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089247203 (509h25m7.960330238s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089543066 (509h25m7.960626101s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089914133 (509h25m7.960997168s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090327259 (509h25m7.961410294s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090645345 (509h25m7.96172838s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091004318 (509h25m7.962087353s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091338976 (509h25m7.962422021s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091732054 (509h25m7.962815099s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:09.565187143 (509h25m9.436270188s) [upload=447022]: Next returned nil segment Dec 26 20:54:09.565201321 (509h25m9.436284356s) [upload=447022]: Waiting for error group managing segments... Dec 26 20:54:10.137065972 (509h25m10.008149007s) [noise=true , address=77.90.31.10:4523 , piece_id=GQYT3Z53Y6HVTY3IBUNFCIWBEE7KVAUWEJYO7QJFRRAJCHTJMYFA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.137147197 (509h25m10.008230232s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.149325196 (509h25m10.020408242s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.159611673 (509h25m10.030694718s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.164411764 (509h25m10.035494809s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.190924278 (509h25m10.062007313s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.232931086 (509h25m10.104014131s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.241658245 (509h25m10.11274129s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.472574370 (509h25m10.343657405s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.866910566 (509h25m10.737993611s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.886583507 (509h25m10.757666542s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.127773215 (509h25m10.99885626s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.192133217 (509h25m11.063216263s) [noise=true , address=145.79.227.13:4819 , piece_id=BQDRL2QJZ75TAMUM2VSGTF4U3NEMZC4FFQLN6QV6UOAFHWB56MZA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256146219 (509h25m11.127229264s) [noise=true , address=145.79.227.40:4062 , piece_id=IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256176597 (509h25m11.127259632s) [seg_pos={0 0}, upload=447022]: Segment reached optimal threshold of 14 pieces. Dec 26 20:54:11.256428137 (509h25m11.127511192s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98): protocol: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection Dec 26 20:54:11.256439549 (509h25m11.127522584s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256458915 (509h25m11.12754198s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF): protocol: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection Dec 26 20:54:11.256490705 (509h25m11.127573741s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256698171 (509h25m11.127781216s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: rpc: dial tcp 145.79.227.30:4796: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:54:11.256706457 (509h25m11.127789502s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256917309 (509h25m11.128000364s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection Dec 26 20:54:11.256926666 (509h25m11.128009711s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256964869 (509h25m11.128047914s) [seg_pos={0 0}, upload=447022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:54:11.257064138 (509h25m11.128147173s) [upload=447022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:11.257070690 (509h25m11.128153725s) [upload=447022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:11.561962810 (509h25m11.433045875s) [upload=447022]: Done uploading segments... Dec 26 20:54:11.561987427 (509h25m11.433070472s) [upload=447022]: Upload finished. err: <nil> Duration: 15.407095146s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..............X....................XX.XX.X..X.X...XX.X......] duration: 5.781914838s trace: https://jaeger.dev.storj.io/trace/2c039742adb0e5fc FAIL (2025-12-26 21:28:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: DZ7L6AH2YZZQVPTHZMYX3SLCD5SXQVOIIJWKWQSR5OID6PZDPCFA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: EXHZ4FJ373LRHLSAIOKRCJKAKVGKDP6LCGFVLDYPF7CQ3VUGEANA) context deadline exceeded ======LOGS====== .. Dec 26 21:24:15.239493539 (509h55m15.110576574s) [upload=447461]: Got next segment. Inline: false Dec 26 21:24:15.239528506 (509h55m15.110611541s) [upload=447461]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 21:24:15.239534708 (509h55m15.110617743s) [upload=447461]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 21:24:15.490306572 (509h55m15.361389627s) [seg_pos={0 0}, upload=447461]: Begin upload segment... Dec 26 21:24:15.490361826 (509h55m15.361444871s) [seg_pos={0 0}, upload=447461]: Done begin upload segment. Dec 26 21:24:15.490401131 (509h55m15.361484166s) [noise=true , address=145.79.227.16:4912 , piece_id=2GXBYHN3YFLX5VIMG6TL5UBIXKAPIZKZ5HD6LJDJB24QW6BEBRYQ, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.490473849 (509h55m15.361556884s) [noise=true , address=77.90.31.15:4700 , piece_id=3SGR6HJ6YQTU4RZNQJGVQPDY5WH7KBJE4ORBXFNGPJM2YNUST6LA, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.490989091 (509h55m15.362072126s) [noise=true , address=77.90.31.10:4518 , piece_id=DZ7L6AH2YZZQVPTHZMYX3SLCD5SXQVOIIJWKWQSR5OID6PZDPCFA, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.491557553 (509h55m15.362640588s) [noise=true , address=145.79.227.9:4365 , piece_id=N5Z2HE3UXZVZFQLVZ42Q2C5T5NQQXPNTPCFU4EA3OPJGCH2PAJ2A, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.491880669 (509h55m15.362963714s) [noise=true , address=145.79.227.49:4337 , piece_id=64P7SZSDDNLR73RFCSELTS6VWZWPX742RRVPUOHL6EFQEVOMQQEA, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.492165521 (509h55m15.363248566s) [noise=true , address=145.79.227.37:4566 , piece_id=43XMAK5WQ24JAKCERKDQXYYFGNXTKHGGAUYEGHJ27DPJMNLRNXWQ, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.492509296 (509h55m15.363592331s) [noise=true , address=45.91.116.12:4226 , piece_id=FXJZP7ZBSEMQDLAEQD7EGKS5BRJJEHPRMARPTVM5F3LLWGSDH7GQ, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.492806461 (509h55m15.363889506s) [noise=true , address=145.79.227.36:4517 , piece_id=QKBV5M2AOSQT2KGSF3BEO7AGKTJ6DIOKUTIWYEKE674Z33H5JDJQ, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.493140848 (509h55m15.364223883s) [noise=true , address=145.79.227.5:4223 , piece_id=OELI44LN4UYNTG2PRI6YJYPTMHGSC2ULVODVE4WYVRTIME6YN7OA, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.494898515 (509h55m15.36598156s) [noise=true , address=45.91.116.14:4191 , piece_id=RE45TDF3GHDCIWRSB7FWZP5CW54RSOI4UGRAQXNVGUG4BGE5OGJA, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.496100484 (509h55m15.367183519s) [noise=true , address=145.79.227.50:4375 , piece_id=V6Z7QHRDZZKVUL5TBUFEU3S4LYEJLACJI4JWOHO6DUMHS7KTQFSA, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.497211439 (509h55m15.368294474s) [noise=true , address=45.91.116.9:4262 , piece_id=XHXR64XKMM4KVFCKYZKJT3XKAAOSIU7TOFHAPCIKAXZZOC336TZQ, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.498489914 (509h55m15.369572959s) [noise=true , address=145.79.227.48:4302 , piece_id=YWW2ZX6H2GA6AL7LGAJAAQ6Q277CE6WP7S673X75PUF5H7HGTN5A, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.499668979 (509h55m15.370752014s) [noise=true , address=77.90.31.19:4844 , piece_id=MTCGLEGNSJMXRIJVFOEY5FUD72V6PRRFQEKXHPCDWS3GJXTYNJSQ, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.500697898 (509h55m15.371780933s) [noise=true , address=45.91.116.13:4156 , piece_id=EXHZ4FJ373LRHLSAIOKRCJKAKVGKDP6LCGFVLDYPF7CQ3VUGEANA, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.501897282 (509h55m15.372980317s) [noise=true , address=145.79.227.43:4159 , piece_id=6VJHRFVXRQYIRFQXLLQRJE7YDHCGHSON5BQTRS3RYWRTPZDWIH6A, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:17.194827862 (509h55m17.065910897s) [upload=447461]: Next returned nil segment Dec 26 21:24:17.194840907 (509h55m17.065923952s) [upload=447461]: Waiting for error group managing segments... Dec 26 21:24:17.480615983 (509h55m17.351699028s) [noise=true , address=145.79.227.9:4365 , piece_id=N5Z2HE3UXZVZFQLVZ42Q2C5T5NQQXPNTPCFU4EA3OPJGCH2PAJ2A, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.488188001 (509h55m17.359271046s) [noise=true , address=145.79.227.37:4566 , piece_id=43XMAK5WQ24JAKCERKDQXYYFGNXTKHGGAUYEGHJ27DPJMNLRNXWQ, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.510872557 (509h55m17.381955592s) [noise=true , address=145.79.227.48:4302 , piece_id=YWW2ZX6H2GA6AL7LGAJAAQ6Q277CE6WP7S673X75PUF5H7HGTN5A, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.512796540 (509h55m17.383879585s) [noise=true , address=145.79.227.49:4337 , piece_id=64P7SZSDDNLR73RFCSELTS6VWZWPX742RRVPUOHL6EFQEVOMQQEA, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.519834743 (509h55m17.390917778s) [noise=true , address=145.79.227.36:4517 , piece_id=QKBV5M2AOSQT2KGSF3BEO7AGKTJ6DIOKUTIWYEKE674Z33H5JDJQ, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.520424034 (509h55m17.39150708s) [noise=true , address=145.79.227.50:4375 , piece_id=V6Z7QHRDZZKVUL5TBUFEU3S4LYEJLACJI4JWOHO6DUMHS7KTQFSA, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.525181103 (509h55m17.396264138s) [noise=true , address=145.79.227.43:4159 , piece_id=6VJHRFVXRQYIRFQXLLQRJE7YDHCGHSON5BQTRS3RYWRTPZDWIH6A, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.666984621 (509h55m17.538067657s) [noise=true , address=77.90.31.10:4518 , piece_id=DZ7L6AH2YZZQVPTHZMYX3SLCD5SXQVOIIJWKWQSR5OID6PZDPCFA, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.723629614 (509h55m17.594712649s) [noise=true , address=77.90.31.15:4700 , piece_id=3SGR6HJ6YQTU4RZNQJGVQPDY5WH7KBJE4ORBXFNGPJM2YNUST6LA, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.786535648 (509h55m17.657618683s) [noise=true , address=45.91.116.13:4156 , piece_id=EXHZ4FJ373LRHLSAIOKRCJKAKVGKDP6LCGFVLDYPF7CQ3VUGEANA, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.798681766 (509h55m17.669764801s) [noise=true , address=77.90.31.19:4844 , piece_id=MTCGLEGNSJMXRIJVFOEY5FUD72V6PRRFQEKXHPCDWS3GJXTYNJSQ, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.850514655 (509h55m17.72159769s) [noise=true , address=45.91.116.14:4191 , piece_id=RE45TDF3GHDCIWRSB7FWZP5CW54RSOI4UGRAQXNVGUG4BGE5OGJA, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.991331216 (509h55m17.862414251s) [noise=true , address=45.91.116.9:4262 , piece_id=XHXR64XKMM4KVFCKYZKJT3XKAAOSIU7TOFHAPCIKAXZZOC336TZQ, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.991350022 (509h55m17.862433057s) [seg_pos={0 0}, upload=447461]: Segment reached optimal threshold of 13 pieces. Dec 26 21:24:17.991545053 (509h55m17.862628088s) [noise=true , address=145.79.227.5:4223 , piece_id=OELI44LN4UYNTG2PRI6YJYPTMHGSC2ULVODVE4WYVRTIME6YN7OA, seg_pos={0 0}, upload=447461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: write tcp 188.166.225.170:33694->145.79.227.5:4223: use of closed network connection; write tcp 188.166.225.170:33694->145.79.227.5:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33694->145.79.227.5:4223: use of closed network connection Dec 26 21:24:17.991554612 (509h55m17.862637647s) [noise=true , address=145.79.227.5:4223 , piece_id=OELI44LN4UYNTG2PRI6YJYPTMHGSC2ULVODVE4WYVRTIME6YN7OA, seg_pos={0 0}, upload=447461]: Long tail cancellation due to optimal threshold Dec 26 21:24:17.991690420 (509h55m17.862773455s) [noise=true , address=145.79.227.16:4912 , piece_id=2GXBYHN3YFLX5VIMG6TL5UBIXKAPIZKZ5HD6LJDJB24QW6BEBRYQ, seg_pos={0 0}, upload=447461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR): protocol: write tcp 188.166.225.170:37658->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:37658->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37658->145.79.227.16:4912: use of closed network connection Dec 26 21:24:17.991699878 (509h55m17.862782913s) [noise=true , address=145.79.227.16:4912 , piece_id=2GXBYHN3YFLX5VIMG6TL5UBIXKAPIZKZ5HD6LJDJB24QW6BEBRYQ, seg_pos={0 0}, upload=447461]: Long tail cancellation due to optimal threshold Dec 26 21:24:17.991825707 (509h55m17.862908742s) [noise=true , address=45.91.116.12:4226 , piece_id=FXJZP7ZBSEMQDLAEQD7EGKS5BRJJEHPRMARPTVM5F3LLWGSDH7GQ, seg_pos={0 0}, upload=447461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 26 21:24:17.991831169 (509h55m17.862914204s) [noise=true , address=45.91.116.12:4226 , piece_id=FXJZP7ZBSEMQDLAEQD7EGKS5BRJJEHPRMARPTVM5F3LLWGSDH7GQ, seg_pos={0 0}, upload=447461]: Long tail cancellation due to optimal threshold Dec 26 21:24:17.991846698 (509h55m17.862929733s) [seg_pos={0 0}, upload=447461]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 21:24:17.991908355 (509h55m17.86299139s) [upload=447461]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:24:17.991912844 (509h55m17.862995879s) [upload=447461]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:24:18.293828483 (509h55m18.164911518s) [upload=447461]: Done uploading segments... Dec 26 21:24:18.293846899 (509h55m18.164929934s) [upload=447461]: Upload finished. err: <nil> Duration: 5.781955216s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 26.065077957s trace: https://jaeger.dev.storj.io/trace/6d2950019117adaf FAIL (2025-12-26 21:28:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Rj4hUGWvUPuU49QDsHYhibiJhyY5KDGus1rzNeP7KusgyvTsU, Piece ID: NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A) context deadline exceeded; piecestore: (Node ID: 12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, Piece ID: N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ) context deadline exceeded; piecestore: (Node ID: 12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, Piece ID: IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ) context deadline exceeded ======LOGS====== ng piece... Dec 26 20:54:08.087336805 (509h25m7.95841984s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087649520 (509h25m7.958732565s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087988717 (509h25m7.959071752s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088323233 (509h25m7.959406268s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088877028 (509h25m7.959960063s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089247203 (509h25m7.960330238s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089543066 (509h25m7.960626101s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089914133 (509h25m7.960997168s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090327259 (509h25m7.961410294s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090645345 (509h25m7.96172838s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091004318 (509h25m7.962087353s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091338976 (509h25m7.962422021s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091732054 (509h25m7.962815099s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:09.565187143 (509h25m9.436270188s) [upload=447022]: Next returned nil segment Dec 26 20:54:09.565201321 (509h25m9.436284356s) [upload=447022]: Waiting for error group managing segments... Dec 26 20:54:10.137065972 (509h25m10.008149007s) [noise=true , address=77.90.31.10:4523 , piece_id=GQYT3Z53Y6HVTY3IBUNFCIWBEE7KVAUWEJYO7QJFRRAJCHTJMYFA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.137147197 (509h25m10.008230232s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.149325196 (509h25m10.020408242s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.159611673 (509h25m10.030694718s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.164411764 (509h25m10.035494809s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.190924278 (509h25m10.062007313s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.232931086 (509h25m10.104014131s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.241658245 (509h25m10.11274129s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.472574370 (509h25m10.343657405s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.866910566 (509h25m10.737993611s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.886583507 (509h25m10.757666542s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.127773215 (509h25m10.99885626s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.192133217 (509h25m11.063216263s) [noise=true , address=145.79.227.13:4819 , piece_id=BQDRL2QJZ75TAMUM2VSGTF4U3NEMZC4FFQLN6QV6UOAFHWB56MZA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256146219 (509h25m11.127229264s) [noise=true , address=145.79.227.40:4062 , piece_id=IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256176597 (509h25m11.127259632s) [seg_pos={0 0}, upload=447022]: Segment reached optimal threshold of 14 pieces. Dec 26 20:54:11.256428137 (509h25m11.127511192s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98): protocol: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection Dec 26 20:54:11.256439549 (509h25m11.127522584s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256458915 (509h25m11.12754198s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF): protocol: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection Dec 26 20:54:11.256490705 (509h25m11.127573741s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256698171 (509h25m11.127781216s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: rpc: dial tcp 145.79.227.30:4796: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:54:11.256706457 (509h25m11.127789502s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256917309 (509h25m11.128000364s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection Dec 26 20:54:11.256926666 (509h25m11.128009711s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256964869 (509h25m11.128047914s) [seg_pos={0 0}, upload=447022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:54:11.257064138 (509h25m11.128147173s) [upload=447022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:11.257070690 (509h25m11.128153725s) [upload=447022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:11.561962810 (509h25m11.433045875s) [upload=447022]: Done uploading segments... Dec 26 20:54:11.561987427 (509h25m11.433070472s) [upload=447022]: Upload finished. err: <nil> Duration: 26.065114297s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 21.756783215s trace: https://jaeger.dev.storj.io/trace/6d2950019117adaf FAIL (2025-12-26 21:27:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Rj4hUGWvUPuU49QDsHYhibiJhyY5KDGus1rzNeP7KusgyvTsU, Piece ID: NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A) context deadline exceeded; piecestore: (Node ID: 12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, Piece ID: N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ) context deadline exceeded; piecestore: (Node ID: 12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, Piece ID: IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ) context deadline exceeded ======LOGS====== ng piece... Dec 26 20:54:08.087336805 (509h25m7.95841984s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087649520 (509h25m7.958732565s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087988717 (509h25m7.959071752s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088323233 (509h25m7.959406268s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088877028 (509h25m7.959960063s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089247203 (509h25m7.960330238s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089543066 (509h25m7.960626101s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089914133 (509h25m7.960997168s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090327259 (509h25m7.961410294s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090645345 (509h25m7.96172838s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091004318 (509h25m7.962087353s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091338976 (509h25m7.962422021s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091732054 (509h25m7.962815099s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:09.565187143 (509h25m9.436270188s) [upload=447022]: Next returned nil segment Dec 26 20:54:09.565201321 (509h25m9.436284356s) [upload=447022]: Waiting for error group managing segments... Dec 26 20:54:10.137065972 (509h25m10.008149007s) [noise=true , address=77.90.31.10:4523 , piece_id=GQYT3Z53Y6HVTY3IBUNFCIWBEE7KVAUWEJYO7QJFRRAJCHTJMYFA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.137147197 (509h25m10.008230232s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.149325196 (509h25m10.020408242s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.159611673 (509h25m10.030694718s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.164411764 (509h25m10.035494809s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.190924278 (509h25m10.062007313s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.232931086 (509h25m10.104014131s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.241658245 (509h25m10.11274129s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.472574370 (509h25m10.343657405s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.866910566 (509h25m10.737993611s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.886583507 (509h25m10.757666542s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.127773215 (509h25m10.99885626s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.192133217 (509h25m11.063216263s) [noise=true , address=145.79.227.13:4819 , piece_id=BQDRL2QJZ75TAMUM2VSGTF4U3NEMZC4FFQLN6QV6UOAFHWB56MZA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256146219 (509h25m11.127229264s) [noise=true , address=145.79.227.40:4062 , piece_id=IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256176597 (509h25m11.127259632s) [seg_pos={0 0}, upload=447022]: Segment reached optimal threshold of 14 pieces. Dec 26 20:54:11.256428137 (509h25m11.127511192s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98): protocol: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection Dec 26 20:54:11.256439549 (509h25m11.127522584s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256458915 (509h25m11.12754198s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF): protocol: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection Dec 26 20:54:11.256490705 (509h25m11.127573741s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256698171 (509h25m11.127781216s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: rpc: dial tcp 145.79.227.30:4796: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:54:11.256706457 (509h25m11.127789502s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256917309 (509h25m11.128000364s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection Dec 26 20:54:11.256926666 (509h25m11.128009711s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256964869 (509h25m11.128047914s) [seg_pos={0 0}, upload=447022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:54:11.257064138 (509h25m11.128147173s) [upload=447022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:11.257070690 (509h25m11.128153725s) [upload=447022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:11.561962810 (509h25m11.433045875s) [upload=447022]: Done uploading segments... Dec 26 20:54:11.561987427 (509h25m11.433070472s) [upload=447022]: Upload finished. err: <nil> Duration: 21.756808713s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [...............X....................XX.XX.X..X.X...XX.X.....] duration: 7.385537411s trace: https://jaeger.dev.storj.io/trace/2c039742adb0e5fc FAIL (2025-12-26 21:27:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: DZ7L6AH2YZZQVPTHZMYX3SLCD5SXQVOIIJWKWQSR5OID6PZDPCFA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: EXHZ4FJ373LRHLSAIOKRCJKAKVGKDP6LCGFVLDYPF7CQ3VUGEANA) context deadline exceeded ======LOGS====== .. Dec 26 21:24:15.239493539 (509h55m15.110576574s) [upload=447461]: Got next segment. Inline: false Dec 26 21:24:15.239528506 (509h55m15.110611541s) [upload=447461]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 21:24:15.239534708 (509h55m15.110617743s) [upload=447461]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 21:24:15.490306572 (509h55m15.361389627s) [seg_pos={0 0}, upload=447461]: Begin upload segment... Dec 26 21:24:15.490361826 (509h55m15.361444871s) [seg_pos={0 0}, upload=447461]: Done begin upload segment. Dec 26 21:24:15.490401131 (509h55m15.361484166s) [noise=true , address=145.79.227.16:4912 , piece_id=2GXBYHN3YFLX5VIMG6TL5UBIXKAPIZKZ5HD6LJDJB24QW6BEBRYQ, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.490473849 (509h55m15.361556884s) [noise=true , address=77.90.31.15:4700 , piece_id=3SGR6HJ6YQTU4RZNQJGVQPDY5WH7KBJE4ORBXFNGPJM2YNUST6LA, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.490989091 (509h55m15.362072126s) [noise=true , address=77.90.31.10:4518 , piece_id=DZ7L6AH2YZZQVPTHZMYX3SLCD5SXQVOIIJWKWQSR5OID6PZDPCFA, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.491557553 (509h55m15.362640588s) [noise=true , address=145.79.227.9:4365 , piece_id=N5Z2HE3UXZVZFQLVZ42Q2C5T5NQQXPNTPCFU4EA3OPJGCH2PAJ2A, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.491880669 (509h55m15.362963714s) [noise=true , address=145.79.227.49:4337 , piece_id=64P7SZSDDNLR73RFCSELTS6VWZWPX742RRVPUOHL6EFQEVOMQQEA, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.492165521 (509h55m15.363248566s) [noise=true , address=145.79.227.37:4566 , piece_id=43XMAK5WQ24JAKCERKDQXYYFGNXTKHGGAUYEGHJ27DPJMNLRNXWQ, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.492509296 (509h55m15.363592331s) [noise=true , address=45.91.116.12:4226 , piece_id=FXJZP7ZBSEMQDLAEQD7EGKS5BRJJEHPRMARPTVM5F3LLWGSDH7GQ, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.492806461 (509h55m15.363889506s) [noise=true , address=145.79.227.36:4517 , piece_id=QKBV5M2AOSQT2KGSF3BEO7AGKTJ6DIOKUTIWYEKE674Z33H5JDJQ, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.493140848 (509h55m15.364223883s) [noise=true , address=145.79.227.5:4223 , piece_id=OELI44LN4UYNTG2PRI6YJYPTMHGSC2ULVODVE4WYVRTIME6YN7OA, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.494898515 (509h55m15.36598156s) [noise=true , address=45.91.116.14:4191 , piece_id=RE45TDF3GHDCIWRSB7FWZP5CW54RSOI4UGRAQXNVGUG4BGE5OGJA, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.496100484 (509h55m15.367183519s) [noise=true , address=145.79.227.50:4375 , piece_id=V6Z7QHRDZZKVUL5TBUFEU3S4LYEJLACJI4JWOHO6DUMHS7KTQFSA, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.497211439 (509h55m15.368294474s) [noise=true , address=45.91.116.9:4262 , piece_id=XHXR64XKMM4KVFCKYZKJT3XKAAOSIU7TOFHAPCIKAXZZOC336TZQ, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.498489914 (509h55m15.369572959s) [noise=true , address=145.79.227.48:4302 , piece_id=YWW2ZX6H2GA6AL7LGAJAAQ6Q277CE6WP7S673X75PUF5H7HGTN5A, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.499668979 (509h55m15.370752014s) [noise=true , address=77.90.31.19:4844 , piece_id=MTCGLEGNSJMXRIJVFOEY5FUD72V6PRRFQEKXHPCDWS3GJXTYNJSQ, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.500697898 (509h55m15.371780933s) [noise=true , address=45.91.116.13:4156 , piece_id=EXHZ4FJ373LRHLSAIOKRCJKAKVGKDP6LCGFVLDYPF7CQ3VUGEANA, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.501897282 (509h55m15.372980317s) [noise=true , address=145.79.227.43:4159 , piece_id=6VJHRFVXRQYIRFQXLLQRJE7YDHCGHSON5BQTRS3RYWRTPZDWIH6A, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:17.194827862 (509h55m17.065910897s) [upload=447461]: Next returned nil segment Dec 26 21:24:17.194840907 (509h55m17.065923952s) [upload=447461]: Waiting for error group managing segments... Dec 26 21:24:17.480615983 (509h55m17.351699028s) [noise=true , address=145.79.227.9:4365 , piece_id=N5Z2HE3UXZVZFQLVZ42Q2C5T5NQQXPNTPCFU4EA3OPJGCH2PAJ2A, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.488188001 (509h55m17.359271046s) [noise=true , address=145.79.227.37:4566 , piece_id=43XMAK5WQ24JAKCERKDQXYYFGNXTKHGGAUYEGHJ27DPJMNLRNXWQ, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.510872557 (509h55m17.381955592s) [noise=true , address=145.79.227.48:4302 , piece_id=YWW2ZX6H2GA6AL7LGAJAAQ6Q277CE6WP7S673X75PUF5H7HGTN5A, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.512796540 (509h55m17.383879585s) [noise=true , address=145.79.227.49:4337 , piece_id=64P7SZSDDNLR73RFCSELTS6VWZWPX742RRVPUOHL6EFQEVOMQQEA, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.519834743 (509h55m17.390917778s) [noise=true , address=145.79.227.36:4517 , piece_id=QKBV5M2AOSQT2KGSF3BEO7AGKTJ6DIOKUTIWYEKE674Z33H5JDJQ, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.520424034 (509h55m17.39150708s) [noise=true , address=145.79.227.50:4375 , piece_id=V6Z7QHRDZZKVUL5TBUFEU3S4LYEJLACJI4JWOHO6DUMHS7KTQFSA, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.525181103 (509h55m17.396264138s) [noise=true , address=145.79.227.43:4159 , piece_id=6VJHRFVXRQYIRFQXLLQRJE7YDHCGHSON5BQTRS3RYWRTPZDWIH6A, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.666984621 (509h55m17.538067657s) [noise=true , address=77.90.31.10:4518 , piece_id=DZ7L6AH2YZZQVPTHZMYX3SLCD5SXQVOIIJWKWQSR5OID6PZDPCFA, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.723629614 (509h55m17.594712649s) [noise=true , address=77.90.31.15:4700 , piece_id=3SGR6HJ6YQTU4RZNQJGVQPDY5WH7KBJE4ORBXFNGPJM2YNUST6LA, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.786535648 (509h55m17.657618683s) [noise=true , address=45.91.116.13:4156 , piece_id=EXHZ4FJ373LRHLSAIOKRCJKAKVGKDP6LCGFVLDYPF7CQ3VUGEANA, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.798681766 (509h55m17.669764801s) [noise=true , address=77.90.31.19:4844 , piece_id=MTCGLEGNSJMXRIJVFOEY5FUD72V6PRRFQEKXHPCDWS3GJXTYNJSQ, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.850514655 (509h55m17.72159769s) [noise=true , address=45.91.116.14:4191 , piece_id=RE45TDF3GHDCIWRSB7FWZP5CW54RSOI4UGRAQXNVGUG4BGE5OGJA, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.991331216 (509h55m17.862414251s) [noise=true , address=45.91.116.9:4262 , piece_id=XHXR64XKMM4KVFCKYZKJT3XKAAOSIU7TOFHAPCIKAXZZOC336TZQ, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.991350022 (509h55m17.862433057s) [seg_pos={0 0}, upload=447461]: Segment reached optimal threshold of 13 pieces. Dec 26 21:24:17.991545053 (509h55m17.862628088s) [noise=true , address=145.79.227.5:4223 , piece_id=OELI44LN4UYNTG2PRI6YJYPTMHGSC2ULVODVE4WYVRTIME6YN7OA, seg_pos={0 0}, upload=447461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: write tcp 188.166.225.170:33694->145.79.227.5:4223: use of closed network connection; write tcp 188.166.225.170:33694->145.79.227.5:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33694->145.79.227.5:4223: use of closed network connection Dec 26 21:24:17.991554612 (509h55m17.862637647s) [noise=true , address=145.79.227.5:4223 , piece_id=OELI44LN4UYNTG2PRI6YJYPTMHGSC2ULVODVE4WYVRTIME6YN7OA, seg_pos={0 0}, upload=447461]: Long tail cancellation due to optimal threshold Dec 26 21:24:17.991690420 (509h55m17.862773455s) [noise=true , address=145.79.227.16:4912 , piece_id=2GXBYHN3YFLX5VIMG6TL5UBIXKAPIZKZ5HD6LJDJB24QW6BEBRYQ, seg_pos={0 0}, upload=447461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR): protocol: write tcp 188.166.225.170:37658->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:37658->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37658->145.79.227.16:4912: use of closed network connection Dec 26 21:24:17.991699878 (509h55m17.862782913s) [noise=true , address=145.79.227.16:4912 , piece_id=2GXBYHN3YFLX5VIMG6TL5UBIXKAPIZKZ5HD6LJDJB24QW6BEBRYQ, seg_pos={0 0}, upload=447461]: Long tail cancellation due to optimal threshold Dec 26 21:24:17.991825707 (509h55m17.862908742s) [noise=true , address=45.91.116.12:4226 , piece_id=FXJZP7ZBSEMQDLAEQD7EGKS5BRJJEHPRMARPTVM5F3LLWGSDH7GQ, seg_pos={0 0}, upload=447461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 26 21:24:17.991831169 (509h55m17.862914204s) [noise=true , address=45.91.116.12:4226 , piece_id=FXJZP7ZBSEMQDLAEQD7EGKS5BRJJEHPRMARPTVM5F3LLWGSDH7GQ, seg_pos={0 0}, upload=447461]: Long tail cancellation due to optimal threshold Dec 26 21:24:17.991846698 (509h55m17.862929733s) [seg_pos={0 0}, upload=447461]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 21:24:17.991908355 (509h55m17.86299139s) [upload=447461]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:24:17.991912844 (509h55m17.862995879s) [upload=447461]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:24:18.293828483 (509h55m18.164911518s) [upload=447461]: Done uploading segments... Dec 26 21:24:18.293846899 (509h55m18.164929934s) [upload=447461]: Upload finished. err: <nil> Duration: 7.385570343s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X...............X....................XX.XX.X..X.X...XX.X....] duration: 6.353213923s trace: https://jaeger.dev.storj.io/trace/2c039742adb0e5fc FAIL (2025-12-26 21:26:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: DZ7L6AH2YZZQVPTHZMYX3SLCD5SXQVOIIJWKWQSR5OID6PZDPCFA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: EXHZ4FJ373LRHLSAIOKRCJKAKVGKDP6LCGFVLDYPF7CQ3VUGEANA) context deadline exceeded ======LOGS====== .. Dec 26 21:24:15.239493539 (509h55m15.110576574s) [upload=447461]: Got next segment. Inline: false Dec 26 21:24:15.239528506 (509h55m15.110611541s) [upload=447461]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 21:24:15.239534708 (509h55m15.110617743s) [upload=447461]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 21:24:15.490306572 (509h55m15.361389627s) [seg_pos={0 0}, upload=447461]: Begin upload segment... Dec 26 21:24:15.490361826 (509h55m15.361444871s) [seg_pos={0 0}, upload=447461]: Done begin upload segment. Dec 26 21:24:15.490401131 (509h55m15.361484166s) [noise=true , address=145.79.227.16:4912 , piece_id=2GXBYHN3YFLX5VIMG6TL5UBIXKAPIZKZ5HD6LJDJB24QW6BEBRYQ, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.490473849 (509h55m15.361556884s) [noise=true , address=77.90.31.15:4700 , piece_id=3SGR6HJ6YQTU4RZNQJGVQPDY5WH7KBJE4ORBXFNGPJM2YNUST6LA, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.490989091 (509h55m15.362072126s) [noise=true , address=77.90.31.10:4518 , piece_id=DZ7L6AH2YZZQVPTHZMYX3SLCD5SXQVOIIJWKWQSR5OID6PZDPCFA, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.491557553 (509h55m15.362640588s) [noise=true , address=145.79.227.9:4365 , piece_id=N5Z2HE3UXZVZFQLVZ42Q2C5T5NQQXPNTPCFU4EA3OPJGCH2PAJ2A, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.491880669 (509h55m15.362963714s) [noise=true , address=145.79.227.49:4337 , piece_id=64P7SZSDDNLR73RFCSELTS6VWZWPX742RRVPUOHL6EFQEVOMQQEA, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.492165521 (509h55m15.363248566s) [noise=true , address=145.79.227.37:4566 , piece_id=43XMAK5WQ24JAKCERKDQXYYFGNXTKHGGAUYEGHJ27DPJMNLRNXWQ, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.492509296 (509h55m15.363592331s) [noise=true , address=45.91.116.12:4226 , piece_id=FXJZP7ZBSEMQDLAEQD7EGKS5BRJJEHPRMARPTVM5F3LLWGSDH7GQ, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.492806461 (509h55m15.363889506s) [noise=true , address=145.79.227.36:4517 , piece_id=QKBV5M2AOSQT2KGSF3BEO7AGKTJ6DIOKUTIWYEKE674Z33H5JDJQ, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.493140848 (509h55m15.364223883s) [noise=true , address=145.79.227.5:4223 , piece_id=OELI44LN4UYNTG2PRI6YJYPTMHGSC2ULVODVE4WYVRTIME6YN7OA, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.494898515 (509h55m15.36598156s) [noise=true , address=45.91.116.14:4191 , piece_id=RE45TDF3GHDCIWRSB7FWZP5CW54RSOI4UGRAQXNVGUG4BGE5OGJA, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.496100484 (509h55m15.367183519s) [noise=true , address=145.79.227.50:4375 , piece_id=V6Z7QHRDZZKVUL5TBUFEU3S4LYEJLACJI4JWOHO6DUMHS7KTQFSA, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.497211439 (509h55m15.368294474s) [noise=true , address=45.91.116.9:4262 , piece_id=XHXR64XKMM4KVFCKYZKJT3XKAAOSIU7TOFHAPCIKAXZZOC336TZQ, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.498489914 (509h55m15.369572959s) [noise=true , address=145.79.227.48:4302 , piece_id=YWW2ZX6H2GA6AL7LGAJAAQ6Q277CE6WP7S673X75PUF5H7HGTN5A, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.499668979 (509h55m15.370752014s) [noise=true , address=77.90.31.19:4844 , piece_id=MTCGLEGNSJMXRIJVFOEY5FUD72V6PRRFQEKXHPCDWS3GJXTYNJSQ, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.500697898 (509h55m15.371780933s) [noise=true , address=45.91.116.13:4156 , piece_id=EXHZ4FJ373LRHLSAIOKRCJKAKVGKDP6LCGFVLDYPF7CQ3VUGEANA, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.501897282 (509h55m15.372980317s) [noise=true , address=145.79.227.43:4159 , piece_id=6VJHRFVXRQYIRFQXLLQRJE7YDHCGHSON5BQTRS3RYWRTPZDWIH6A, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:17.194827862 (509h55m17.065910897s) [upload=447461]: Next returned nil segment Dec 26 21:24:17.194840907 (509h55m17.065923952s) [upload=447461]: Waiting for error group managing segments... Dec 26 21:24:17.480615983 (509h55m17.351699028s) [noise=true , address=145.79.227.9:4365 , piece_id=N5Z2HE3UXZVZFQLVZ42Q2C5T5NQQXPNTPCFU4EA3OPJGCH2PAJ2A, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.488188001 (509h55m17.359271046s) [noise=true , address=145.79.227.37:4566 , piece_id=43XMAK5WQ24JAKCERKDQXYYFGNXTKHGGAUYEGHJ27DPJMNLRNXWQ, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.510872557 (509h55m17.381955592s) [noise=true , address=145.79.227.48:4302 , piece_id=YWW2ZX6H2GA6AL7LGAJAAQ6Q277CE6WP7S673X75PUF5H7HGTN5A, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.512796540 (509h55m17.383879585s) [noise=true , address=145.79.227.49:4337 , piece_id=64P7SZSDDNLR73RFCSELTS6VWZWPX742RRVPUOHL6EFQEVOMQQEA, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.519834743 (509h55m17.390917778s) [noise=true , address=145.79.227.36:4517 , piece_id=QKBV5M2AOSQT2KGSF3BEO7AGKTJ6DIOKUTIWYEKE674Z33H5JDJQ, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.520424034 (509h55m17.39150708s) [noise=true , address=145.79.227.50:4375 , piece_id=V6Z7QHRDZZKVUL5TBUFEU3S4LYEJLACJI4JWOHO6DUMHS7KTQFSA, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.525181103 (509h55m17.396264138s) [noise=true , address=145.79.227.43:4159 , piece_id=6VJHRFVXRQYIRFQXLLQRJE7YDHCGHSON5BQTRS3RYWRTPZDWIH6A, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.666984621 (509h55m17.538067657s) [noise=true , address=77.90.31.10:4518 , piece_id=DZ7L6AH2YZZQVPTHZMYX3SLCD5SXQVOIIJWKWQSR5OID6PZDPCFA, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.723629614 (509h55m17.594712649s) [noise=true , address=77.90.31.15:4700 , piece_id=3SGR6HJ6YQTU4RZNQJGVQPDY5WH7KBJE4ORBXFNGPJM2YNUST6LA, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.786535648 (509h55m17.657618683s) [noise=true , address=45.91.116.13:4156 , piece_id=EXHZ4FJ373LRHLSAIOKRCJKAKVGKDP6LCGFVLDYPF7CQ3VUGEANA, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.798681766 (509h55m17.669764801s) [noise=true , address=77.90.31.19:4844 , piece_id=MTCGLEGNSJMXRIJVFOEY5FUD72V6PRRFQEKXHPCDWS3GJXTYNJSQ, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.850514655 (509h55m17.72159769s) [noise=true , address=45.91.116.14:4191 , piece_id=RE45TDF3GHDCIWRSB7FWZP5CW54RSOI4UGRAQXNVGUG4BGE5OGJA, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.991331216 (509h55m17.862414251s) [noise=true , address=45.91.116.9:4262 , piece_id=XHXR64XKMM4KVFCKYZKJT3XKAAOSIU7TOFHAPCIKAXZZOC336TZQ, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.991350022 (509h55m17.862433057s) [seg_pos={0 0}, upload=447461]: Segment reached optimal threshold of 13 pieces. Dec 26 21:24:17.991545053 (509h55m17.862628088s) [noise=true , address=145.79.227.5:4223 , piece_id=OELI44LN4UYNTG2PRI6YJYPTMHGSC2ULVODVE4WYVRTIME6YN7OA, seg_pos={0 0}, upload=447461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: write tcp 188.166.225.170:33694->145.79.227.5:4223: use of closed network connection; write tcp 188.166.225.170:33694->145.79.227.5:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33694->145.79.227.5:4223: use of closed network connection Dec 26 21:24:17.991554612 (509h55m17.862637647s) [noise=true , address=145.79.227.5:4223 , piece_id=OELI44LN4UYNTG2PRI6YJYPTMHGSC2ULVODVE4WYVRTIME6YN7OA, seg_pos={0 0}, upload=447461]: Long tail cancellation due to optimal threshold Dec 26 21:24:17.991690420 (509h55m17.862773455s) [noise=true , address=145.79.227.16:4912 , piece_id=2GXBYHN3YFLX5VIMG6TL5UBIXKAPIZKZ5HD6LJDJB24QW6BEBRYQ, seg_pos={0 0}, upload=447461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR): protocol: write tcp 188.166.225.170:37658->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:37658->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37658->145.79.227.16:4912: use of closed network connection Dec 26 21:24:17.991699878 (509h55m17.862782913s) [noise=true , address=145.79.227.16:4912 , piece_id=2GXBYHN3YFLX5VIMG6TL5UBIXKAPIZKZ5HD6LJDJB24QW6BEBRYQ, seg_pos={0 0}, upload=447461]: Long tail cancellation due to optimal threshold Dec 26 21:24:17.991825707 (509h55m17.862908742s) [noise=true , address=45.91.116.12:4226 , piece_id=FXJZP7ZBSEMQDLAEQD7EGKS5BRJJEHPRMARPTVM5F3LLWGSDH7GQ, seg_pos={0 0}, upload=447461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 26 21:24:17.991831169 (509h55m17.862914204s) [noise=true , address=45.91.116.12:4226 , piece_id=FXJZP7ZBSEMQDLAEQD7EGKS5BRJJEHPRMARPTVM5F3LLWGSDH7GQ, seg_pos={0 0}, upload=447461]: Long tail cancellation due to optimal threshold Dec 26 21:24:17.991846698 (509h55m17.862929733s) [seg_pos={0 0}, upload=447461]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 21:24:17.991908355 (509h55m17.86299139s) [upload=447461]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:24:17.991912844 (509h55m17.862995879s) [upload=447461]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:24:18.293828483 (509h55m18.164911518s) [upload=447461]: Done uploading segments... Dec 26 21:24:18.293846899 (509h55m18.164929934s) [upload=447461]: Upload finished. err: <nil> Duration: 6.353236817s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 20.17182822s trace: https://jaeger.dev.storj.io/trace/6d2950019117adaf FAIL (2025-12-26 21:26:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Rj4hUGWvUPuU49QDsHYhibiJhyY5KDGus1rzNeP7KusgyvTsU, Piece ID: NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A) context deadline exceeded; piecestore: (Node ID: 12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, Piece ID: N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ) context deadline exceeded; piecestore: (Node ID: 12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, Piece ID: IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ) context deadline exceeded ======LOGS====== ng piece... Dec 26 20:54:08.087336805 (509h25m7.95841984s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087649520 (509h25m7.958732565s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087988717 (509h25m7.959071752s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088323233 (509h25m7.959406268s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088877028 (509h25m7.959960063s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089247203 (509h25m7.960330238s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089543066 (509h25m7.960626101s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089914133 (509h25m7.960997168s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090327259 (509h25m7.961410294s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090645345 (509h25m7.96172838s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091004318 (509h25m7.962087353s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091338976 (509h25m7.962422021s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091732054 (509h25m7.962815099s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:09.565187143 (509h25m9.436270188s) [upload=447022]: Next returned nil segment Dec 26 20:54:09.565201321 (509h25m9.436284356s) [upload=447022]: Waiting for error group managing segments... Dec 26 20:54:10.137065972 (509h25m10.008149007s) [noise=true , address=77.90.31.10:4523 , piece_id=GQYT3Z53Y6HVTY3IBUNFCIWBEE7KVAUWEJYO7QJFRRAJCHTJMYFA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.137147197 (509h25m10.008230232s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.149325196 (509h25m10.020408242s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.159611673 (509h25m10.030694718s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.164411764 (509h25m10.035494809s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.190924278 (509h25m10.062007313s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.232931086 (509h25m10.104014131s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.241658245 (509h25m10.11274129s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.472574370 (509h25m10.343657405s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.866910566 (509h25m10.737993611s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.886583507 (509h25m10.757666542s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.127773215 (509h25m10.99885626s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.192133217 (509h25m11.063216263s) [noise=true , address=145.79.227.13:4819 , piece_id=BQDRL2QJZ75TAMUM2VSGTF4U3NEMZC4FFQLN6QV6UOAFHWB56MZA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256146219 (509h25m11.127229264s) [noise=true , address=145.79.227.40:4062 , piece_id=IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256176597 (509h25m11.127259632s) [seg_pos={0 0}, upload=447022]: Segment reached optimal threshold of 14 pieces. Dec 26 20:54:11.256428137 (509h25m11.127511192s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98): protocol: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection Dec 26 20:54:11.256439549 (509h25m11.127522584s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256458915 (509h25m11.12754198s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF): protocol: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection Dec 26 20:54:11.256490705 (509h25m11.127573741s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256698171 (509h25m11.127781216s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: rpc: dial tcp 145.79.227.30:4796: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:54:11.256706457 (509h25m11.127789502s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256917309 (509h25m11.128000364s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection Dec 26 20:54:11.256926666 (509h25m11.128009711s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256964869 (509h25m11.128047914s) [seg_pos={0 0}, upload=447022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:54:11.257064138 (509h25m11.128147173s) [upload=447022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:11.257070690 (509h25m11.128153725s) [upload=447022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:11.561962810 (509h25m11.433045875s) [upload=447022]: Done uploading segments... Dec 26 20:54:11.561987427 (509h25m11.433070472s) [upload=447022]: Upload finished. err: <nil> Duration: 20.171853889s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 14.533400777s trace: https://jaeger.dev.storj.io/trace/6d2950019117adaf FAIL (2025-12-26 21:25:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Rj4hUGWvUPuU49QDsHYhibiJhyY5KDGus1rzNeP7KusgyvTsU, Piece ID: NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A) context deadline exceeded; piecestore: (Node ID: 12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, Piece ID: N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ) context deadline exceeded; piecestore: (Node ID: 12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, Piece ID: IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ) context deadline exceeded ======LOGS====== ng piece... Dec 26 20:54:08.087336805 (509h25m7.95841984s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087649520 (509h25m7.958732565s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087988717 (509h25m7.959071752s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088323233 (509h25m7.959406268s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088877028 (509h25m7.959960063s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089247203 (509h25m7.960330238s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089543066 (509h25m7.960626101s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089914133 (509h25m7.960997168s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090327259 (509h25m7.961410294s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090645345 (509h25m7.96172838s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091004318 (509h25m7.962087353s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091338976 (509h25m7.962422021s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091732054 (509h25m7.962815099s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:09.565187143 (509h25m9.436270188s) [upload=447022]: Next returned nil segment Dec 26 20:54:09.565201321 (509h25m9.436284356s) [upload=447022]: Waiting for error group managing segments... Dec 26 20:54:10.137065972 (509h25m10.008149007s) [noise=true , address=77.90.31.10:4523 , piece_id=GQYT3Z53Y6HVTY3IBUNFCIWBEE7KVAUWEJYO7QJFRRAJCHTJMYFA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.137147197 (509h25m10.008230232s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.149325196 (509h25m10.020408242s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.159611673 (509h25m10.030694718s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.164411764 (509h25m10.035494809s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.190924278 (509h25m10.062007313s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.232931086 (509h25m10.104014131s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.241658245 (509h25m10.11274129s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.472574370 (509h25m10.343657405s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.866910566 (509h25m10.737993611s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.886583507 (509h25m10.757666542s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.127773215 (509h25m10.99885626s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.192133217 (509h25m11.063216263s) [noise=true , address=145.79.227.13:4819 , piece_id=BQDRL2QJZ75TAMUM2VSGTF4U3NEMZC4FFQLN6QV6UOAFHWB56MZA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256146219 (509h25m11.127229264s) [noise=true , address=145.79.227.40:4062 , piece_id=IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256176597 (509h25m11.127259632s) [seg_pos={0 0}, upload=447022]: Segment reached optimal threshold of 14 pieces. Dec 26 20:54:11.256428137 (509h25m11.127511192s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98): protocol: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection Dec 26 20:54:11.256439549 (509h25m11.127522584s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256458915 (509h25m11.12754198s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF): protocol: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection Dec 26 20:54:11.256490705 (509h25m11.127573741s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256698171 (509h25m11.127781216s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: rpc: dial tcp 145.79.227.30:4796: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:54:11.256706457 (509h25m11.127789502s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256917309 (509h25m11.128000364s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection Dec 26 20:54:11.256926666 (509h25m11.128009711s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256964869 (509h25m11.128047914s) [seg_pos={0 0}, upload=447022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:54:11.257064138 (509h25m11.128147173s) [upload=447022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:11.257070690 (509h25m11.128153725s) [upload=447022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:11.561962810 (509h25m11.433045875s) [upload=447022]: Done uploading segments... Dec 26 20:54:11.561987427 (509h25m11.433070472s) [upload=447022]: Upload finished. err: <nil> Duration: 14.53341871s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [XX...............X....................XX.XX.X..X.X...XX.X...] duration: 6.181101461s trace: https://jaeger.dev.storj.io/trace/2c039742adb0e5fc FAIL (2025-12-26 21:25:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: DZ7L6AH2YZZQVPTHZMYX3SLCD5SXQVOIIJWKWQSR5OID6PZDPCFA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: EXHZ4FJ373LRHLSAIOKRCJKAKVGKDP6LCGFVLDYPF7CQ3VUGEANA) context deadline exceeded ======LOGS====== .. Dec 26 21:24:15.239493539 (509h55m15.110576574s) [upload=447461]: Got next segment. Inline: false Dec 26 21:24:15.239528506 (509h55m15.110611541s) [upload=447461]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 21:24:15.239534708 (509h55m15.110617743s) [upload=447461]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 21:24:15.490306572 (509h55m15.361389627s) [seg_pos={0 0}, upload=447461]: Begin upload segment... Dec 26 21:24:15.490361826 (509h55m15.361444871s) [seg_pos={0 0}, upload=447461]: Done begin upload segment. Dec 26 21:24:15.490401131 (509h55m15.361484166s) [noise=true , address=145.79.227.16:4912 , piece_id=2GXBYHN3YFLX5VIMG6TL5UBIXKAPIZKZ5HD6LJDJB24QW6BEBRYQ, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.490473849 (509h55m15.361556884s) [noise=true , address=77.90.31.15:4700 , piece_id=3SGR6HJ6YQTU4RZNQJGVQPDY5WH7KBJE4ORBXFNGPJM2YNUST6LA, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.490989091 (509h55m15.362072126s) [noise=true , address=77.90.31.10:4518 , piece_id=DZ7L6AH2YZZQVPTHZMYX3SLCD5SXQVOIIJWKWQSR5OID6PZDPCFA, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.491557553 (509h55m15.362640588s) [noise=true , address=145.79.227.9:4365 , piece_id=N5Z2HE3UXZVZFQLVZ42Q2C5T5NQQXPNTPCFU4EA3OPJGCH2PAJ2A, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.491880669 (509h55m15.362963714s) [noise=true , address=145.79.227.49:4337 , piece_id=64P7SZSDDNLR73RFCSELTS6VWZWPX742RRVPUOHL6EFQEVOMQQEA, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.492165521 (509h55m15.363248566s) [noise=true , address=145.79.227.37:4566 , piece_id=43XMAK5WQ24JAKCERKDQXYYFGNXTKHGGAUYEGHJ27DPJMNLRNXWQ, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.492509296 (509h55m15.363592331s) [noise=true , address=45.91.116.12:4226 , piece_id=FXJZP7ZBSEMQDLAEQD7EGKS5BRJJEHPRMARPTVM5F3LLWGSDH7GQ, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.492806461 (509h55m15.363889506s) [noise=true , address=145.79.227.36:4517 , piece_id=QKBV5M2AOSQT2KGSF3BEO7AGKTJ6DIOKUTIWYEKE674Z33H5JDJQ, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.493140848 (509h55m15.364223883s) [noise=true , address=145.79.227.5:4223 , piece_id=OELI44LN4UYNTG2PRI6YJYPTMHGSC2ULVODVE4WYVRTIME6YN7OA, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.494898515 (509h55m15.36598156s) [noise=true , address=45.91.116.14:4191 , piece_id=RE45TDF3GHDCIWRSB7FWZP5CW54RSOI4UGRAQXNVGUG4BGE5OGJA, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.496100484 (509h55m15.367183519s) [noise=true , address=145.79.227.50:4375 , piece_id=V6Z7QHRDZZKVUL5TBUFEU3S4LYEJLACJI4JWOHO6DUMHS7KTQFSA, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.497211439 (509h55m15.368294474s) [noise=true , address=45.91.116.9:4262 , piece_id=XHXR64XKMM4KVFCKYZKJT3XKAAOSIU7TOFHAPCIKAXZZOC336TZQ, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.498489914 (509h55m15.369572959s) [noise=true , address=145.79.227.48:4302 , piece_id=YWW2ZX6H2GA6AL7LGAJAAQ6Q277CE6WP7S673X75PUF5H7HGTN5A, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.499668979 (509h55m15.370752014s) [noise=true , address=77.90.31.19:4844 , piece_id=MTCGLEGNSJMXRIJVFOEY5FUD72V6PRRFQEKXHPCDWS3GJXTYNJSQ, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.500697898 (509h55m15.371780933s) [noise=true , address=45.91.116.13:4156 , piece_id=EXHZ4FJ373LRHLSAIOKRCJKAKVGKDP6LCGFVLDYPF7CQ3VUGEANA, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.501897282 (509h55m15.372980317s) [noise=true , address=145.79.227.43:4159 , piece_id=6VJHRFVXRQYIRFQXLLQRJE7YDHCGHSON5BQTRS3RYWRTPZDWIH6A, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:17.194827862 (509h55m17.065910897s) [upload=447461]: Next returned nil segment Dec 26 21:24:17.194840907 (509h55m17.065923952s) [upload=447461]: Waiting for error group managing segments... Dec 26 21:24:17.480615983 (509h55m17.351699028s) [noise=true , address=145.79.227.9:4365 , piece_id=N5Z2HE3UXZVZFQLVZ42Q2C5T5NQQXPNTPCFU4EA3OPJGCH2PAJ2A, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.488188001 (509h55m17.359271046s) [noise=true , address=145.79.227.37:4566 , piece_id=43XMAK5WQ24JAKCERKDQXYYFGNXTKHGGAUYEGHJ27DPJMNLRNXWQ, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.510872557 (509h55m17.381955592s) [noise=true , address=145.79.227.48:4302 , piece_id=YWW2ZX6H2GA6AL7LGAJAAQ6Q277CE6WP7S673X75PUF5H7HGTN5A, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.512796540 (509h55m17.383879585s) [noise=true , address=145.79.227.49:4337 , piece_id=64P7SZSDDNLR73RFCSELTS6VWZWPX742RRVPUOHL6EFQEVOMQQEA, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.519834743 (509h55m17.390917778s) [noise=true , address=145.79.227.36:4517 , piece_id=QKBV5M2AOSQT2KGSF3BEO7AGKTJ6DIOKUTIWYEKE674Z33H5JDJQ, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.520424034 (509h55m17.39150708s) [noise=true , address=145.79.227.50:4375 , piece_id=V6Z7QHRDZZKVUL5TBUFEU3S4LYEJLACJI4JWOHO6DUMHS7KTQFSA, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.525181103 (509h55m17.396264138s) [noise=true , address=145.79.227.43:4159 , piece_id=6VJHRFVXRQYIRFQXLLQRJE7YDHCGHSON5BQTRS3RYWRTPZDWIH6A, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.666984621 (509h55m17.538067657s) [noise=true , address=77.90.31.10:4518 , piece_id=DZ7L6AH2YZZQVPTHZMYX3SLCD5SXQVOIIJWKWQSR5OID6PZDPCFA, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.723629614 (509h55m17.594712649s) [noise=true , address=77.90.31.15:4700 , piece_id=3SGR6HJ6YQTU4RZNQJGVQPDY5WH7KBJE4ORBXFNGPJM2YNUST6LA, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.786535648 (509h55m17.657618683s) [noise=true , address=45.91.116.13:4156 , piece_id=EXHZ4FJ373LRHLSAIOKRCJKAKVGKDP6LCGFVLDYPF7CQ3VUGEANA, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.798681766 (509h55m17.669764801s) [noise=true , address=77.90.31.19:4844 , piece_id=MTCGLEGNSJMXRIJVFOEY5FUD72V6PRRFQEKXHPCDWS3GJXTYNJSQ, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.850514655 (509h55m17.72159769s) [noise=true , address=45.91.116.14:4191 , piece_id=RE45TDF3GHDCIWRSB7FWZP5CW54RSOI4UGRAQXNVGUG4BGE5OGJA, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.991331216 (509h55m17.862414251s) [noise=true , address=45.91.116.9:4262 , piece_id=XHXR64XKMM4KVFCKYZKJT3XKAAOSIU7TOFHAPCIKAXZZOC336TZQ, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.991350022 (509h55m17.862433057s) [seg_pos={0 0}, upload=447461]: Segment reached optimal threshold of 13 pieces. Dec 26 21:24:17.991545053 (509h55m17.862628088s) [noise=true , address=145.79.227.5:4223 , piece_id=OELI44LN4UYNTG2PRI6YJYPTMHGSC2ULVODVE4WYVRTIME6YN7OA, seg_pos={0 0}, upload=447461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: write tcp 188.166.225.170:33694->145.79.227.5:4223: use of closed network connection; write tcp 188.166.225.170:33694->145.79.227.5:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33694->145.79.227.5:4223: use of closed network connection Dec 26 21:24:17.991554612 (509h55m17.862637647s) [noise=true , address=145.79.227.5:4223 , piece_id=OELI44LN4UYNTG2PRI6YJYPTMHGSC2ULVODVE4WYVRTIME6YN7OA, seg_pos={0 0}, upload=447461]: Long tail cancellation due to optimal threshold Dec 26 21:24:17.991690420 (509h55m17.862773455s) [noise=true , address=145.79.227.16:4912 , piece_id=2GXBYHN3YFLX5VIMG6TL5UBIXKAPIZKZ5HD6LJDJB24QW6BEBRYQ, seg_pos={0 0}, upload=447461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR): protocol: write tcp 188.166.225.170:37658->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:37658->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37658->145.79.227.16:4912: use of closed network connection Dec 26 21:24:17.991699878 (509h55m17.862782913s) [noise=true , address=145.79.227.16:4912 , piece_id=2GXBYHN3YFLX5VIMG6TL5UBIXKAPIZKZ5HD6LJDJB24QW6BEBRYQ, seg_pos={0 0}, upload=447461]: Long tail cancellation due to optimal threshold Dec 26 21:24:17.991825707 (509h55m17.862908742s) [noise=true , address=45.91.116.12:4226 , piece_id=FXJZP7ZBSEMQDLAEQD7EGKS5BRJJEHPRMARPTVM5F3LLWGSDH7GQ, seg_pos={0 0}, upload=447461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 26 21:24:17.991831169 (509h55m17.862914204s) [noise=true , address=45.91.116.12:4226 , piece_id=FXJZP7ZBSEMQDLAEQD7EGKS5BRJJEHPRMARPTVM5F3LLWGSDH7GQ, seg_pos={0 0}, upload=447461]: Long tail cancellation due to optimal threshold Dec 26 21:24:17.991846698 (509h55m17.862929733s) [seg_pos={0 0}, upload=447461]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 21:24:17.991908355 (509h55m17.86299139s) [upload=447461]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:24:17.991912844 (509h55m17.862995879s) [upload=447461]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:24:18.293828483 (509h55m18.164911518s) [upload=447461]: Done uploading segments... Dec 26 21:24:18.293846899 (509h55m18.164929934s) [upload=447461]: Upload finished. err: <nil> Duration: 6.181134374s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 26.147444581s trace: https://jaeger.dev.storj.io/trace/6d2950019117adaf FAIL (2025-12-26 21:25:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Rj4hUGWvUPuU49QDsHYhibiJhyY5KDGus1rzNeP7KusgyvTsU, Piece ID: NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A) context deadline exceeded; piecestore: (Node ID: 12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, Piece ID: N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ) context deadline exceeded; piecestore: (Node ID: 12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, Piece ID: IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ) context deadline exceeded ======LOGS====== ng piece... Dec 26 20:54:08.087336805 (509h25m7.95841984s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087649520 (509h25m7.958732565s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087988717 (509h25m7.959071752s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088323233 (509h25m7.959406268s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088877028 (509h25m7.959960063s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089247203 (509h25m7.960330238s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089543066 (509h25m7.960626101s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089914133 (509h25m7.960997168s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090327259 (509h25m7.961410294s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090645345 (509h25m7.96172838s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091004318 (509h25m7.962087353s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091338976 (509h25m7.962422021s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091732054 (509h25m7.962815099s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:09.565187143 (509h25m9.436270188s) [upload=447022]: Next returned nil segment Dec 26 20:54:09.565201321 (509h25m9.436284356s) [upload=447022]: Waiting for error group managing segments... Dec 26 20:54:10.137065972 (509h25m10.008149007s) [noise=true , address=77.90.31.10:4523 , piece_id=GQYT3Z53Y6HVTY3IBUNFCIWBEE7KVAUWEJYO7QJFRRAJCHTJMYFA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.137147197 (509h25m10.008230232s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.149325196 (509h25m10.020408242s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.159611673 (509h25m10.030694718s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.164411764 (509h25m10.035494809s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.190924278 (509h25m10.062007313s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.232931086 (509h25m10.104014131s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.241658245 (509h25m10.11274129s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.472574370 (509h25m10.343657405s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.866910566 (509h25m10.737993611s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.886583507 (509h25m10.757666542s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.127773215 (509h25m10.99885626s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.192133217 (509h25m11.063216263s) [noise=true , address=145.79.227.13:4819 , piece_id=BQDRL2QJZ75TAMUM2VSGTF4U3NEMZC4FFQLN6QV6UOAFHWB56MZA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256146219 (509h25m11.127229264s) [noise=true , address=145.79.227.40:4062 , piece_id=IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256176597 (509h25m11.127259632s) [seg_pos={0 0}, upload=447022]: Segment reached optimal threshold of 14 pieces. Dec 26 20:54:11.256428137 (509h25m11.127511192s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98): protocol: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection Dec 26 20:54:11.256439549 (509h25m11.127522584s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256458915 (509h25m11.12754198s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF): protocol: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection Dec 26 20:54:11.256490705 (509h25m11.127573741s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256698171 (509h25m11.127781216s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: rpc: dial tcp 145.79.227.30:4796: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:54:11.256706457 (509h25m11.127789502s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256917309 (509h25m11.128000364s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection Dec 26 20:54:11.256926666 (509h25m11.128009711s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256964869 (509h25m11.128047914s) [seg_pos={0 0}, upload=447022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:54:11.257064138 (509h25m11.128147173s) [upload=447022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:11.257070690 (509h25m11.128153725s) [upload=447022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:11.561962810 (509h25m11.433045875s) [upload=447022]: Done uploading segments... Dec 26 20:54:11.561987427 (509h25m11.433070472s) [upload=447022]: Upload finished. err: <nil> Duration: 26.147477965s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [XXX...............X....................XX.XX.X..X.X...XX.X..] duration: 6.096008278s trace: https://jaeger.dev.storj.io/trace/2c039742adb0e5fc FAIL (2025-12-26 21:25:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: DZ7L6AH2YZZQVPTHZMYX3SLCD5SXQVOIIJWKWQSR5OID6PZDPCFA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: EXHZ4FJ373LRHLSAIOKRCJKAKVGKDP6LCGFVLDYPF7CQ3VUGEANA) context deadline exceeded ======LOGS====== .. Dec 26 21:24:15.239493539 (509h55m15.110576574s) [upload=447461]: Got next segment. Inline: false Dec 26 21:24:15.239528506 (509h55m15.110611541s) [upload=447461]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 21:24:15.239534708 (509h55m15.110617743s) [upload=447461]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 21:24:15.490306572 (509h55m15.361389627s) [seg_pos={0 0}, upload=447461]: Begin upload segment... Dec 26 21:24:15.490361826 (509h55m15.361444871s) [seg_pos={0 0}, upload=447461]: Done begin upload segment. Dec 26 21:24:15.490401131 (509h55m15.361484166s) [noise=true , address=145.79.227.16:4912 , piece_id=2GXBYHN3YFLX5VIMG6TL5UBIXKAPIZKZ5HD6LJDJB24QW6BEBRYQ, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.490473849 (509h55m15.361556884s) [noise=true , address=77.90.31.15:4700 , piece_id=3SGR6HJ6YQTU4RZNQJGVQPDY5WH7KBJE4ORBXFNGPJM2YNUST6LA, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.490989091 (509h55m15.362072126s) [noise=true , address=77.90.31.10:4518 , piece_id=DZ7L6AH2YZZQVPTHZMYX3SLCD5SXQVOIIJWKWQSR5OID6PZDPCFA, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.491557553 (509h55m15.362640588s) [noise=true , address=145.79.227.9:4365 , piece_id=N5Z2HE3UXZVZFQLVZ42Q2C5T5NQQXPNTPCFU4EA3OPJGCH2PAJ2A, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.491880669 (509h55m15.362963714s) [noise=true , address=145.79.227.49:4337 , piece_id=64P7SZSDDNLR73RFCSELTS6VWZWPX742RRVPUOHL6EFQEVOMQQEA, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.492165521 (509h55m15.363248566s) [noise=true , address=145.79.227.37:4566 , piece_id=43XMAK5WQ24JAKCERKDQXYYFGNXTKHGGAUYEGHJ27DPJMNLRNXWQ, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.492509296 (509h55m15.363592331s) [noise=true , address=45.91.116.12:4226 , piece_id=FXJZP7ZBSEMQDLAEQD7EGKS5BRJJEHPRMARPTVM5F3LLWGSDH7GQ, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.492806461 (509h55m15.363889506s) [noise=true , address=145.79.227.36:4517 , piece_id=QKBV5M2AOSQT2KGSF3BEO7AGKTJ6DIOKUTIWYEKE674Z33H5JDJQ, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.493140848 (509h55m15.364223883s) [noise=true , address=145.79.227.5:4223 , piece_id=OELI44LN4UYNTG2PRI6YJYPTMHGSC2ULVODVE4WYVRTIME6YN7OA, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.494898515 (509h55m15.36598156s) [noise=true , address=45.91.116.14:4191 , piece_id=RE45TDF3GHDCIWRSB7FWZP5CW54RSOI4UGRAQXNVGUG4BGE5OGJA, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.496100484 (509h55m15.367183519s) [noise=true , address=145.79.227.50:4375 , piece_id=V6Z7QHRDZZKVUL5TBUFEU3S4LYEJLACJI4JWOHO6DUMHS7KTQFSA, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.497211439 (509h55m15.368294474s) [noise=true , address=45.91.116.9:4262 , piece_id=XHXR64XKMM4KVFCKYZKJT3XKAAOSIU7TOFHAPCIKAXZZOC336TZQ, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.498489914 (509h55m15.369572959s) [noise=true , address=145.79.227.48:4302 , piece_id=YWW2ZX6H2GA6AL7LGAJAAQ6Q277CE6WP7S673X75PUF5H7HGTN5A, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.499668979 (509h55m15.370752014s) [noise=true , address=77.90.31.19:4844 , piece_id=MTCGLEGNSJMXRIJVFOEY5FUD72V6PRRFQEKXHPCDWS3GJXTYNJSQ, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.500697898 (509h55m15.371780933s) [noise=true , address=45.91.116.13:4156 , piece_id=EXHZ4FJ373LRHLSAIOKRCJKAKVGKDP6LCGFVLDYPF7CQ3VUGEANA, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.501897282 (509h55m15.372980317s) [noise=true , address=145.79.227.43:4159 , piece_id=6VJHRFVXRQYIRFQXLLQRJE7YDHCGHSON5BQTRS3RYWRTPZDWIH6A, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:17.194827862 (509h55m17.065910897s) [upload=447461]: Next returned nil segment Dec 26 21:24:17.194840907 (509h55m17.065923952s) [upload=447461]: Waiting for error group managing segments... Dec 26 21:24:17.480615983 (509h55m17.351699028s) [noise=true , address=145.79.227.9:4365 , piece_id=N5Z2HE3UXZVZFQLVZ42Q2C5T5NQQXPNTPCFU4EA3OPJGCH2PAJ2A, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.488188001 (509h55m17.359271046s) [noise=true , address=145.79.227.37:4566 , piece_id=43XMAK5WQ24JAKCERKDQXYYFGNXTKHGGAUYEGHJ27DPJMNLRNXWQ, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.510872557 (509h55m17.381955592s) [noise=true , address=145.79.227.48:4302 , piece_id=YWW2ZX6H2GA6AL7LGAJAAQ6Q277CE6WP7S673X75PUF5H7HGTN5A, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.512796540 (509h55m17.383879585s) [noise=true , address=145.79.227.49:4337 , piece_id=64P7SZSDDNLR73RFCSELTS6VWZWPX742RRVPUOHL6EFQEVOMQQEA, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.519834743 (509h55m17.390917778s) [noise=true , address=145.79.227.36:4517 , piece_id=QKBV5M2AOSQT2KGSF3BEO7AGKTJ6DIOKUTIWYEKE674Z33H5JDJQ, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.520424034 (509h55m17.39150708s) [noise=true , address=145.79.227.50:4375 , piece_id=V6Z7QHRDZZKVUL5TBUFEU3S4LYEJLACJI4JWOHO6DUMHS7KTQFSA, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.525181103 (509h55m17.396264138s) [noise=true , address=145.79.227.43:4159 , piece_id=6VJHRFVXRQYIRFQXLLQRJE7YDHCGHSON5BQTRS3RYWRTPZDWIH6A, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.666984621 (509h55m17.538067657s) [noise=true , address=77.90.31.10:4518 , piece_id=DZ7L6AH2YZZQVPTHZMYX3SLCD5SXQVOIIJWKWQSR5OID6PZDPCFA, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.723629614 (509h55m17.594712649s) [noise=true , address=77.90.31.15:4700 , piece_id=3SGR6HJ6YQTU4RZNQJGVQPDY5WH7KBJE4ORBXFNGPJM2YNUST6LA, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.786535648 (509h55m17.657618683s) [noise=true , address=45.91.116.13:4156 , piece_id=EXHZ4FJ373LRHLSAIOKRCJKAKVGKDP6LCGFVLDYPF7CQ3VUGEANA, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.798681766 (509h55m17.669764801s) [noise=true , address=77.90.31.19:4844 , piece_id=MTCGLEGNSJMXRIJVFOEY5FUD72V6PRRFQEKXHPCDWS3GJXTYNJSQ, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.850514655 (509h55m17.72159769s) [noise=true , address=45.91.116.14:4191 , piece_id=RE45TDF3GHDCIWRSB7FWZP5CW54RSOI4UGRAQXNVGUG4BGE5OGJA, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.991331216 (509h55m17.862414251s) [noise=true , address=45.91.116.9:4262 , piece_id=XHXR64XKMM4KVFCKYZKJT3XKAAOSIU7TOFHAPCIKAXZZOC336TZQ, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.991350022 (509h55m17.862433057s) [seg_pos={0 0}, upload=447461]: Segment reached optimal threshold of 13 pieces. Dec 26 21:24:17.991545053 (509h55m17.862628088s) [noise=true , address=145.79.227.5:4223 , piece_id=OELI44LN4UYNTG2PRI6YJYPTMHGSC2ULVODVE4WYVRTIME6YN7OA, seg_pos={0 0}, upload=447461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: write tcp 188.166.225.170:33694->145.79.227.5:4223: use of closed network connection; write tcp 188.166.225.170:33694->145.79.227.5:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33694->145.79.227.5:4223: use of closed network connection Dec 26 21:24:17.991554612 (509h55m17.862637647s) [noise=true , address=145.79.227.5:4223 , piece_id=OELI44LN4UYNTG2PRI6YJYPTMHGSC2ULVODVE4WYVRTIME6YN7OA, seg_pos={0 0}, upload=447461]: Long tail cancellation due to optimal threshold Dec 26 21:24:17.991690420 (509h55m17.862773455s) [noise=true , address=145.79.227.16:4912 , piece_id=2GXBYHN3YFLX5VIMG6TL5UBIXKAPIZKZ5HD6LJDJB24QW6BEBRYQ, seg_pos={0 0}, upload=447461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR): protocol: write tcp 188.166.225.170:37658->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:37658->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37658->145.79.227.16:4912: use of closed network connection Dec 26 21:24:17.991699878 (509h55m17.862782913s) [noise=true , address=145.79.227.16:4912 , piece_id=2GXBYHN3YFLX5VIMG6TL5UBIXKAPIZKZ5HD6LJDJB24QW6BEBRYQ, seg_pos={0 0}, upload=447461]: Long tail cancellation due to optimal threshold Dec 26 21:24:17.991825707 (509h55m17.862908742s) [noise=true , address=45.91.116.12:4226 , piece_id=FXJZP7ZBSEMQDLAEQD7EGKS5BRJJEHPRMARPTVM5F3LLWGSDH7GQ, seg_pos={0 0}, upload=447461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 26 21:24:17.991831169 (509h55m17.862914204s) [noise=true , address=45.91.116.12:4226 , piece_id=FXJZP7ZBSEMQDLAEQD7EGKS5BRJJEHPRMARPTVM5F3LLWGSDH7GQ, seg_pos={0 0}, upload=447461]: Long tail cancellation due to optimal threshold Dec 26 21:24:17.991846698 (509h55m17.862929733s) [seg_pos={0 0}, upload=447461]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 21:24:17.991908355 (509h55m17.86299139s) [upload=447461]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:24:17.991912844 (509h55m17.862995879s) [upload=447461]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:24:18.293828483 (509h55m18.164911518s) [upload=447461]: Done uploading segments... Dec 26 21:24:18.293846899 (509h55m18.164929934s) [upload=447461]: Upload finished. err: <nil> Duration: 6.096203509s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.XXX...............X....................XX.XX.X..X.X...XX.X.] duration: 7.257695337s trace: https://jaeger.dev.storj.io/trace/2c039742adb0e5fc FAIL (2025-12-26 21:24:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: DZ7L6AH2YZZQVPTHZMYX3SLCD5SXQVOIIJWKWQSR5OID6PZDPCFA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: EXHZ4FJ373LRHLSAIOKRCJKAKVGKDP6LCGFVLDYPF7CQ3VUGEANA) context deadline exceeded ======LOGS====== .. Dec 26 21:24:15.239493539 (509h55m15.110576574s) [upload=447461]: Got next segment. Inline: false Dec 26 21:24:15.239528506 (509h55m15.110611541s) [upload=447461]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 21:24:15.239534708 (509h55m15.110617743s) [upload=447461]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 21:24:15.490306572 (509h55m15.361389627s) [seg_pos={0 0}, upload=447461]: Begin upload segment... Dec 26 21:24:15.490361826 (509h55m15.361444871s) [seg_pos={0 0}, upload=447461]: Done begin upload segment. Dec 26 21:24:15.490401131 (509h55m15.361484166s) [noise=true , address=145.79.227.16:4912 , piece_id=2GXBYHN3YFLX5VIMG6TL5UBIXKAPIZKZ5HD6LJDJB24QW6BEBRYQ, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.490473849 (509h55m15.361556884s) [noise=true , address=77.90.31.15:4700 , piece_id=3SGR6HJ6YQTU4RZNQJGVQPDY5WH7KBJE4ORBXFNGPJM2YNUST6LA, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.490989091 (509h55m15.362072126s) [noise=true , address=77.90.31.10:4518 , piece_id=DZ7L6AH2YZZQVPTHZMYX3SLCD5SXQVOIIJWKWQSR5OID6PZDPCFA, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.491557553 (509h55m15.362640588s) [noise=true , address=145.79.227.9:4365 , piece_id=N5Z2HE3UXZVZFQLVZ42Q2C5T5NQQXPNTPCFU4EA3OPJGCH2PAJ2A, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.491880669 (509h55m15.362963714s) [noise=true , address=145.79.227.49:4337 , piece_id=64P7SZSDDNLR73RFCSELTS6VWZWPX742RRVPUOHL6EFQEVOMQQEA, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.492165521 (509h55m15.363248566s) [noise=true , address=145.79.227.37:4566 , piece_id=43XMAK5WQ24JAKCERKDQXYYFGNXTKHGGAUYEGHJ27DPJMNLRNXWQ, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.492509296 (509h55m15.363592331s) [noise=true , address=45.91.116.12:4226 , piece_id=FXJZP7ZBSEMQDLAEQD7EGKS5BRJJEHPRMARPTVM5F3LLWGSDH7GQ, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.492806461 (509h55m15.363889506s) [noise=true , address=145.79.227.36:4517 , piece_id=QKBV5M2AOSQT2KGSF3BEO7AGKTJ6DIOKUTIWYEKE674Z33H5JDJQ, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.493140848 (509h55m15.364223883s) [noise=true , address=145.79.227.5:4223 , piece_id=OELI44LN4UYNTG2PRI6YJYPTMHGSC2ULVODVE4WYVRTIME6YN7OA, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.494898515 (509h55m15.36598156s) [noise=true , address=45.91.116.14:4191 , piece_id=RE45TDF3GHDCIWRSB7FWZP5CW54RSOI4UGRAQXNVGUG4BGE5OGJA, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.496100484 (509h55m15.367183519s) [noise=true , address=145.79.227.50:4375 , piece_id=V6Z7QHRDZZKVUL5TBUFEU3S4LYEJLACJI4JWOHO6DUMHS7KTQFSA, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.497211439 (509h55m15.368294474s) [noise=true , address=45.91.116.9:4262 , piece_id=XHXR64XKMM4KVFCKYZKJT3XKAAOSIU7TOFHAPCIKAXZZOC336TZQ, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.498489914 (509h55m15.369572959s) [noise=true , address=145.79.227.48:4302 , piece_id=YWW2ZX6H2GA6AL7LGAJAAQ6Q277CE6WP7S673X75PUF5H7HGTN5A, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.499668979 (509h55m15.370752014s) [noise=true , address=77.90.31.19:4844 , piece_id=MTCGLEGNSJMXRIJVFOEY5FUD72V6PRRFQEKXHPCDWS3GJXTYNJSQ, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.500697898 (509h55m15.371780933s) [noise=true , address=45.91.116.13:4156 , piece_id=EXHZ4FJ373LRHLSAIOKRCJKAKVGKDP6LCGFVLDYPF7CQ3VUGEANA, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.501897282 (509h55m15.372980317s) [noise=true , address=145.79.227.43:4159 , piece_id=6VJHRFVXRQYIRFQXLLQRJE7YDHCGHSON5BQTRS3RYWRTPZDWIH6A, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:17.194827862 (509h55m17.065910897s) [upload=447461]: Next returned nil segment Dec 26 21:24:17.194840907 (509h55m17.065923952s) [upload=447461]: Waiting for error group managing segments... Dec 26 21:24:17.480615983 (509h55m17.351699028s) [noise=true , address=145.79.227.9:4365 , piece_id=N5Z2HE3UXZVZFQLVZ42Q2C5T5NQQXPNTPCFU4EA3OPJGCH2PAJ2A, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.488188001 (509h55m17.359271046s) [noise=true , address=145.79.227.37:4566 , piece_id=43XMAK5WQ24JAKCERKDQXYYFGNXTKHGGAUYEGHJ27DPJMNLRNXWQ, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.510872557 (509h55m17.381955592s) [noise=true , address=145.79.227.48:4302 , piece_id=YWW2ZX6H2GA6AL7LGAJAAQ6Q277CE6WP7S673X75PUF5H7HGTN5A, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.512796540 (509h55m17.383879585s) [noise=true , address=145.79.227.49:4337 , piece_id=64P7SZSDDNLR73RFCSELTS6VWZWPX742RRVPUOHL6EFQEVOMQQEA, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.519834743 (509h55m17.390917778s) [noise=true , address=145.79.227.36:4517 , piece_id=QKBV5M2AOSQT2KGSF3BEO7AGKTJ6DIOKUTIWYEKE674Z33H5JDJQ, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.520424034 (509h55m17.39150708s) [noise=true , address=145.79.227.50:4375 , piece_id=V6Z7QHRDZZKVUL5TBUFEU3S4LYEJLACJI4JWOHO6DUMHS7KTQFSA, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.525181103 (509h55m17.396264138s) [noise=true , address=145.79.227.43:4159 , piece_id=6VJHRFVXRQYIRFQXLLQRJE7YDHCGHSON5BQTRS3RYWRTPZDWIH6A, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.666984621 (509h55m17.538067657s) [noise=true , address=77.90.31.10:4518 , piece_id=DZ7L6AH2YZZQVPTHZMYX3SLCD5SXQVOIIJWKWQSR5OID6PZDPCFA, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.723629614 (509h55m17.594712649s) [noise=true , address=77.90.31.15:4700 , piece_id=3SGR6HJ6YQTU4RZNQJGVQPDY5WH7KBJE4ORBXFNGPJM2YNUST6LA, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.786535648 (509h55m17.657618683s) [noise=true , address=45.91.116.13:4156 , piece_id=EXHZ4FJ373LRHLSAIOKRCJKAKVGKDP6LCGFVLDYPF7CQ3VUGEANA, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.798681766 (509h55m17.669764801s) [noise=true , address=77.90.31.19:4844 , piece_id=MTCGLEGNSJMXRIJVFOEY5FUD72V6PRRFQEKXHPCDWS3GJXTYNJSQ, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.850514655 (509h55m17.72159769s) [noise=true , address=45.91.116.14:4191 , piece_id=RE45TDF3GHDCIWRSB7FWZP5CW54RSOI4UGRAQXNVGUG4BGE5OGJA, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.991331216 (509h55m17.862414251s) [noise=true , address=45.91.116.9:4262 , piece_id=XHXR64XKMM4KVFCKYZKJT3XKAAOSIU7TOFHAPCIKAXZZOC336TZQ, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.991350022 (509h55m17.862433057s) [seg_pos={0 0}, upload=447461]: Segment reached optimal threshold of 13 pieces. Dec 26 21:24:17.991545053 (509h55m17.862628088s) [noise=true , address=145.79.227.5:4223 , piece_id=OELI44LN4UYNTG2PRI6YJYPTMHGSC2ULVODVE4WYVRTIME6YN7OA, seg_pos={0 0}, upload=447461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: write tcp 188.166.225.170:33694->145.79.227.5:4223: use of closed network connection; write tcp 188.166.225.170:33694->145.79.227.5:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33694->145.79.227.5:4223: use of closed network connection Dec 26 21:24:17.991554612 (509h55m17.862637647s) [noise=true , address=145.79.227.5:4223 , piece_id=OELI44LN4UYNTG2PRI6YJYPTMHGSC2ULVODVE4WYVRTIME6YN7OA, seg_pos={0 0}, upload=447461]: Long tail cancellation due to optimal threshold Dec 26 21:24:17.991690420 (509h55m17.862773455s) [noise=true , address=145.79.227.16:4912 , piece_id=2GXBYHN3YFLX5VIMG6TL5UBIXKAPIZKZ5HD6LJDJB24QW6BEBRYQ, seg_pos={0 0}, upload=447461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR): protocol: write tcp 188.166.225.170:37658->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:37658->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37658->145.79.227.16:4912: use of closed network connection Dec 26 21:24:17.991699878 (509h55m17.862782913s) [noise=true , address=145.79.227.16:4912 , piece_id=2GXBYHN3YFLX5VIMG6TL5UBIXKAPIZKZ5HD6LJDJB24QW6BEBRYQ, seg_pos={0 0}, upload=447461]: Long tail cancellation due to optimal threshold Dec 26 21:24:17.991825707 (509h55m17.862908742s) [noise=true , address=45.91.116.12:4226 , piece_id=FXJZP7ZBSEMQDLAEQD7EGKS5BRJJEHPRMARPTVM5F3LLWGSDH7GQ, seg_pos={0 0}, upload=447461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 26 21:24:17.991831169 (509h55m17.862914204s) [noise=true , address=45.91.116.12:4226 , piece_id=FXJZP7ZBSEMQDLAEQD7EGKS5BRJJEHPRMARPTVM5F3LLWGSDH7GQ, seg_pos={0 0}, upload=447461]: Long tail cancellation due to optimal threshold Dec 26 21:24:17.991846698 (509h55m17.862929733s) [seg_pos={0 0}, upload=447461]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 21:24:17.991908355 (509h55m17.86299139s) [upload=447461]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:24:17.991912844 (509h55m17.862995879s) [upload=447461]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:24:18.293828483 (509h55m18.164911518s) [upload=447461]: Done uploading segments... Dec 26 21:24:18.293846899 (509h55m18.164929934s) [upload=447461]: Upload finished. err: <nil> Duration: 7.257723461s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 14.709725029s trace: https://jaeger.dev.storj.io/trace/6d2950019117adaf FAIL (2025-12-26 21:24:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Rj4hUGWvUPuU49QDsHYhibiJhyY5KDGus1rzNeP7KusgyvTsU, Piece ID: NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A) context deadline exceeded; piecestore: (Node ID: 12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, Piece ID: N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ) context deadline exceeded; piecestore: (Node ID: 12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, Piece ID: IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ) context deadline exceeded ======LOGS====== ng piece... Dec 26 20:54:08.087336805 (509h25m7.95841984s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087649520 (509h25m7.958732565s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087988717 (509h25m7.959071752s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088323233 (509h25m7.959406268s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088877028 (509h25m7.959960063s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089247203 (509h25m7.960330238s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089543066 (509h25m7.960626101s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089914133 (509h25m7.960997168s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090327259 (509h25m7.961410294s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090645345 (509h25m7.96172838s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091004318 (509h25m7.962087353s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091338976 (509h25m7.962422021s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091732054 (509h25m7.962815099s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:09.565187143 (509h25m9.436270188s) [upload=447022]: Next returned nil segment Dec 26 20:54:09.565201321 (509h25m9.436284356s) [upload=447022]: Waiting for error group managing segments... Dec 26 20:54:10.137065972 (509h25m10.008149007s) [noise=true , address=77.90.31.10:4523 , piece_id=GQYT3Z53Y6HVTY3IBUNFCIWBEE7KVAUWEJYO7QJFRRAJCHTJMYFA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.137147197 (509h25m10.008230232s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.149325196 (509h25m10.020408242s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.159611673 (509h25m10.030694718s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.164411764 (509h25m10.035494809s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.190924278 (509h25m10.062007313s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.232931086 (509h25m10.104014131s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.241658245 (509h25m10.11274129s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.472574370 (509h25m10.343657405s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.866910566 (509h25m10.737993611s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.886583507 (509h25m10.757666542s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.127773215 (509h25m10.99885626s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.192133217 (509h25m11.063216263s) [noise=true , address=145.79.227.13:4819 , piece_id=BQDRL2QJZ75TAMUM2VSGTF4U3NEMZC4FFQLN6QV6UOAFHWB56MZA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256146219 (509h25m11.127229264s) [noise=true , address=145.79.227.40:4062 , piece_id=IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256176597 (509h25m11.127259632s) [seg_pos={0 0}, upload=447022]: Segment reached optimal threshold of 14 pieces. Dec 26 20:54:11.256428137 (509h25m11.127511192s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98): protocol: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection Dec 26 20:54:11.256439549 (509h25m11.127522584s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256458915 (509h25m11.12754198s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF): protocol: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection Dec 26 20:54:11.256490705 (509h25m11.127573741s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256698171 (509h25m11.127781216s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: rpc: dial tcp 145.79.227.30:4796: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:54:11.256706457 (509h25m11.127789502s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256917309 (509h25m11.128000364s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection Dec 26 20:54:11.256926666 (509h25m11.128009711s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256964869 (509h25m11.128047914s) [seg_pos={0 0}, upload=447022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:54:11.257064138 (509h25m11.128147173s) [upload=447022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:11.257070690 (509h25m11.128153725s) [upload=447022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:11.561962810 (509h25m11.433045875s) [upload=447022]: Done uploading segments... Dec 26 20:54:11.561987427 (509h25m11.433070472s) [upload=447022]: Upload finished. err: <nil> Duration: 14.70974153s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..XXX...............X....................XX.XX.X..X.X...XX.X] duration: 30.002172937s trace: https://jaeger.dev.storj.io/trace/2c039742adb0e5fc FAIL (2025-12-26 21:24:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: DZ7L6AH2YZZQVPTHZMYX3SLCD5SXQVOIIJWKWQSR5OID6PZDPCFA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: EXHZ4FJ373LRHLSAIOKRCJKAKVGKDP6LCGFVLDYPF7CQ3VUGEANA) context deadline exceeded ======LOGS====== .. Dec 26 21:24:15.239493539 (509h55m15.110576574s) [upload=447461]: Got next segment. Inline: false Dec 26 21:24:15.239528506 (509h55m15.110611541s) [upload=447461]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 21:24:15.239534708 (509h55m15.110617743s) [upload=447461]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 21:24:15.490306572 (509h55m15.361389627s) [seg_pos={0 0}, upload=447461]: Begin upload segment... Dec 26 21:24:15.490361826 (509h55m15.361444871s) [seg_pos={0 0}, upload=447461]: Done begin upload segment. Dec 26 21:24:15.490401131 (509h55m15.361484166s) [noise=true , address=145.79.227.16:4912 , piece_id=2GXBYHN3YFLX5VIMG6TL5UBIXKAPIZKZ5HD6LJDJB24QW6BEBRYQ, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.490473849 (509h55m15.361556884s) [noise=true , address=77.90.31.15:4700 , piece_id=3SGR6HJ6YQTU4RZNQJGVQPDY5WH7KBJE4ORBXFNGPJM2YNUST6LA, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.490989091 (509h55m15.362072126s) [noise=true , address=77.90.31.10:4518 , piece_id=DZ7L6AH2YZZQVPTHZMYX3SLCD5SXQVOIIJWKWQSR5OID6PZDPCFA, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.491557553 (509h55m15.362640588s) [noise=true , address=145.79.227.9:4365 , piece_id=N5Z2HE3UXZVZFQLVZ42Q2C5T5NQQXPNTPCFU4EA3OPJGCH2PAJ2A, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.491880669 (509h55m15.362963714s) [noise=true , address=145.79.227.49:4337 , piece_id=64P7SZSDDNLR73RFCSELTS6VWZWPX742RRVPUOHL6EFQEVOMQQEA, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.492165521 (509h55m15.363248566s) [noise=true , address=145.79.227.37:4566 , piece_id=43XMAK5WQ24JAKCERKDQXYYFGNXTKHGGAUYEGHJ27DPJMNLRNXWQ, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.492509296 (509h55m15.363592331s) [noise=true , address=45.91.116.12:4226 , piece_id=FXJZP7ZBSEMQDLAEQD7EGKS5BRJJEHPRMARPTVM5F3LLWGSDH7GQ, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.492806461 (509h55m15.363889506s) [noise=true , address=145.79.227.36:4517 , piece_id=QKBV5M2AOSQT2KGSF3BEO7AGKTJ6DIOKUTIWYEKE674Z33H5JDJQ, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.493140848 (509h55m15.364223883s) [noise=true , address=145.79.227.5:4223 , piece_id=OELI44LN4UYNTG2PRI6YJYPTMHGSC2ULVODVE4WYVRTIME6YN7OA, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.494898515 (509h55m15.36598156s) [noise=true , address=45.91.116.14:4191 , piece_id=RE45TDF3GHDCIWRSB7FWZP5CW54RSOI4UGRAQXNVGUG4BGE5OGJA, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.496100484 (509h55m15.367183519s) [noise=true , address=145.79.227.50:4375 , piece_id=V6Z7QHRDZZKVUL5TBUFEU3S4LYEJLACJI4JWOHO6DUMHS7KTQFSA, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.497211439 (509h55m15.368294474s) [noise=true , address=45.91.116.9:4262 , piece_id=XHXR64XKMM4KVFCKYZKJT3XKAAOSIU7TOFHAPCIKAXZZOC336TZQ, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.498489914 (509h55m15.369572959s) [noise=true , address=145.79.227.48:4302 , piece_id=YWW2ZX6H2GA6AL7LGAJAAQ6Q277CE6WP7S673X75PUF5H7HGTN5A, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.499668979 (509h55m15.370752014s) [noise=true , address=77.90.31.19:4844 , piece_id=MTCGLEGNSJMXRIJVFOEY5FUD72V6PRRFQEKXHPCDWS3GJXTYNJSQ, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.500697898 (509h55m15.371780933s) [noise=true , address=45.91.116.13:4156 , piece_id=EXHZ4FJ373LRHLSAIOKRCJKAKVGKDP6LCGFVLDYPF7CQ3VUGEANA, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:15.501897282 (509h55m15.372980317s) [noise=true , address=145.79.227.43:4159 , piece_id=6VJHRFVXRQYIRFQXLLQRJE7YDHCGHSON5BQTRS3RYWRTPZDWIH6A, seg_pos={0 0}, upload=447461]: Uploading piece... Dec 26 21:24:17.194827862 (509h55m17.065910897s) [upload=447461]: Next returned nil segment Dec 26 21:24:17.194840907 (509h55m17.065923952s) [upload=447461]: Waiting for error group managing segments... Dec 26 21:24:17.480615983 (509h55m17.351699028s) [noise=true , address=145.79.227.9:4365 , piece_id=N5Z2HE3UXZVZFQLVZ42Q2C5T5NQQXPNTPCFU4EA3OPJGCH2PAJ2A, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.488188001 (509h55m17.359271046s) [noise=true , address=145.79.227.37:4566 , piece_id=43XMAK5WQ24JAKCERKDQXYYFGNXTKHGGAUYEGHJ27DPJMNLRNXWQ, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.510872557 (509h55m17.381955592s) [noise=true , address=145.79.227.48:4302 , piece_id=YWW2ZX6H2GA6AL7LGAJAAQ6Q277CE6WP7S673X75PUF5H7HGTN5A, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.512796540 (509h55m17.383879585s) [noise=true , address=145.79.227.49:4337 , piece_id=64P7SZSDDNLR73RFCSELTS6VWZWPX742RRVPUOHL6EFQEVOMQQEA, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.519834743 (509h55m17.390917778s) [noise=true , address=145.79.227.36:4517 , piece_id=QKBV5M2AOSQT2KGSF3BEO7AGKTJ6DIOKUTIWYEKE674Z33H5JDJQ, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.520424034 (509h55m17.39150708s) [noise=true , address=145.79.227.50:4375 , piece_id=V6Z7QHRDZZKVUL5TBUFEU3S4LYEJLACJI4JWOHO6DUMHS7KTQFSA, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.525181103 (509h55m17.396264138s) [noise=true , address=145.79.227.43:4159 , piece_id=6VJHRFVXRQYIRFQXLLQRJE7YDHCGHSON5BQTRS3RYWRTPZDWIH6A, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.666984621 (509h55m17.538067657s) [noise=true , address=77.90.31.10:4518 , piece_id=DZ7L6AH2YZZQVPTHZMYX3SLCD5SXQVOIIJWKWQSR5OID6PZDPCFA, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.723629614 (509h55m17.594712649s) [noise=true , address=77.90.31.15:4700 , piece_id=3SGR6HJ6YQTU4RZNQJGVQPDY5WH7KBJE4ORBXFNGPJM2YNUST6LA, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.786535648 (509h55m17.657618683s) [noise=true , address=45.91.116.13:4156 , piece_id=EXHZ4FJ373LRHLSAIOKRCJKAKVGKDP6LCGFVLDYPF7CQ3VUGEANA, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.798681766 (509h55m17.669764801s) [noise=true , address=77.90.31.19:4844 , piece_id=MTCGLEGNSJMXRIJVFOEY5FUD72V6PRRFQEKXHPCDWS3GJXTYNJSQ, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.850514655 (509h55m17.72159769s) [noise=true , address=45.91.116.14:4191 , piece_id=RE45TDF3GHDCIWRSB7FWZP5CW54RSOI4UGRAQXNVGUG4BGE5OGJA, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.991331216 (509h55m17.862414251s) [noise=true , address=45.91.116.9:4262 , piece_id=XHXR64XKMM4KVFCKYZKJT3XKAAOSIU7TOFHAPCIKAXZZOC336TZQ, seg_pos={0 0}, upload=447461]: Done uploading piece. err: <nil> Dec 26 21:24:17.991350022 (509h55m17.862433057s) [seg_pos={0 0}, upload=447461]: Segment reached optimal threshold of 13 pieces. Dec 26 21:24:17.991545053 (509h55m17.862628088s) [noise=true , address=145.79.227.5:4223 , piece_id=OELI44LN4UYNTG2PRI6YJYPTMHGSC2ULVODVE4WYVRTIME6YN7OA, seg_pos={0 0}, upload=447461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: write tcp 188.166.225.170:33694->145.79.227.5:4223: use of closed network connection; write tcp 188.166.225.170:33694->145.79.227.5:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33694->145.79.227.5:4223: use of closed network connection Dec 26 21:24:17.991554612 (509h55m17.862637647s) [noise=true , address=145.79.227.5:4223 , piece_id=OELI44LN4UYNTG2PRI6YJYPTMHGSC2ULVODVE4WYVRTIME6YN7OA, seg_pos={0 0}, upload=447461]: Long tail cancellation due to optimal threshold Dec 26 21:24:17.991690420 (509h55m17.862773455s) [noise=true , address=145.79.227.16:4912 , piece_id=2GXBYHN3YFLX5VIMG6TL5UBIXKAPIZKZ5HD6LJDJB24QW6BEBRYQ, seg_pos={0 0}, upload=447461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR): protocol: write tcp 188.166.225.170:37658->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:37658->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37658->145.79.227.16:4912: use of closed network connection Dec 26 21:24:17.991699878 (509h55m17.862782913s) [noise=true , address=145.79.227.16:4912 , piece_id=2GXBYHN3YFLX5VIMG6TL5UBIXKAPIZKZ5HD6LJDJB24QW6BEBRYQ, seg_pos={0 0}, upload=447461]: Long tail cancellation due to optimal threshold Dec 26 21:24:17.991825707 (509h55m17.862908742s) [noise=true , address=45.91.116.12:4226 , piece_id=FXJZP7ZBSEMQDLAEQD7EGKS5BRJJEHPRMARPTVM5F3LLWGSDH7GQ, seg_pos={0 0}, upload=447461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 26 21:24:17.991831169 (509h55m17.862914204s) [noise=true , address=45.91.116.12:4226 , piece_id=FXJZP7ZBSEMQDLAEQD7EGKS5BRJJEHPRMARPTVM5F3LLWGSDH7GQ, seg_pos={0 0}, upload=447461]: Long tail cancellation due to optimal threshold Dec 26 21:24:17.991846698 (509h55m17.862929733s) [seg_pos={0 0}, upload=447461]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 21:24:17.991908355 (509h55m17.86299139s) [upload=447461]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:24:17.991912844 (509h55m17.862995879s) [upload=447461]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:24:18.293828483 (509h55m18.164911518s) [upload=447461]: Done uploading segments... Dec 26 21:24:18.293846899 (509h55m18.164929934s) [upload=447461]: Upload finished. err: <nil> Duration: 30.002184158s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 28.806482881s trace: https://jaeger.dev.storj.io/trace/6d2950019117adaf FAIL (2025-12-26 21:24:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Rj4hUGWvUPuU49QDsHYhibiJhyY5KDGus1rzNeP7KusgyvTsU, Piece ID: NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A) context deadline exceeded; piecestore: (Node ID: 12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, Piece ID: N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ) context deadline exceeded; piecestore: (Node ID: 12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, Piece ID: IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ) context deadline exceeded ======LOGS====== ng piece... Dec 26 20:54:08.087336805 (509h25m7.95841984s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087649520 (509h25m7.958732565s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087988717 (509h25m7.959071752s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088323233 (509h25m7.959406268s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088877028 (509h25m7.959960063s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089247203 (509h25m7.960330238s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089543066 (509h25m7.960626101s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089914133 (509h25m7.960997168s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090327259 (509h25m7.961410294s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090645345 (509h25m7.96172838s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091004318 (509h25m7.962087353s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091338976 (509h25m7.962422021s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091732054 (509h25m7.962815099s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:09.565187143 (509h25m9.436270188s) [upload=447022]: Next returned nil segment Dec 26 20:54:09.565201321 (509h25m9.436284356s) [upload=447022]: Waiting for error group managing segments... Dec 26 20:54:10.137065972 (509h25m10.008149007s) [noise=true , address=77.90.31.10:4523 , piece_id=GQYT3Z53Y6HVTY3IBUNFCIWBEE7KVAUWEJYO7QJFRRAJCHTJMYFA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.137147197 (509h25m10.008230232s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.149325196 (509h25m10.020408242s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.159611673 (509h25m10.030694718s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.164411764 (509h25m10.035494809s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.190924278 (509h25m10.062007313s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.232931086 (509h25m10.104014131s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.241658245 (509h25m10.11274129s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.472574370 (509h25m10.343657405s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.866910566 (509h25m10.737993611s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.886583507 (509h25m10.757666542s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.127773215 (509h25m10.99885626s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.192133217 (509h25m11.063216263s) [noise=true , address=145.79.227.13:4819 , piece_id=BQDRL2QJZ75TAMUM2VSGTF4U3NEMZC4FFQLN6QV6UOAFHWB56MZA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256146219 (509h25m11.127229264s) [noise=true , address=145.79.227.40:4062 , piece_id=IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256176597 (509h25m11.127259632s) [seg_pos={0 0}, upload=447022]: Segment reached optimal threshold of 14 pieces. Dec 26 20:54:11.256428137 (509h25m11.127511192s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98): protocol: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection Dec 26 20:54:11.256439549 (509h25m11.127522584s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256458915 (509h25m11.12754198s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF): protocol: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection Dec 26 20:54:11.256490705 (509h25m11.127573741s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256698171 (509h25m11.127781216s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: rpc: dial tcp 145.79.227.30:4796: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:54:11.256706457 (509h25m11.127789502s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256917309 (509h25m11.128000364s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection Dec 26 20:54:11.256926666 (509h25m11.128009711s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256964869 (509h25m11.128047914s) [seg_pos={0 0}, upload=447022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:54:11.257064138 (509h25m11.128147173s) [upload=447022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:11.257070690 (509h25m11.128153725s) [upload=447022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:11.561962810 (509h25m11.433045875s) [upload=447022]: Done uploading segments... Dec 26 20:54:11.561987427 (509h25m11.433070472s) [upload=447022]: Upload finished. err: <nil> Duration: 28.806511065s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X..XXX...............X....................XX.XX.X..X.X...XX.] duration: 26.961411337s trace: https://jaeger.dev.storj.io/trace/27de0021e38e23a5 FAIL (2025-12-26 21:24:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: AGCDBO3VTGC3Z6BQGCCSZVTDPP2S7PLGPMNZEZM342TYRDDAM4QA) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: VOALUOLLXWANXIENOTKL32XV33ZUSTEQHRNR5AAB23WTHOD6SVXQ) context deadline exceeded ======LOGS====== Dec 26 21:22:51.443130441 (509h53m51.314213476s) [upload=447440]: Starting upload Dec 26 21:22:51.443186076 (509h53m51.314269122s) [upload=447440]: Done starting upload Dec 26 21:22:51.443377111 (509h53m51.314460146s) [upload=447440]: Uploading segments... Dec 26 21:22:51.443412418 (509h53m51.314495453s) [upload=447440]: Got next segment. Inline: false Dec 26 21:22:51.443438567 (509h53m51.314521602s) [upload=447440]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 21:22:51.443443287 (509h53m51.314526332s) [upload=447440]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 21:22:51.694666459 (509h53m51.565749494s) [seg_pos={0 0}, upload=447440]: Begin upload segment... Dec 26 21:22:51.694729990 (509h53m51.565813026s) [seg_pos={0 0}, upload=447440]: Done begin upload segment. Dec 26 21:22:51.694756630 (509h53m51.565839666s) [noise=true , address=145.79.227.48:4302 , piece_id=BAODS4H277JV6DNOCKLZM7MHXWWHWU6C7O2DM37TGJ7TCLUBI5HA, seg_pos={0 0}, upload=447440]: Uploading piece... Dec 26 21:22:51.694770507 (509h53m51.565853543s) [noise=true , address=45.91.116.7:4012 , piece_id=42WOXXPFJRJJKNAHQ6Q6AG5NOB2PSDBFSRRJRM47S3NTMTEEL3MQ, seg_pos={0 0}, upload=447440]: Uploading piece... Dec 26 21:22:51.694937285 (509h53m51.56602032s) [noise=true , address=77.90.31.19:4844 , piece_id=NUTUOYXXG4VPT4A65N6GISJSTE6CKUAVRA5RLKKYPT5RESKA56BQ, seg_pos={0 0}, upload=447440]: Uploading piece... Dec 26 21:22:51.695299875 (509h53m51.56638291s) [noise=true , address=45.91.116.5:4297 , piece_id=H5YE677VCH2Y2MGGG7K3FSJJTMAKGQ3Y7QKX2P2B6I7YAZMY4DWQ, seg_pos={0 0}, upload=447440]: Uploading piece... Dec 26 21:22:51.695719174 (509h53m51.566802219s) [noise=true , address=77.90.31.18:4805 , piece_id=56CSGRUOSQRXQDQIWZ7HL5G6O3LTYILRNDQ76F6OOJI37DB6F5XA, seg_pos={0 0}, upload=447440]: Uploading piece... Dec 26 21:22:51.696180682 (509h53m51.567263727s) [noise=true , address=77.90.31.14:4664 , piece_id=RLUNPWMAVASOCVJRPACXDRK2JAAD5CLRXLP467NSHXWKXHMIAYUQ, seg_pos={0 0}, upload=447440]: Uploading piece... Dec 26 21:22:51.696598648 (509h53m51.567681693s) [noise=true , address=145.79.227.50:4374 , piece_id=YGN53ENE4HUN4ZQ3YETZYUOJ4T3XFK3RULCJOWHUMVZIDRA7EVJA, seg_pos={0 0}, upload=447440]: Uploading piece... Dec 26 21:22:51.697011244 (509h53m51.568094289s) [noise=true , address=145.79.227.35:4483 , piece_id=2YD5GKNW4GIDZIAO7DQTX355G4UUIXIBUS2CSZNJ4ME332GOLQSQ, seg_pos={0 0}, upload=447440]: Uploading piece... Dec 26 21:22:51.697461050 (509h53m51.568544096s) [noise=true , address=145.79.227.49:4339 , piece_id=YMXSTZIP5PGYGX6EP65EXMVL7IG33ALG36VNQXSDBAWJAK3VBPHA, seg_pos={0 0}, upload=447440]: Uploading piece... Dec 26 21:22:51.698897045 (509h53m51.56998008s) [noise=true , address=145.79.227.9:4365 , piece_id=JHOKJXHL34PCMXPNDIM6C2CA6D4MPRLKZVRCT4YMRVPBUW3KWJAQ, seg_pos={0 0}, upload=447440]: Uploading piece... Dec 26 21:22:51.700267505 (509h53m51.57135055s) [noise=true , address=45.91.116.12:4225 , piece_id=AGCDBO3VTGC3Z6BQGCCSZVTDPP2S7PLGPMNZEZM342TYRDDAM4QA, seg_pos={0 0}, upload=447440]: Uploading piece... Dec 26 21:22:51.701762812 (509h53m51.572845847s) [noise=true , address=45.91.116.4:4940 , piece_id=W7HDPPWZZ6IYQCHM6GOJOWJDPKDTTHPEMXPSEESDRRV4TLXKTUZQ, seg_pos={0 0}, upload=447440]: Uploading piece... Dec 26 21:22:51.703483689 (509h53m51.574566734s) [noise=true , address=77.90.31.11:4556 , piece_id=VOALUOLLXWANXIENOTKL32XV33ZUSTEQHRNR5AAB23WTHOD6SVXQ, seg_pos={0 0}, upload=447440]: Uploading piece... Dec 26 21:22:51.704807359 (509h53m51.575890404s) [noise=true , address=145.79.227.30:4762 , piece_id=UYARILMDP6WWGU5UUTVHDMJAW2PEJUQFWDCXWN5AGJDNTUWFD6UA, seg_pos={0 0}, upload=447440]: Uploading piece... Dec 26 21:22:51.706224508 (509h53m51.577307543s) [noise=true , address=145.79.227.47:4266 , piece_id=J4LDPXLCOHZEHCTJPO4PZ5KRBZEHYF7ABRLCV7QPOYGZCGVOOYEQ, seg_pos={0 0}, upload=447440]: Uploading piece... Dec 26 21:22:51.710113333 (509h53m51.581196368s) [noise=true , address=145.79.227.37:4565 , piece_id=UV6OMILKAOXT2ON4L4T437KDTTDJTF4WX4GIC2FSAJQINIBFBSBA, seg_pos={0 0}, upload=447440]: Uploading piece... Dec 26 21:22:52.841497688 (509h53m52.712580753s) [noise=true , address=77.90.31.18:4805 , piece_id=56CSGRUOSQRXQDQIWZ7HL5G6O3LTYILRNDQ76F6OOJI37DB6F5XA, seg_pos={0 0}, upload=447440]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 21:22:53.409507129 (509h53m53.280590164s) [upload=447440]: Next returned nil segment Dec 26 21:22:53.409523310 (509h53m53.280606355s) [upload=447440]: Waiting for error group managing segments... Dec 26 21:22:53.705573311 (509h53m53.576656346s) [noise=true , address=145.79.227.49:4339 , piece_id=YMXSTZIP5PGYGX6EP65EXMVL7IG33ALG36VNQXSDBAWJAK3VBPHA, seg_pos={0 0}, upload=447440]: Done uploading piece. err: <nil> Dec 26 21:22:53.708222416 (509h53m53.579305461s) [noise=true , address=145.79.227.50:4374 , piece_id=YGN53ENE4HUN4ZQ3YETZYUOJ4T3XFK3RULCJOWHUMVZIDRA7EVJA, seg_pos={0 0}, upload=447440]: Done uploading piece. err: <nil> Dec 26 21:22:53.710220169 (509h53m53.581303214s) [noise=true , address=145.79.227.48:4302 , piece_id=BAODS4H277JV6DNOCKLZM7MHXWWHWU6C7O2DM37TGJ7TCLUBI5HA, seg_pos={0 0}, upload=447440]: Done uploading piece. err: <nil> Dec 26 21:22:53.730149519 (509h53m53.601232564s) [noise=true , address=145.79.227.9:4365 , piece_id=JHOKJXHL34PCMXPNDIM6C2CA6D4MPRLKZVRCT4YMRVPBUW3KWJAQ, seg_pos={0 0}, upload=447440]: Done uploading piece. err: <nil> Dec 26 21:22:53.743559884 (509h53m53.614642939s) [noise=true , address=145.79.227.47:4266 , piece_id=J4LDPXLCOHZEHCTJPO4PZ5KRBZEHYF7ABRLCV7QPOYGZCGVOOYEQ, seg_pos={0 0}, upload=447440]: Done uploading piece. err: <nil> Dec 26 21:22:53.899958786 (509h53m53.771041821s) [noise=true , address=77.90.31.19:4844 , piece_id=NUTUOYXXG4VPT4A65N6GISJSTE6CKUAVRA5RLKKYPT5RESKA56BQ, seg_pos={0 0}, upload=447440]: Done uploading piece. err: <nil> Dec 26 21:22:53.925752281 (509h53m53.796835316s) [noise=true , address=77.90.31.11:4556 , piece_id=VOALUOLLXWANXIENOTKL32XV33ZUSTEQHRNR5AAB23WTHOD6SVXQ, seg_pos={0 0}, upload=447440]: Done uploading piece. err: <nil> Dec 26 21:22:53.936553899 (509h53m53.807636934s) [noise=true , address=45.91.116.4:4940 , piece_id=W7HDPPWZZ6IYQCHM6GOJOWJDPKDTTHPEMXPSEESDRRV4TLXKTUZQ, seg_pos={0 0}, upload=447440]: Done uploading piece. err: <nil> Dec 26 21:22:53.963815280 (509h53m53.834898315s) [noise=true , address=45.91.116.7:4012 , piece_id=42WOXXPFJRJJKNAHQ6Q6AG5NOB2PSDBFSRRJRM47S3NTMTEEL3MQ, seg_pos={0 0}, upload=447440]: Done uploading piece. err: <nil> Dec 26 21:22:53.985263070 (509h53m53.856346105s) [noise=true , address=77.90.31.14:4664 , piece_id=RLUNPWMAVASOCVJRPACXDRK2JAAD5CLRXLP467NSHXWKXHMIAYUQ, seg_pos={0 0}, upload=447440]: Done uploading piece. err: <nil> Dec 26 21:22:54.016942363 (509h53m53.888025408s) [noise=true , address=45.91.116.5:4297 , piece_id=H5YE677VCH2Y2MGGG7K3FSJJTMAKGQ3Y7QKX2P2B6I7YAZMY4DWQ, seg_pos={0 0}, upload=447440]: Done uploading piece. err: <nil> Dec 26 21:22:54.148269182 (509h53m54.019352217s) [noise=true , address=45.91.116.12:4225 , piece_id=AGCDBO3VTGC3Z6BQGCCSZVTDPP2S7PLGPMNZEZM342TYRDDAM4QA, seg_pos={0 0}, upload=447440]: Done uploading piece. err: <nil> Dec 26 21:22:56.577858901 (509h53m56.448941936s) [noise=true , address=145.79.227.37:4565 , piece_id=UV6OMILKAOXT2ON4L4T437KDTTDJTF4WX4GIC2FSAJQINIBFBSBA, seg_pos={0 0}, upload=447440]: Done uploading piece. err: <nil> Dec 26 21:22:56.577882917 (509h53m56.448965962s) [seg_pos={0 0}, upload=447440]: Segment reached optimal threshold of 13 pieces. Dec 26 21:22:56.578195843 (509h53m56.449278898s) [noise=true , address=145.79.227.30:4762 , piece_id=UYARILMDP6WWGU5UUTVHDMJAW2PEJUQFWDCXWN5AGJDNTUWFD6UA, seg_pos={0 0}, upload=447440]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 26 21:22:56.578207185 (509h53m56.44929022s) [noise=true , address=145.79.227.30:4762 , piece_id=UYARILMDP6WWGU5UUTVHDMJAW2PEJUQFWDCXWN5AGJDNTUWFD6UA, seg_pos={0 0}, upload=447440]: Long tail cancellation due to optimal threshold Dec 26 21:22:56.578383991 (509h53m56.449467026s) [noise=true , address=145.79.227.35:4483 , piece_id=2YD5GKNW4GIDZIAO7DQTX355G4UUIXIBUS2CSZNJ4ME332GOLQSQ, seg_pos={0 0}, upload=447440]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 26 21:22:56.578399039 (509h53m56.449482074s) [noise=true , address=145.79.227.35:4483 , piece_id=2YD5GKNW4GIDZIAO7DQTX355G4UUIXIBUS2CSZNJ4ME332GOLQSQ, seg_pos={0 0}, upload=447440]: Long tail cancellation due to optimal threshold Dec 26 21:22:56.578418707 (509h53m56.449501752s) [seg_pos={0 0}, upload=447440]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 21:22:56.578510793 (509h53m56.449593828s) [upload=447440]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:22:56.578516303 (509h53m56.449599338s) [upload=447440]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:22:56.878176540 (509h53m56.749259575s) [upload=447440]: Done uploading segments... Dec 26 21:22:56.878195275 (509h53m56.74927831s) [upload=447440]: Upload finished. err: <nil> Duration: 26.961439499s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 17.246771595s trace: https://jaeger.dev.storj.io/trace/6d2950019117adaf FAIL (2025-12-26 21:23:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Rj4hUGWvUPuU49QDsHYhibiJhyY5KDGus1rzNeP7KusgyvTsU, Piece ID: NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A) context deadline exceeded; piecestore: (Node ID: 12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, Piece ID: N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ) context deadline exceeded; piecestore: (Node ID: 12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, Piece ID: IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ) context deadline exceeded ======LOGS====== ng piece... Dec 26 20:54:08.087336805 (509h25m7.95841984s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087649520 (509h25m7.958732565s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087988717 (509h25m7.959071752s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088323233 (509h25m7.959406268s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088877028 (509h25m7.959960063s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089247203 (509h25m7.960330238s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089543066 (509h25m7.960626101s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089914133 (509h25m7.960997168s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090327259 (509h25m7.961410294s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090645345 (509h25m7.96172838s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091004318 (509h25m7.962087353s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091338976 (509h25m7.962422021s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091732054 (509h25m7.962815099s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:09.565187143 (509h25m9.436270188s) [upload=447022]: Next returned nil segment Dec 26 20:54:09.565201321 (509h25m9.436284356s) [upload=447022]: Waiting for error group managing segments... Dec 26 20:54:10.137065972 (509h25m10.008149007s) [noise=true , address=77.90.31.10:4523 , piece_id=GQYT3Z53Y6HVTY3IBUNFCIWBEE7KVAUWEJYO7QJFRRAJCHTJMYFA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.137147197 (509h25m10.008230232s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.149325196 (509h25m10.020408242s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.159611673 (509h25m10.030694718s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.164411764 (509h25m10.035494809s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.190924278 (509h25m10.062007313s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.232931086 (509h25m10.104014131s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.241658245 (509h25m10.11274129s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.472574370 (509h25m10.343657405s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.866910566 (509h25m10.737993611s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.886583507 (509h25m10.757666542s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.127773215 (509h25m10.99885626s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.192133217 (509h25m11.063216263s) [noise=true , address=145.79.227.13:4819 , piece_id=BQDRL2QJZ75TAMUM2VSGTF4U3NEMZC4FFQLN6QV6UOAFHWB56MZA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256146219 (509h25m11.127229264s) [noise=true , address=145.79.227.40:4062 , piece_id=IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256176597 (509h25m11.127259632s) [seg_pos={0 0}, upload=447022]: Segment reached optimal threshold of 14 pieces. Dec 26 20:54:11.256428137 (509h25m11.127511192s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98): protocol: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection Dec 26 20:54:11.256439549 (509h25m11.127522584s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256458915 (509h25m11.12754198s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF): protocol: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection Dec 26 20:54:11.256490705 (509h25m11.127573741s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256698171 (509h25m11.127781216s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: rpc: dial tcp 145.79.227.30:4796: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:54:11.256706457 (509h25m11.127789502s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256917309 (509h25m11.128000364s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection Dec 26 20:54:11.256926666 (509h25m11.128009711s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256964869 (509h25m11.128047914s) [seg_pos={0 0}, upload=447022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:54:11.257064138 (509h25m11.128147173s) [upload=447022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:11.257070690 (509h25m11.128153725s) [upload=447022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:11.561962810 (509h25m11.433045875s) [upload=447022]: Done uploading segments... Dec 26 20:54:11.561987427 (509h25m11.433070472s) [upload=447022]: Upload finished. err: <nil> Duration: 17.246804487s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [XX..XXX...............X....................XX.XX.X..X.X...XX] duration: 30.002493905s trace: https://jaeger.dev.storj.io/trace/27de0021e38e23a5 FAIL (2025-12-26 21:23:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: AGCDBO3VTGC3Z6BQGCCSZVTDPP2S7PLGPMNZEZM342TYRDDAM4QA) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: VOALUOLLXWANXIENOTKL32XV33ZUSTEQHRNR5AAB23WTHOD6SVXQ) context deadline exceeded ======LOGS====== Dec 26 21:22:51.443130441 (509h53m51.314213476s) [upload=447440]: Starting upload Dec 26 21:22:51.443186076 (509h53m51.314269122s) [upload=447440]: Done starting upload Dec 26 21:22:51.443377111 (509h53m51.314460146s) [upload=447440]: Uploading segments... Dec 26 21:22:51.443412418 (509h53m51.314495453s) [upload=447440]: Got next segment. Inline: false Dec 26 21:22:51.443438567 (509h53m51.314521602s) [upload=447440]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 21:22:51.443443287 (509h53m51.314526332s) [upload=447440]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 21:22:51.694666459 (509h53m51.565749494s) [seg_pos={0 0}, upload=447440]: Begin upload segment... Dec 26 21:22:51.694729990 (509h53m51.565813026s) [seg_pos={0 0}, upload=447440]: Done begin upload segment. Dec 26 21:22:51.694756630 (509h53m51.565839666s) [noise=true , address=145.79.227.48:4302 , piece_id=BAODS4H277JV6DNOCKLZM7MHXWWHWU6C7O2DM37TGJ7TCLUBI5HA, seg_pos={0 0}, upload=447440]: Uploading piece... Dec 26 21:22:51.694770507 (509h53m51.565853543s) [noise=true , address=45.91.116.7:4012 , piece_id=42WOXXPFJRJJKNAHQ6Q6AG5NOB2PSDBFSRRJRM47S3NTMTEEL3MQ, seg_pos={0 0}, upload=447440]: Uploading piece... Dec 26 21:22:51.694937285 (509h53m51.56602032s) [noise=true , address=77.90.31.19:4844 , piece_id=NUTUOYXXG4VPT4A65N6GISJSTE6CKUAVRA5RLKKYPT5RESKA56BQ, seg_pos={0 0}, upload=447440]: Uploading piece... Dec 26 21:22:51.695299875 (509h53m51.56638291s) [noise=true , address=45.91.116.5:4297 , piece_id=H5YE677VCH2Y2MGGG7K3FSJJTMAKGQ3Y7QKX2P2B6I7YAZMY4DWQ, seg_pos={0 0}, upload=447440]: Uploading piece... Dec 26 21:22:51.695719174 (509h53m51.566802219s) [noise=true , address=77.90.31.18:4805 , piece_id=56CSGRUOSQRXQDQIWZ7HL5G6O3LTYILRNDQ76F6OOJI37DB6F5XA, seg_pos={0 0}, upload=447440]: Uploading piece... Dec 26 21:22:51.696180682 (509h53m51.567263727s) [noise=true , address=77.90.31.14:4664 , piece_id=RLUNPWMAVASOCVJRPACXDRK2JAAD5CLRXLP467NSHXWKXHMIAYUQ, seg_pos={0 0}, upload=447440]: Uploading piece... Dec 26 21:22:51.696598648 (509h53m51.567681693s) [noise=true , address=145.79.227.50:4374 , piece_id=YGN53ENE4HUN4ZQ3YETZYUOJ4T3XFK3RULCJOWHUMVZIDRA7EVJA, seg_pos={0 0}, upload=447440]: Uploading piece... Dec 26 21:22:51.697011244 (509h53m51.568094289s) [noise=true , address=145.79.227.35:4483 , piece_id=2YD5GKNW4GIDZIAO7DQTX355G4UUIXIBUS2CSZNJ4ME332GOLQSQ, seg_pos={0 0}, upload=447440]: Uploading piece... Dec 26 21:22:51.697461050 (509h53m51.568544096s) [noise=true , address=145.79.227.49:4339 , piece_id=YMXSTZIP5PGYGX6EP65EXMVL7IG33ALG36VNQXSDBAWJAK3VBPHA, seg_pos={0 0}, upload=447440]: Uploading piece... Dec 26 21:22:51.698897045 (509h53m51.56998008s) [noise=true , address=145.79.227.9:4365 , piece_id=JHOKJXHL34PCMXPNDIM6C2CA6D4MPRLKZVRCT4YMRVPBUW3KWJAQ, seg_pos={0 0}, upload=447440]: Uploading piece... Dec 26 21:22:51.700267505 (509h53m51.57135055s) [noise=true , address=45.91.116.12:4225 , piece_id=AGCDBO3VTGC3Z6BQGCCSZVTDPP2S7PLGPMNZEZM342TYRDDAM4QA, seg_pos={0 0}, upload=447440]: Uploading piece... Dec 26 21:22:51.701762812 (509h53m51.572845847s) [noise=true , address=45.91.116.4:4940 , piece_id=W7HDPPWZZ6IYQCHM6GOJOWJDPKDTTHPEMXPSEESDRRV4TLXKTUZQ, seg_pos={0 0}, upload=447440]: Uploading piece... Dec 26 21:22:51.703483689 (509h53m51.574566734s) [noise=true , address=77.90.31.11:4556 , piece_id=VOALUOLLXWANXIENOTKL32XV33ZUSTEQHRNR5AAB23WTHOD6SVXQ, seg_pos={0 0}, upload=447440]: Uploading piece... Dec 26 21:22:51.704807359 (509h53m51.575890404s) [noise=true , address=145.79.227.30:4762 , piece_id=UYARILMDP6WWGU5UUTVHDMJAW2PEJUQFWDCXWN5AGJDNTUWFD6UA, seg_pos={0 0}, upload=447440]: Uploading piece... Dec 26 21:22:51.706224508 (509h53m51.577307543s) [noise=true , address=145.79.227.47:4266 , piece_id=J4LDPXLCOHZEHCTJPO4PZ5KRBZEHYF7ABRLCV7QPOYGZCGVOOYEQ, seg_pos={0 0}, upload=447440]: Uploading piece... Dec 26 21:22:51.710113333 (509h53m51.581196368s) [noise=true , address=145.79.227.37:4565 , piece_id=UV6OMILKAOXT2ON4L4T437KDTTDJTF4WX4GIC2FSAJQINIBFBSBA, seg_pos={0 0}, upload=447440]: Uploading piece... Dec 26 21:22:52.841497688 (509h53m52.712580753s) [noise=true , address=77.90.31.18:4805 , piece_id=56CSGRUOSQRXQDQIWZ7HL5G6O3LTYILRNDQ76F6OOJI37DB6F5XA, seg_pos={0 0}, upload=447440]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 21:22:53.409507129 (509h53m53.280590164s) [upload=447440]: Next returned nil segment Dec 26 21:22:53.409523310 (509h53m53.280606355s) [upload=447440]: Waiting for error group managing segments... Dec 26 21:22:53.705573311 (509h53m53.576656346s) [noise=true , address=145.79.227.49:4339 , piece_id=YMXSTZIP5PGYGX6EP65EXMVL7IG33ALG36VNQXSDBAWJAK3VBPHA, seg_pos={0 0}, upload=447440]: Done uploading piece. err: <nil> Dec 26 21:22:53.708222416 (509h53m53.579305461s) [noise=true , address=145.79.227.50:4374 , piece_id=YGN53ENE4HUN4ZQ3YETZYUOJ4T3XFK3RULCJOWHUMVZIDRA7EVJA, seg_pos={0 0}, upload=447440]: Done uploading piece. err: <nil> Dec 26 21:22:53.710220169 (509h53m53.581303214s) [noise=true , address=145.79.227.48:4302 , piece_id=BAODS4H277JV6DNOCKLZM7MHXWWHWU6C7O2DM37TGJ7TCLUBI5HA, seg_pos={0 0}, upload=447440]: Done uploading piece. err: <nil> Dec 26 21:22:53.730149519 (509h53m53.601232564s) [noise=true , address=145.79.227.9:4365 , piece_id=JHOKJXHL34PCMXPNDIM6C2CA6D4MPRLKZVRCT4YMRVPBUW3KWJAQ, seg_pos={0 0}, upload=447440]: Done uploading piece. err: <nil> Dec 26 21:22:53.743559884 (509h53m53.614642939s) [noise=true , address=145.79.227.47:4266 , piece_id=J4LDPXLCOHZEHCTJPO4PZ5KRBZEHYF7ABRLCV7QPOYGZCGVOOYEQ, seg_pos={0 0}, upload=447440]: Done uploading piece. err: <nil> Dec 26 21:22:53.899958786 (509h53m53.771041821s) [noise=true , address=77.90.31.19:4844 , piece_id=NUTUOYXXG4VPT4A65N6GISJSTE6CKUAVRA5RLKKYPT5RESKA56BQ, seg_pos={0 0}, upload=447440]: Done uploading piece. err: <nil> Dec 26 21:22:53.925752281 (509h53m53.796835316s) [noise=true , address=77.90.31.11:4556 , piece_id=VOALUOLLXWANXIENOTKL32XV33ZUSTEQHRNR5AAB23WTHOD6SVXQ, seg_pos={0 0}, upload=447440]: Done uploading piece. err: <nil> Dec 26 21:22:53.936553899 (509h53m53.807636934s) [noise=true , address=45.91.116.4:4940 , piece_id=W7HDPPWZZ6IYQCHM6GOJOWJDPKDTTHPEMXPSEESDRRV4TLXKTUZQ, seg_pos={0 0}, upload=447440]: Done uploading piece. err: <nil> Dec 26 21:22:53.963815280 (509h53m53.834898315s) [noise=true , address=45.91.116.7:4012 , piece_id=42WOXXPFJRJJKNAHQ6Q6AG5NOB2PSDBFSRRJRM47S3NTMTEEL3MQ, seg_pos={0 0}, upload=447440]: Done uploading piece. err: <nil> Dec 26 21:22:53.985263070 (509h53m53.856346105s) [noise=true , address=77.90.31.14:4664 , piece_id=RLUNPWMAVASOCVJRPACXDRK2JAAD5CLRXLP467NSHXWKXHMIAYUQ, seg_pos={0 0}, upload=447440]: Done uploading piece. err: <nil> Dec 26 21:22:54.016942363 (509h53m53.888025408s) [noise=true , address=45.91.116.5:4297 , piece_id=H5YE677VCH2Y2MGGG7K3FSJJTMAKGQ3Y7QKX2P2B6I7YAZMY4DWQ, seg_pos={0 0}, upload=447440]: Done uploading piece. err: <nil> Dec 26 21:22:54.148269182 (509h53m54.019352217s) [noise=true , address=45.91.116.12:4225 , piece_id=AGCDBO3VTGC3Z6BQGCCSZVTDPP2S7PLGPMNZEZM342TYRDDAM4QA, seg_pos={0 0}, upload=447440]: Done uploading piece. err: <nil> Dec 26 21:22:56.577858901 (509h53m56.448941936s) [noise=true , address=145.79.227.37:4565 , piece_id=UV6OMILKAOXT2ON4L4T437KDTTDJTF4WX4GIC2FSAJQINIBFBSBA, seg_pos={0 0}, upload=447440]: Done uploading piece. err: <nil> Dec 26 21:22:56.577882917 (509h53m56.448965962s) [seg_pos={0 0}, upload=447440]: Segment reached optimal threshold of 13 pieces. Dec 26 21:22:56.578195843 (509h53m56.449278898s) [noise=true , address=145.79.227.30:4762 , piece_id=UYARILMDP6WWGU5UUTVHDMJAW2PEJUQFWDCXWN5AGJDNTUWFD6UA, seg_pos={0 0}, upload=447440]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 26 21:22:56.578207185 (509h53m56.44929022s) [noise=true , address=145.79.227.30:4762 , piece_id=UYARILMDP6WWGU5UUTVHDMJAW2PEJUQFWDCXWN5AGJDNTUWFD6UA, seg_pos={0 0}, upload=447440]: Long tail cancellation due to optimal threshold Dec 26 21:22:56.578383991 (509h53m56.449467026s) [noise=true , address=145.79.227.35:4483 , piece_id=2YD5GKNW4GIDZIAO7DQTX355G4UUIXIBUS2CSZNJ4ME332GOLQSQ, seg_pos={0 0}, upload=447440]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 26 21:22:56.578399039 (509h53m56.449482074s) [noise=true , address=145.79.227.35:4483 , piece_id=2YD5GKNW4GIDZIAO7DQTX355G4UUIXIBUS2CSZNJ4ME332GOLQSQ, seg_pos={0 0}, upload=447440]: Long tail cancellation due to optimal threshold Dec 26 21:22:56.578418707 (509h53m56.449501752s) [seg_pos={0 0}, upload=447440]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 21:22:56.578510793 (509h53m56.449593828s) [upload=447440]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:22:56.578516303 (509h53m56.449599338s) [upload=447440]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:22:56.878176540 (509h53m56.749259575s) [upload=447440]: Done uploading segments... Dec 26 21:22:56.878195275 (509h53m56.74927831s) [upload=447440]: Upload finished. err: <nil> Duration: 30.002510177s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 30.943047129s trace: https://jaeger.dev.storj.io/trace/6d2950019117adaf FAIL (2025-12-26 21:22:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Rj4hUGWvUPuU49QDsHYhibiJhyY5KDGus1rzNeP7KusgyvTsU, Piece ID: NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A) context deadline exceeded; piecestore: (Node ID: 12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, Piece ID: N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ) context deadline exceeded; piecestore: (Node ID: 12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, Piece ID: IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ) context deadline exceeded ======LOGS====== ng piece... Dec 26 20:54:08.087336805 (509h25m7.95841984s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087649520 (509h25m7.958732565s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087988717 (509h25m7.959071752s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088323233 (509h25m7.959406268s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088877028 (509h25m7.959960063s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089247203 (509h25m7.960330238s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089543066 (509h25m7.960626101s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089914133 (509h25m7.960997168s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090327259 (509h25m7.961410294s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090645345 (509h25m7.96172838s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091004318 (509h25m7.962087353s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091338976 (509h25m7.962422021s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091732054 (509h25m7.962815099s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:09.565187143 (509h25m9.436270188s) [upload=447022]: Next returned nil segment Dec 26 20:54:09.565201321 (509h25m9.436284356s) [upload=447022]: Waiting for error group managing segments... Dec 26 20:54:10.137065972 (509h25m10.008149007s) [noise=true , address=77.90.31.10:4523 , piece_id=GQYT3Z53Y6HVTY3IBUNFCIWBEE7KVAUWEJYO7QJFRRAJCHTJMYFA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.137147197 (509h25m10.008230232s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.149325196 (509h25m10.020408242s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.159611673 (509h25m10.030694718s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.164411764 (509h25m10.035494809s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.190924278 (509h25m10.062007313s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.232931086 (509h25m10.104014131s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.241658245 (509h25m10.11274129s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.472574370 (509h25m10.343657405s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.866910566 (509h25m10.737993611s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.886583507 (509h25m10.757666542s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.127773215 (509h25m10.99885626s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.192133217 (509h25m11.063216263s) [noise=true , address=145.79.227.13:4819 , piece_id=BQDRL2QJZ75TAMUM2VSGTF4U3NEMZC4FFQLN6QV6UOAFHWB56MZA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256146219 (509h25m11.127229264s) [noise=true , address=145.79.227.40:4062 , piece_id=IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256176597 (509h25m11.127259632s) [seg_pos={0 0}, upload=447022]: Segment reached optimal threshold of 14 pieces. Dec 26 20:54:11.256428137 (509h25m11.127511192s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98): protocol: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection Dec 26 20:54:11.256439549 (509h25m11.127522584s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256458915 (509h25m11.12754198s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF): protocol: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection Dec 26 20:54:11.256490705 (509h25m11.127573741s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256698171 (509h25m11.127781216s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: rpc: dial tcp 145.79.227.30:4796: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:54:11.256706457 (509h25m11.127789502s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256917309 (509h25m11.128000364s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection Dec 26 20:54:11.256926666 (509h25m11.128009711s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256964869 (509h25m11.128047914s) [seg_pos={0 0}, upload=447022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:54:11.257064138 (509h25m11.128147173s) [upload=447022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:11.257070690 (509h25m11.128153725s) [upload=447022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:11.561962810 (509h25m11.433045875s) [upload=447022]: Done uploading segments... Dec 26 20:54:11.561987427 (509h25m11.433070472s) [upload=447022]: Upload finished. err: <nil> Duration: 30.943075351s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.XX..XXX...............X....................XX.XX.X..X.X...X] duration: 30.002444061s trace: https://jaeger.dev.storj.io/trace/3a203ba1c753b8f0 FAIL (2025-12-26 21:22:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ... Dec 26 21:21:54.031974420 (509h52m53.903057455s) [upload=447428]: Got next segment. Inline: false Dec 26 21:21:54.032019987 (509h52m53.903103022s) [upload=447428]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 21:21:54.032024475 (509h52m53.90310751s) [upload=447428]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 21:21:54.281423104 (509h52m54.152506149s) [seg_pos={0 0}, upload=447428]: Begin upload segment... Dec 26 21:21:54.281472377 (509h52m54.152555412s) [seg_pos={0 0}, upload=447428]: Done begin upload segment. Dec 26 21:21:54.281504689 (509h52m54.152587724s) [noise=true , address=145.79.227.46:4603 , piece_id=B6MO2ZCY6J4ZVGTG5DFHMKH5TR4I57PRLLNTFNOUQR2KZU4DUS2Q, seg_pos={0 0}, upload=447428]: Uploading piece... Dec 26 21:21:54.281516472 (509h52m54.152599507s) [noise=true , address=145.79.227.39:4013 , piece_id=J6YSTTUKBN3J4AHHBGOU5VW2LOMSKSNXXLPIJKTE6R3DGHNZ6JAQ, seg_pos={0 0}, upload=447428]: Uploading piece... Dec 26 21:21:54.281851098 (509h52m54.152934134s) [noise=true , address=145.79.227.12:4767 , piece_id=N7KXWOVOPCHWXY3ZA2HIHMHMAINBJPE4PTQM7SR5FUETDPSOSACQ, seg_pos={0 0}, upload=447428]: Uploading piece... Dec 26 21:21:54.282372471 (509h52m54.153455516s) [noise=true , address=145.79.227.50:4375 , piece_id=AGOY5XKMDDSPCOFLVXU2P6PP7EQ2DQSUYTTFF7PXBS5SYLWXU7WA, seg_pos={0 0}, upload=447428]: Uploading piece... Dec 26 21:21:54.282662304 (509h52m54.153745339s) [noise=true , address=45.91.116.14:4192 , piece_id=M4MSOB2BL3ZNUKFJMUID5ML6OETFAOWY7TUKNFQZUPURSCRPXJZA, seg_pos={0 0}, upload=447428]: Uploading piece... Dec 26 21:21:54.282915375 (509h52m54.153998411s) [noise=true , address=77.90.31.13:4629 , piece_id=OFVKCT6RQHHVFTLBH2ZB5VV55LFWUNF74USH24H5OSOGSACSL47A, seg_pos={0 0}, upload=447428]: Uploading piece... Dec 26 21:21:54.283166223 (509h52m54.154249268s) [noise=true , address=77.90.31.21:4877 , piece_id=NVSMRM2YRKSX7XHP5OWPPSPBA7NW6YGXKHJREPXK3NIEVWXTSUJA, seg_pos={0 0}, upload=447428]: Uploading piece... Dec 26 21:21:54.283428873 (509h52m54.154511909s) [noise=true , address=145.79.227.51:4409 , piece_id=GWWDX3HK5BKUSJWGMB7LPAK3XZXDGAJEPTKIHLCPEOOCYJYDXFGA, seg_pos={0 0}, upload=447428]: Uploading piece... Dec 26 21:21:54.283440355 (509h52m54.15452339s) [noise=true , address=145.79.227.34:4907 , piece_id=MSW6FY2AHCLQYOESRHNV66UCLH2OU2EB3I5L2OQJPY225IGWUVNQ, seg_pos={0 0}, upload=447428]: Uploading piece... Dec 26 21:21:54.283704969 (509h52m54.154788004s) [noise=true , address=45.91.116.4:4937 , piece_id=PIFFBSULE5E5W66XSNS7LPYL3X6E7VQWDWSRCTHPK6GDO2NSSA6Q, seg_pos={0 0}, upload=447428]: Uploading piece... Dec 26 21:21:54.283828103 (509h52m54.154911138s) [noise=true , address=145.79.227.16:4911 , piece_id=RNWWS3Q4SWTRP556HV5DIG3DWMO46YCWFFLKDVA3VFE4FHIN6BFA, seg_pos={0 0}, upload=447428]: Uploading piece... Dec 26 21:21:54.284145878 (509h52m54.155228913s) [noise=true , address=145.79.227.36:4518 , piece_id=HKFTNDNTZUHXZC2VY2G6BC2V2WKG3KGDNPJTSVXFYMHSDY6RNCLA, seg_pos={0 0}, upload=447428]: Uploading piece... Dec 26 21:21:54.284543676 (509h52m54.155626711s) [noise=true , address=45.91.116.6:4975 , piece_id=TGMC5IY3OXXHDODBEBREPOSJPS3V6CRVTETTKC27R5ZVZDZWBCIQ, seg_pos={0 0}, upload=447428]: Uploading piece... Dec 26 21:21:54.284872031 (509h52m54.155955076s) [noise=true , address=45.91.116.5:4297 , piece_id=XPGYUBDXEMDNQBNJLN2RMHLNYGNPDONLAZRDANIVJ6VYUKOHO2VA, seg_pos={0 0}, upload=447428]: Uploading piece... Dec 26 21:21:54.285198963 (509h52m54.156281998s) [noise=true , address=145.79.227.52:4446 , piece_id=NFPLNRZR7QMKUIH2MUPTP3VWKFHBGOYDF2SOQHYTLNSXINF6J3JQ, seg_pos={0 0}, upload=447428]: Uploading piece... Dec 26 21:21:54.286407495 (509h52m54.15749054s) [noise=true , address=77.90.31.16:4736 , piece_id=CEWA2SXBQT7HLU2C7VAT23JZO2CUUHNRTX775DQJOCZYC7CYH4UA, seg_pos={0 0}, upload=447428]: Uploading piece... Dec 26 21:21:55.991754237 (509h52m55.862837272s) [upload=447428]: Next returned nil segment Dec 26 21:21:55.991794945 (509h52m55.86287798s) [upload=447428]: Waiting for error group managing segments... Dec 26 21:21:56.287076934 (509h52m56.158159979s) [noise=true , address=145.79.227.51:4409 , piece_id=GWWDX3HK5BKUSJWGMB7LPAK3XZXDGAJEPTKIHLCPEOOCYJYDXFGA, seg_pos={0 0}, upload=447428]: Done uploading piece. err: <nil> Dec 26 21:21:56.290064273 (509h52m56.161147328s) [noise=true , address=145.79.227.50:4375 , piece_id=AGOY5XKMDDSPCOFLVXU2P6PP7EQ2DQSUYTTFF7PXBS5SYLWXU7WA, seg_pos={0 0}, upload=447428]: Done uploading piece. err: <nil> Dec 26 21:21:56.300966993 (509h52m56.172050038s) [noise=true , address=145.79.227.52:4446 , piece_id=NFPLNRZR7QMKUIH2MUPTP3VWKFHBGOYDF2SOQHYTLNSXINF6J3JQ, seg_pos={0 0}, upload=447428]: Done uploading piece. err: <nil> Dec 26 21:21:56.301271653 (509h52m56.172354698s) [noise=true , address=145.79.227.12:4767 , piece_id=N7KXWOVOPCHWXY3ZA2HIHMHMAINBJPE4PTQM7SR5FUETDPSOSACQ, seg_pos={0 0}, upload=447428]: Done uploading piece. err: <nil> Dec 26 21:21:56.305260910 (509h52m56.176343945s) [noise=true , address=145.79.227.39:4013 , piece_id=J6YSTTUKBN3J4AHHBGOU5VW2LOMSKSNXXLPIJKTE6R3DGHNZ6JAQ, seg_pos={0 0}, upload=447428]: Done uploading piece. err: <nil> Dec 26 21:21:56.305291709 (509h52m56.176374754s) [noise=true , address=145.79.227.46:4603 , piece_id=B6MO2ZCY6J4ZVGTG5DFHMKH5TR4I57PRLLNTFNOUQR2KZU4DUS2Q, seg_pos={0 0}, upload=447428]: Done uploading piece. err: <nil> Dec 26 21:21:56.307857193 (509h52m56.178940238s) [noise=true , address=145.79.227.36:4518 , piece_id=HKFTNDNTZUHXZC2VY2G6BC2V2WKG3KGDNPJTSVXFYMHSDY6RNCLA, seg_pos={0 0}, upload=447428]: Done uploading piece. err: <nil> Dec 26 21:21:56.556071377 (509h52m56.427154413s) [noise=true , address=45.91.116.4:4937 , piece_id=PIFFBSULE5E5W66XSNS7LPYL3X6E7VQWDWSRCTHPK6GDO2NSSA6Q, seg_pos={0 0}, upload=447428]: Done uploading piece. err: <nil> Dec 26 21:21:56.611635551 (509h52m56.482718596s) [noise=true , address=77.90.31.16:4736 , piece_id=CEWA2SXBQT7HLU2C7VAT23JZO2CUUHNRTX775DQJOCZYC7CYH4UA, seg_pos={0 0}, upload=447428]: Done uploading piece. err: <nil> Dec 26 21:21:56.643811740 (509h52m56.514894775s) [noise=true , address=45.91.116.6:4975 , piece_id=TGMC5IY3OXXHDODBEBREPOSJPS3V6CRVTETTKC27R5ZVZDZWBCIQ, seg_pos={0 0}, upload=447428]: Done uploading piece. err: <nil> Dec 26 21:21:56.648557086 (509h52m56.519640121s) [noise=true , address=77.90.31.13:4629 , piece_id=OFVKCT6RQHHVFTLBH2ZB5VV55LFWUNF74USH24H5OSOGSACSL47A, seg_pos={0 0}, upload=447428]: Done uploading piece. err: <nil> Dec 26 21:21:57.024180380 (509h52m56.895263425s) [noise=true , address=77.90.31.21:4877 , piece_id=NVSMRM2YRKSX7XHP5OWPPSPBA7NW6YGXKHJREPXK3NIEVWXTSUJA, seg_pos={0 0}, upload=447428]: Done uploading piece. err: <nil> Dec 26 21:21:57.039063419 (509h52m56.910146454s) [noise=true , address=145.79.227.34:4907 , piece_id=MSW6FY2AHCLQYOESRHNV66UCLH2OU2EB3I5L2OQJPY225IGWUVNQ, seg_pos={0 0}, upload=447428]: Done uploading piece. err: <nil> Dec 26 21:21:57.039082315 (509h52m56.91016536s) [seg_pos={0 0}, upload=447428]: Segment reached optimal threshold of 13 pieces. Dec 26 21:21:57.039253982 (509h52m56.910337017s) [noise=true , address=45.91.116.5:4297 , piece_id=XPGYUBDXEMDNQBNJLN2RMHLNYGNPDONLAZRDANIVJ6VYUKOHO2VA, seg_pos={0 0}, upload=447428]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 26 21:21:57.039260244 (509h52m56.910343279s) [noise=true , address=45.91.116.5:4297 , piece_id=XPGYUBDXEMDNQBNJLN2RMHLNYGNPDONLAZRDANIVJ6VYUKOHO2VA, seg_pos={0 0}, upload=447428]: Long tail cancellation due to optimal threshold Dec 26 21:21:57.039380733 (509h52m56.910463768s) [noise=true , address=145.79.227.16:4911 , piece_id=RNWWS3Q4SWTRP556HV5DIG3DWMO46YCWFFLKDVA3VFE4FHIN6BFA, seg_pos={0 0}, upload=447428]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: write tcp 188.166.225.170:53390->145.79.227.16:4911: use of closed network connection; write tcp 188.166.225.170:53390->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53390->145.79.227.16:4911: use of closed network connection Dec 26 21:21:57.039385323 (509h52m56.910468358s) [noise=true , address=145.79.227.16:4911 , piece_id=RNWWS3Q4SWTRP556HV5DIG3DWMO46YCWFFLKDVA3VFE4FHIN6BFA, seg_pos={0 0}, upload=447428]: Long tail cancellation due to optimal threshold Dec 26 21:21:57.039638284 (509h52m56.910721319s) [noise=true , address=45.91.116.14:4192 , piece_id=M4MSOB2BL3ZNUKFJMUID5ML6OETFAOWY7TUKNFQZUPURSCRPXJZA, seg_pos={0 0}, upload=447428]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:39414->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:39414->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39414->45.91.116.14:4192: use of closed network connection Dec 26 21:21:57.039653042 (509h52m56.910736077s) [noise=true , address=45.91.116.14:4192 , piece_id=M4MSOB2BL3ZNUKFJMUID5ML6OETFAOWY7TUKNFQZUPURSCRPXJZA, seg_pos={0 0}, upload=447428]: Long tail cancellation due to optimal threshold Dec 26 21:21:57.039696776 (509h52m56.910779821s) [seg_pos={0 0}, upload=447428]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 21:21:57.039791786 (509h52m56.910874821s) [upload=447428]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:21:57.039796996 (509h52m56.910880031s) [upload=447428]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:21:57.336078008 (509h52m57.207161053s) [upload=447428]: Done uploading segments... Dec 26 21:21:57.336099499 (509h52m57.207182534s) [upload=447428]: Upload finished. err: <nil> Duration: 30.002470071s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 18.73682503s trace: https://jaeger.dev.storj.io/trace/6d2950019117adaf FAIL (2025-12-26 21:21:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Rj4hUGWvUPuU49QDsHYhibiJhyY5KDGus1rzNeP7KusgyvTsU, Piece ID: NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A) context deadline exceeded; piecestore: (Node ID: 12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, Piece ID: N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ) context deadline exceeded; piecestore: (Node ID: 12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, Piece ID: IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ) context deadline exceeded ======LOGS====== ng piece... Dec 26 20:54:08.087336805 (509h25m7.95841984s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087649520 (509h25m7.958732565s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087988717 (509h25m7.959071752s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088323233 (509h25m7.959406268s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088877028 (509h25m7.959960063s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089247203 (509h25m7.960330238s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089543066 (509h25m7.960626101s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089914133 (509h25m7.960997168s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090327259 (509h25m7.961410294s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090645345 (509h25m7.96172838s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091004318 (509h25m7.962087353s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091338976 (509h25m7.962422021s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091732054 (509h25m7.962815099s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:09.565187143 (509h25m9.436270188s) [upload=447022]: Next returned nil segment Dec 26 20:54:09.565201321 (509h25m9.436284356s) [upload=447022]: Waiting for error group managing segments... Dec 26 20:54:10.137065972 (509h25m10.008149007s) [noise=true , address=77.90.31.10:4523 , piece_id=GQYT3Z53Y6HVTY3IBUNFCIWBEE7KVAUWEJYO7QJFRRAJCHTJMYFA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.137147197 (509h25m10.008230232s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.149325196 (509h25m10.020408242s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.159611673 (509h25m10.030694718s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.164411764 (509h25m10.035494809s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.190924278 (509h25m10.062007313s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.232931086 (509h25m10.104014131s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.241658245 (509h25m10.11274129s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.472574370 (509h25m10.343657405s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.866910566 (509h25m10.737993611s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.886583507 (509h25m10.757666542s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.127773215 (509h25m10.99885626s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.192133217 (509h25m11.063216263s) [noise=true , address=145.79.227.13:4819 , piece_id=BQDRL2QJZ75TAMUM2VSGTF4U3NEMZC4FFQLN6QV6UOAFHWB56MZA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256146219 (509h25m11.127229264s) [noise=true , address=145.79.227.40:4062 , piece_id=IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256176597 (509h25m11.127259632s) [seg_pos={0 0}, upload=447022]: Segment reached optimal threshold of 14 pieces. Dec 26 20:54:11.256428137 (509h25m11.127511192s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98): protocol: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection Dec 26 20:54:11.256439549 (509h25m11.127522584s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256458915 (509h25m11.12754198s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF): protocol: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection Dec 26 20:54:11.256490705 (509h25m11.127573741s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256698171 (509h25m11.127781216s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: rpc: dial tcp 145.79.227.30:4796: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:54:11.256706457 (509h25m11.127789502s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256917309 (509h25m11.128000364s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection Dec 26 20:54:11.256926666 (509h25m11.128009711s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256964869 (509h25m11.128047914s) [seg_pos={0 0}, upload=447022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:54:11.257064138 (509h25m11.128147173s) [upload=447022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:11.257070690 (509h25m11.128153725s) [upload=447022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:11.561962810 (509h25m11.433045875s) [upload=447022]: Done uploading segments... Dec 26 20:54:11.561987427 (509h25m11.433070472s) [upload=447022]: Upload finished. err: <nil> Duration: 18.736851972s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X.XX..XXX...............X....................XX.XX.X..X.X...] duration: 6.321330943s trace: https://jaeger.dev.storj.io/trace/52a98d2dbac7a51b FAIL (2025-12-26 21:21:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: RN57IIPN3M5M6WOZN7A7YZNNNLYRFHZIUDHRIOUWSLS3FGJVKECA) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: 5IZWBK5B7MQXWCICRR6E36QYK4M63ZJ2B7E45X4IF7W7O4Y4FMJA) context deadline exceeded ======LOGS====== Dec 26 21:18:56.839795227 (509h49m56.710878273s) [upload=447386]: Starting upload Dec 26 21:18:56.839872194 (509h49m56.710955229s) [upload=447386]: Done starting upload Dec 26 21:18:56.839941245 (509h49m56.71102428s) [upload=447386]: Uploading segments... Dec 26 21:18:56.840095990 (509h49m56.711179036s) [upload=447386]: Got next segment. Inline: false Dec 26 21:18:56.840168879 (509h49m56.711251914s) [upload=447386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 21:18:56.840175952 (509h49m56.711258998s) [upload=447386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 21:18:57.087227783 (509h49m56.958310828s) [seg_pos={0 0}, upload=447386]: Begin upload segment... Dec 26 21:18:57.087280504 (509h49m56.958363539s) [seg_pos={0 0}, upload=447386]: Done begin upload segment. Dec 26 21:18:57.087308827 (509h49m56.958391862s) [noise=true , address=145.79.227.39:4013 , piece_id=H7I7UVNY3EH52X63DNL2OWFPZZQO5LURNTSFTZDP5PDLHQBFWTVA, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.090272601 (509h49m56.961355636s) [noise=true , address=77.90.31.21:4878 , piece_id=5IZWBK5B7MQXWCICRR6E36QYK4M63ZJ2B7E45X4IF7W7O4Y4FMJA, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.091104715 (509h49m56.96218775s) [noise=true , address=45.91.116.10:4082 , piece_id=62BXZLLGCGFF2YORSGTVHWFK6GTOOSTR5BTDM7HSNUJL5P3QGTDQ, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.092393389 (509h49m56.963476424s) [noise=true , address=145.79.227.36:4517 , piece_id=PHP2PFEWABX75YRICE3XBBUYN7CBHIHSWURMFU5SU3SWZWALIVEQ, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.093966876 (509h49m56.965049911s) [noise=true , address=45.91.116.6:4973 , piece_id=HNP5WYG7JHR6MCVHF6OVFFTVAQU45INBMIH2WFIV43CNLRUEFAPA, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.094580473 (509h49m56.965663508s) [noise=true , address=145.79.227.30:4762 , piece_id=AFKDGFTGHL2MQK3MBY2PR7XAZCNDR2FFDMZ2I27HRHTO37VSEOZA, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.097678362 (509h49m56.968761407s) [noise=true , address=45.91.116.12:4226 , piece_id=6NRLVJYCQHPNEVXEEXACITPMG73ROWZKRGSIQXR7RDTSQYIMB6HQ, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.099508498 (509h49m56.970591533s) [noise=true , address=145.79.227.35:4482 , piece_id=RIZMSKLNROEITUYIJ7NBZ6OYFMG7KPMWS5SAEWLIT6GJG65YXXPQ, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.100614183 (509h49m56.971697218s) [noise=true , address=145.79.227.37:4565 , piece_id=PPXVNAGKHKNUW5EGUSHFXFYXDW6TAQX3XR2ICUE3KN4Z36SCVNAQ, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.101125126 (509h49m56.972208161s) [noise=true , address=145.79.227.46:4603 , piece_id=4A5LP6BDGBFUIAUHEFBFWYKNGXGPY7BMAR6I72B3VL3P7KOVP7PA, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.102221874 (509h49m56.973304909s) [noise=true , address=77.90.31.15:4700 , piece_id=RN57IIPN3M5M6WOZN7A7YZNNNLYRFHZIUDHRIOUWSLS3FGJVKECA, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.103709337 (509h49m56.974792372s) [noise=true , address=45.91.116.5:4297 , piece_id=IZEKS65I6MH3GUSWOJGOJOY2MON562DPQUHPCC2IW4FZBVBJEKWA, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.105036975 (509h49m56.97612001s) [noise=true , address=145.79.227.45:4230 , piece_id=LF7M7FR3AVO6VTVO5ODB7PHD4BYGKDZGER24M5VHRYENFRK7D5SQ, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.106130256 (509h49m56.977213291s) [noise=true , address=145.79.227.11:4730 , piece_id=BCMYARWGSLUDZ2MOF7DP3JTWXCIJ4MMBULBKVZDQE6OEUTWBSIPQ, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.106541630 (509h49m56.977624675s) [noise=true , address=145.79.227.49:4339 , piece_id=T444IMRG2TRES5AFEUIGULALLDQ67A4G3TYDSECAZ24E4XWS2FEA, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.109591237 (509h49m56.980674282s) [noise=true , address=145.79.227.52:4447 , piece_id=QOP7WPMXARDPUASGMHKPCPIXBM4ZYHZLT6YKQNXCUVIXBOVMXK6Q, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:58.813768592 (509h49m58.684851637s) [upload=447386]: Next returned nil segment Dec 26 21:18:58.813785474 (509h49m58.684868509s) [upload=447386]: Waiting for error group managing segments... Dec 26 21:18:59.097082502 (509h49m58.968165547s) [noise=true , address=145.79.227.52:4447 , piece_id=QOP7WPMXARDPUASGMHKPCPIXBM4ZYHZLT6YKQNXCUVIXBOVMXK6Q, seg_pos={0 0}, upload=447386]: Done uploading piece. err: <nil> Dec 26 21:18:59.103230278 (509h49m58.974313313s) [noise=true , address=145.79.227.35:4482 , piece_id=RIZMSKLNROEITUYIJ7NBZ6OYFMG7KPMWS5SAEWLIT6GJG65YXXPQ, seg_pos={0 0}, upload=447386]: Done uploading piece. err: <nil> Dec 26 21:18:59.106484484 (509h49m58.977567519s) [noise=true , address=145.79.227.45:4230 , piece_id=LF7M7FR3AVO6VTVO5ODB7PHD4BYGKDZGER24M5VHRYENFRK7D5SQ, seg_pos={0 0}, upload=447386]: Done uploading piece. err: <nil> Dec 26 21:18:59.109972736 (509h49m58.981055771s) [noise=true , address=145.79.227.46:4603 , piece_id=4A5LP6BDGBFUIAUHEFBFWYKNGXGPY7BMAR6I72B3VL3P7KOVP7PA, seg_pos={0 0}, upload=447386]: Done uploading piece. err: <nil> Dec 26 21:18:59.113482520 (509h49m58.984565555s) [noise=true , address=145.79.227.37:4565 , piece_id=PPXVNAGKHKNUW5EGUSHFXFYXDW6TAQX3XR2ICUE3KN4Z36SCVNAQ, seg_pos={0 0}, upload=447386]: Done uploading piece. err: <nil> Dec 26 21:18:59.115557742 (509h49m58.986640777s) [noise=true , address=145.79.227.39:4013 , piece_id=H7I7UVNY3EH52X63DNL2OWFPZZQO5LURNTSFTZDP5PDLHQBFWTVA, seg_pos={0 0}, upload=447386]: Done uploading piece. err: <nil> Dec 26 21:18:59.120013276 (509h49m58.991096311s) [noise=true , address=145.79.227.36:4517 , piece_id=PHP2PFEWABX75YRICE3XBBUYN7CBHIHSWURMFU5SU3SWZWALIVEQ, seg_pos={0 0}, upload=447386]: Done uploading piece. err: <nil> Dec 26 21:18:59.138557387 (509h49m59.009640422s) [noise=true , address=145.79.227.49:4339 , piece_id=T444IMRG2TRES5AFEUIGULALLDQ67A4G3TYDSECAZ24E4XWS2FEA, seg_pos={0 0}, upload=447386]: Done uploading piece. err: <nil> Dec 26 21:18:59.343914273 (509h49m59.214997318s) [noise=true , address=45.91.116.5:4297 , piece_id=IZEKS65I6MH3GUSWOJGOJOY2MON562DPQUHPCC2IW4FZBVBJEKWA, seg_pos={0 0}, upload=447386]: Done uploading piece. err: <nil> Dec 26 21:18:59.459298776 (509h49m59.330381811s) [noise=true , address=45.91.116.12:4226 , piece_id=6NRLVJYCQHPNEVXEEXACITPMG73ROWZKRGSIQXR7RDTSQYIMB6HQ, seg_pos={0 0}, upload=447386]: Done uploading piece. err: <nil> Dec 26 21:18:59.465685226 (509h49m59.336768261s) [noise=true , address=45.91.116.10:4082 , piece_id=62BXZLLGCGFF2YORSGTVHWFK6GTOOSTR5BTDM7HSNUJL5P3QGTDQ, seg_pos={0 0}, upload=447386]: Done uploading piece. err: <nil> Dec 26 21:18:59.494136463 (509h49m59.365219498s) [noise=true , address=77.90.31.21:4878 , piece_id=5IZWBK5B7MQXWCICRR6E36QYK4M63ZJ2B7E45X4IF7W7O4Y4FMJA, seg_pos={0 0}, upload=447386]: Done uploading piece. err: <nil> Dec 26 21:18:59.739373698 (509h49m59.610456733s) [noise=true , address=77.90.31.15:4700 , piece_id=RN57IIPN3M5M6WOZN7A7YZNNNLYRFHZIUDHRIOUWSLS3FGJVKECA, seg_pos={0 0}, upload=447386]: Done uploading piece. err: <nil> Dec 26 21:18:59.739397604 (509h49m59.610480649s) [seg_pos={0 0}, upload=447386]: Segment reached optimal threshold of 13 pieces. Dec 26 21:18:59.739638092 (509h49m59.610721147s) [noise=true , address=45.91.116.6:4973 , piece_id=HNP5WYG7JHR6MCVHF6OVFFTVAQU45INBMIH2WFIV43CNLRUEFAPA, seg_pos={0 0}, upload=447386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 26 21:18:59.739652338 (509h49m59.610735384s) [noise=true , address=45.91.116.6:4973 , piece_id=HNP5WYG7JHR6MCVHF6OVFFTVAQU45INBMIH2WFIV43CNLRUEFAPA, seg_pos={0 0}, upload=447386]: Long tail cancellation due to optimal threshold Dec 26 21:18:59.739737371 (509h49m59.610820406s) [noise=true , address=145.79.227.30:4762 , piece_id=AFKDGFTGHL2MQK3MBY2PR7XAZCNDR2FFDMZ2I27HRHTO37VSEOZA, seg_pos={0 0}, upload=447386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 26 21:18:59.739751397 (509h49m59.610834432s) [noise=true , address=145.79.227.30:4762 , piece_id=AFKDGFTGHL2MQK3MBY2PR7XAZCNDR2FFDMZ2I27HRHTO37VSEOZA, seg_pos={0 0}, upload=447386]: Long tail cancellation due to optimal threshold Dec 26 21:18:59.739953262 (509h49m59.611036297s) [noise=true , address=145.79.227.11:4730 , piece_id=BCMYARWGSLUDZ2MOF7DP3JTWXCIJ4MMBULBKVZDQE6OEUTWBSIPQ, seg_pos={0 0}, upload=447386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: expected piece hash; context canceled; EOF Dec 26 21:18:59.739960696 (509h49m59.611043731s) [noise=true , address=145.79.227.11:4730 , piece_id=BCMYARWGSLUDZ2MOF7DP3JTWXCIJ4MMBULBKVZDQE6OEUTWBSIPQ, seg_pos={0 0}, upload=447386]: Long tail cancellation due to optimal threshold Dec 26 21:18:59.739979582 (509h49m59.611062617s) [seg_pos={0 0}, upload=447386]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 21:18:59.740071196 (509h49m59.611154231s) [upload=447386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:18:59.740076255 (509h49m59.611159291s) [upload=447386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:19:00.037299622 (509h49m59.908382657s) [upload=447386]: Done uploading segments... Dec 26 21:19:00.037322675 (509h49m59.90840572s) [upload=447386]: Upload finished. err: <nil> Duration: 6.321356581s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 7.415209399s trace: https://jaeger.dev.storj.io/trace/6d2950019117adaf FAIL (2025-12-26 21:21:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Rj4hUGWvUPuU49QDsHYhibiJhyY5KDGus1rzNeP7KusgyvTsU, Piece ID: NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A) context deadline exceeded; piecestore: (Node ID: 12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, Piece ID: N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ) context deadline exceeded; piecestore: (Node ID: 12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, Piece ID: IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ) context deadline exceeded ======LOGS====== ng piece... Dec 26 20:54:08.087336805 (509h25m7.95841984s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087649520 (509h25m7.958732565s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087988717 (509h25m7.959071752s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088323233 (509h25m7.959406268s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088877028 (509h25m7.959960063s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089247203 (509h25m7.960330238s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089543066 (509h25m7.960626101s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089914133 (509h25m7.960997168s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090327259 (509h25m7.961410294s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090645345 (509h25m7.96172838s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091004318 (509h25m7.962087353s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091338976 (509h25m7.962422021s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091732054 (509h25m7.962815099s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:09.565187143 (509h25m9.436270188s) [upload=447022]: Next returned nil segment Dec 26 20:54:09.565201321 (509h25m9.436284356s) [upload=447022]: Waiting for error group managing segments... Dec 26 20:54:10.137065972 (509h25m10.008149007s) [noise=true , address=77.90.31.10:4523 , piece_id=GQYT3Z53Y6HVTY3IBUNFCIWBEE7KVAUWEJYO7QJFRRAJCHTJMYFA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.137147197 (509h25m10.008230232s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.149325196 (509h25m10.020408242s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.159611673 (509h25m10.030694718s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.164411764 (509h25m10.035494809s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.190924278 (509h25m10.062007313s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.232931086 (509h25m10.104014131s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.241658245 (509h25m10.11274129s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.472574370 (509h25m10.343657405s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.866910566 (509h25m10.737993611s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.886583507 (509h25m10.757666542s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.127773215 (509h25m10.99885626s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.192133217 (509h25m11.063216263s) [noise=true , address=145.79.227.13:4819 , piece_id=BQDRL2QJZ75TAMUM2VSGTF4U3NEMZC4FFQLN6QV6UOAFHWB56MZA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256146219 (509h25m11.127229264s) [noise=true , address=145.79.227.40:4062 , piece_id=IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256176597 (509h25m11.127259632s) [seg_pos={0 0}, upload=447022]: Segment reached optimal threshold of 14 pieces. Dec 26 20:54:11.256428137 (509h25m11.127511192s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98): protocol: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection Dec 26 20:54:11.256439549 (509h25m11.127522584s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256458915 (509h25m11.12754198s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF): protocol: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection Dec 26 20:54:11.256490705 (509h25m11.127573741s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256698171 (509h25m11.127781216s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: rpc: dial tcp 145.79.227.30:4796: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:54:11.256706457 (509h25m11.127789502s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256917309 (509h25m11.128000364s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection Dec 26 20:54:11.256926666 (509h25m11.128009711s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256964869 (509h25m11.128047914s) [seg_pos={0 0}, upload=447022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:54:11.257064138 (509h25m11.128147173s) [upload=447022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:11.257070690 (509h25m11.128153725s) [upload=447022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:11.561962810 (509h25m11.433045875s) [upload=447022]: Done uploading segments... Dec 26 20:54:11.561987427 (509h25m11.433070472s) [upload=447022]: Upload finished. err: <nil> Duration: 7.415244257s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 6.446722822s trace: https://jaeger.dev.storj.io/trace/6d2950019117adaf FAIL (2025-12-26 21:20:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Rj4hUGWvUPuU49QDsHYhibiJhyY5KDGus1rzNeP7KusgyvTsU, Piece ID: NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A) context deadline exceeded; piecestore: (Node ID: 12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, Piece ID: N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ) context deadline exceeded; piecestore: (Node ID: 12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, Piece ID: IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ) context deadline exceeded ======LOGS====== ng piece... Dec 26 20:54:08.087336805 (509h25m7.95841984s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087649520 (509h25m7.958732565s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087988717 (509h25m7.959071752s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088323233 (509h25m7.959406268s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088877028 (509h25m7.959960063s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089247203 (509h25m7.960330238s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089543066 (509h25m7.960626101s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089914133 (509h25m7.960997168s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090327259 (509h25m7.961410294s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090645345 (509h25m7.96172838s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091004318 (509h25m7.962087353s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091338976 (509h25m7.962422021s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091732054 (509h25m7.962815099s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:09.565187143 (509h25m9.436270188s) [upload=447022]: Next returned nil segment Dec 26 20:54:09.565201321 (509h25m9.436284356s) [upload=447022]: Waiting for error group managing segments... Dec 26 20:54:10.137065972 (509h25m10.008149007s) [noise=true , address=77.90.31.10:4523 , piece_id=GQYT3Z53Y6HVTY3IBUNFCIWBEE7KVAUWEJYO7QJFRRAJCHTJMYFA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.137147197 (509h25m10.008230232s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.149325196 (509h25m10.020408242s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.159611673 (509h25m10.030694718s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.164411764 (509h25m10.035494809s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.190924278 (509h25m10.062007313s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.232931086 (509h25m10.104014131s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.241658245 (509h25m10.11274129s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.472574370 (509h25m10.343657405s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.866910566 (509h25m10.737993611s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.886583507 (509h25m10.757666542s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.127773215 (509h25m10.99885626s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.192133217 (509h25m11.063216263s) [noise=true , address=145.79.227.13:4819 , piece_id=BQDRL2QJZ75TAMUM2VSGTF4U3NEMZC4FFQLN6QV6UOAFHWB56MZA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256146219 (509h25m11.127229264s) [noise=true , address=145.79.227.40:4062 , piece_id=IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256176597 (509h25m11.127259632s) [seg_pos={0 0}, upload=447022]: Segment reached optimal threshold of 14 pieces. Dec 26 20:54:11.256428137 (509h25m11.127511192s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98): protocol: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection Dec 26 20:54:11.256439549 (509h25m11.127522584s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256458915 (509h25m11.12754198s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF): protocol: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection Dec 26 20:54:11.256490705 (509h25m11.127573741s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256698171 (509h25m11.127781216s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: rpc: dial tcp 145.79.227.30:4796: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:54:11.256706457 (509h25m11.127789502s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256917309 (509h25m11.128000364s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection Dec 26 20:54:11.256926666 (509h25m11.128009711s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256964869 (509h25m11.128047914s) [seg_pos={0 0}, upload=447022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:54:11.257064138 (509h25m11.128147173s) [upload=447022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:11.257070690 (509h25m11.128153725s) [upload=447022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:11.561962810 (509h25m11.433045875s) [upload=447022]: Done uploading segments... Dec 26 20:54:11.561987427 (509h25m11.433070472s) [upload=447022]: Upload finished. err: <nil> Duration: 6.446782495s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.X.XX..XXX...............X....................XX.XX.X..X.X..] duration: 6.297460149s trace: https://jaeger.dev.storj.io/trace/52a98d2dbac7a51b FAIL (2025-12-26 21:19:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: RN57IIPN3M5M6WOZN7A7YZNNNLYRFHZIUDHRIOUWSLS3FGJVKECA) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: 5IZWBK5B7MQXWCICRR6E36QYK4M63ZJ2B7E45X4IF7W7O4Y4FMJA) context deadline exceeded ======LOGS====== Dec 26 21:18:56.839795227 (509h49m56.710878273s) [upload=447386]: Starting upload Dec 26 21:18:56.839872194 (509h49m56.710955229s) [upload=447386]: Done starting upload Dec 26 21:18:56.839941245 (509h49m56.71102428s) [upload=447386]: Uploading segments... Dec 26 21:18:56.840095990 (509h49m56.711179036s) [upload=447386]: Got next segment. Inline: false Dec 26 21:18:56.840168879 (509h49m56.711251914s) [upload=447386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 21:18:56.840175952 (509h49m56.711258998s) [upload=447386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 21:18:57.087227783 (509h49m56.958310828s) [seg_pos={0 0}, upload=447386]: Begin upload segment... Dec 26 21:18:57.087280504 (509h49m56.958363539s) [seg_pos={0 0}, upload=447386]: Done begin upload segment. Dec 26 21:18:57.087308827 (509h49m56.958391862s) [noise=true , address=145.79.227.39:4013 , piece_id=H7I7UVNY3EH52X63DNL2OWFPZZQO5LURNTSFTZDP5PDLHQBFWTVA, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.090272601 (509h49m56.961355636s) [noise=true , address=77.90.31.21:4878 , piece_id=5IZWBK5B7MQXWCICRR6E36QYK4M63ZJ2B7E45X4IF7W7O4Y4FMJA, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.091104715 (509h49m56.96218775s) [noise=true , address=45.91.116.10:4082 , piece_id=62BXZLLGCGFF2YORSGTVHWFK6GTOOSTR5BTDM7HSNUJL5P3QGTDQ, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.092393389 (509h49m56.963476424s) [noise=true , address=145.79.227.36:4517 , piece_id=PHP2PFEWABX75YRICE3XBBUYN7CBHIHSWURMFU5SU3SWZWALIVEQ, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.093966876 (509h49m56.965049911s) [noise=true , address=45.91.116.6:4973 , piece_id=HNP5WYG7JHR6MCVHF6OVFFTVAQU45INBMIH2WFIV43CNLRUEFAPA, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.094580473 (509h49m56.965663508s) [noise=true , address=145.79.227.30:4762 , piece_id=AFKDGFTGHL2MQK3MBY2PR7XAZCNDR2FFDMZ2I27HRHTO37VSEOZA, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.097678362 (509h49m56.968761407s) [noise=true , address=45.91.116.12:4226 , piece_id=6NRLVJYCQHPNEVXEEXACITPMG73ROWZKRGSIQXR7RDTSQYIMB6HQ, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.099508498 (509h49m56.970591533s) [noise=true , address=145.79.227.35:4482 , piece_id=RIZMSKLNROEITUYIJ7NBZ6OYFMG7KPMWS5SAEWLIT6GJG65YXXPQ, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.100614183 (509h49m56.971697218s) [noise=true , address=145.79.227.37:4565 , piece_id=PPXVNAGKHKNUW5EGUSHFXFYXDW6TAQX3XR2ICUE3KN4Z36SCVNAQ, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.101125126 (509h49m56.972208161s) [noise=true , address=145.79.227.46:4603 , piece_id=4A5LP6BDGBFUIAUHEFBFWYKNGXGPY7BMAR6I72B3VL3P7KOVP7PA, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.102221874 (509h49m56.973304909s) [noise=true , address=77.90.31.15:4700 , piece_id=RN57IIPN3M5M6WOZN7A7YZNNNLYRFHZIUDHRIOUWSLS3FGJVKECA, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.103709337 (509h49m56.974792372s) [noise=true , address=45.91.116.5:4297 , piece_id=IZEKS65I6MH3GUSWOJGOJOY2MON562DPQUHPCC2IW4FZBVBJEKWA, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.105036975 (509h49m56.97612001s) [noise=true , address=145.79.227.45:4230 , piece_id=LF7M7FR3AVO6VTVO5ODB7PHD4BYGKDZGER24M5VHRYENFRK7D5SQ, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.106130256 (509h49m56.977213291s) [noise=true , address=145.79.227.11:4730 , piece_id=BCMYARWGSLUDZ2MOF7DP3JTWXCIJ4MMBULBKVZDQE6OEUTWBSIPQ, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.106541630 (509h49m56.977624675s) [noise=true , address=145.79.227.49:4339 , piece_id=T444IMRG2TRES5AFEUIGULALLDQ67A4G3TYDSECAZ24E4XWS2FEA, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.109591237 (509h49m56.980674282s) [noise=true , address=145.79.227.52:4447 , piece_id=QOP7WPMXARDPUASGMHKPCPIXBM4ZYHZLT6YKQNXCUVIXBOVMXK6Q, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:58.813768592 (509h49m58.684851637s) [upload=447386]: Next returned nil segment Dec 26 21:18:58.813785474 (509h49m58.684868509s) [upload=447386]: Waiting for error group managing segments... Dec 26 21:18:59.097082502 (509h49m58.968165547s) [noise=true , address=145.79.227.52:4447 , piece_id=QOP7WPMXARDPUASGMHKPCPIXBM4ZYHZLT6YKQNXCUVIXBOVMXK6Q, seg_pos={0 0}, upload=447386]: Done uploading piece. err: <nil> Dec 26 21:18:59.103230278 (509h49m58.974313313s) [noise=true , address=145.79.227.35:4482 , piece_id=RIZMSKLNROEITUYIJ7NBZ6OYFMG7KPMWS5SAEWLIT6GJG65YXXPQ, seg_pos={0 0}, upload=447386]: Done uploading piece. err: <nil> Dec 26 21:18:59.106484484 (509h49m58.977567519s) [noise=true , address=145.79.227.45:4230 , piece_id=LF7M7FR3AVO6VTVO5ODB7PHD4BYGKDZGER24M5VHRYENFRK7D5SQ, seg_pos={0 0}, upload=447386]: Done uploading piece. err: <nil> Dec 26 21:18:59.109972736 (509h49m58.981055771s) [noise=true , address=145.79.227.46:4603 , piece_id=4A5LP6BDGBFUIAUHEFBFWYKNGXGPY7BMAR6I72B3VL3P7KOVP7PA, seg_pos={0 0}, upload=447386]: Done uploading piece. err: <nil> Dec 26 21:18:59.113482520 (509h49m58.984565555s) [noise=true , address=145.79.227.37:4565 , piece_id=PPXVNAGKHKNUW5EGUSHFXFYXDW6TAQX3XR2ICUE3KN4Z36SCVNAQ, seg_pos={0 0}, upload=447386]: Done uploading piece. err: <nil> Dec 26 21:18:59.115557742 (509h49m58.986640777s) [noise=true , address=145.79.227.39:4013 , piece_id=H7I7UVNY3EH52X63DNL2OWFPZZQO5LURNTSFTZDP5PDLHQBFWTVA, seg_pos={0 0}, upload=447386]: Done uploading piece. err: <nil> Dec 26 21:18:59.120013276 (509h49m58.991096311s) [noise=true , address=145.79.227.36:4517 , piece_id=PHP2PFEWABX75YRICE3XBBUYN7CBHIHSWURMFU5SU3SWZWALIVEQ, seg_pos={0 0}, upload=447386]: Done uploading piece. err: <nil> Dec 26 21:18:59.138557387 (509h49m59.009640422s) [noise=true , address=145.79.227.49:4339 , piece_id=T444IMRG2TRES5AFEUIGULALLDQ67A4G3TYDSECAZ24E4XWS2FEA, seg_pos={0 0}, upload=447386]: Done uploading piece. err: <nil> Dec 26 21:18:59.343914273 (509h49m59.214997318s) [noise=true , address=45.91.116.5:4297 , piece_id=IZEKS65I6MH3GUSWOJGOJOY2MON562DPQUHPCC2IW4FZBVBJEKWA, seg_pos={0 0}, upload=447386]: Done uploading piece. err: <nil> Dec 26 21:18:59.459298776 (509h49m59.330381811s) [noise=true , address=45.91.116.12:4226 , piece_id=6NRLVJYCQHPNEVXEEXACITPMG73ROWZKRGSIQXR7RDTSQYIMB6HQ, seg_pos={0 0}, upload=447386]: Done uploading piece. err: <nil> Dec 26 21:18:59.465685226 (509h49m59.336768261s) [noise=true , address=45.91.116.10:4082 , piece_id=62BXZLLGCGFF2YORSGTVHWFK6GTOOSTR5BTDM7HSNUJL5P3QGTDQ, seg_pos={0 0}, upload=447386]: Done uploading piece. err: <nil> Dec 26 21:18:59.494136463 (509h49m59.365219498s) [noise=true , address=77.90.31.21:4878 , piece_id=5IZWBK5B7MQXWCICRR6E36QYK4M63ZJ2B7E45X4IF7W7O4Y4FMJA, seg_pos={0 0}, upload=447386]: Done uploading piece. err: <nil> Dec 26 21:18:59.739373698 (509h49m59.610456733s) [noise=true , address=77.90.31.15:4700 , piece_id=RN57IIPN3M5M6WOZN7A7YZNNNLYRFHZIUDHRIOUWSLS3FGJVKECA, seg_pos={0 0}, upload=447386]: Done uploading piece. err: <nil> Dec 26 21:18:59.739397604 (509h49m59.610480649s) [seg_pos={0 0}, upload=447386]: Segment reached optimal threshold of 13 pieces. Dec 26 21:18:59.739638092 (509h49m59.610721147s) [noise=true , address=45.91.116.6:4973 , piece_id=HNP5WYG7JHR6MCVHF6OVFFTVAQU45INBMIH2WFIV43CNLRUEFAPA, seg_pos={0 0}, upload=447386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 26 21:18:59.739652338 (509h49m59.610735384s) [noise=true , address=45.91.116.6:4973 , piece_id=HNP5WYG7JHR6MCVHF6OVFFTVAQU45INBMIH2WFIV43CNLRUEFAPA, seg_pos={0 0}, upload=447386]: Long tail cancellation due to optimal threshold Dec 26 21:18:59.739737371 (509h49m59.610820406s) [noise=true , address=145.79.227.30:4762 , piece_id=AFKDGFTGHL2MQK3MBY2PR7XAZCNDR2FFDMZ2I27HRHTO37VSEOZA, seg_pos={0 0}, upload=447386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 26 21:18:59.739751397 (509h49m59.610834432s) [noise=true , address=145.79.227.30:4762 , piece_id=AFKDGFTGHL2MQK3MBY2PR7XAZCNDR2FFDMZ2I27HRHTO37VSEOZA, seg_pos={0 0}, upload=447386]: Long tail cancellation due to optimal threshold Dec 26 21:18:59.739953262 (509h49m59.611036297s) [noise=true , address=145.79.227.11:4730 , piece_id=BCMYARWGSLUDZ2MOF7DP3JTWXCIJ4MMBULBKVZDQE6OEUTWBSIPQ, seg_pos={0 0}, upload=447386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: expected piece hash; context canceled; EOF Dec 26 21:18:59.739960696 (509h49m59.611043731s) [noise=true , address=145.79.227.11:4730 , piece_id=BCMYARWGSLUDZ2MOF7DP3JTWXCIJ4MMBULBKVZDQE6OEUTWBSIPQ, seg_pos={0 0}, upload=447386]: Long tail cancellation due to optimal threshold Dec 26 21:18:59.739979582 (509h49m59.611062617s) [seg_pos={0 0}, upload=447386]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 21:18:59.740071196 (509h49m59.611154231s) [upload=447386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:18:59.740076255 (509h49m59.611159291s) [upload=447386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:19:00.037299622 (509h49m59.908382657s) [upload=447386]: Done uploading segments... Dec 26 21:19:00.037322675 (509h49m59.90840572s) [upload=447386]: Upload finished. err: <nil> Duration: 6.297708191s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..X.XX..XXX...............X....................XX.XX.X..X.X.] duration: 6.165207775s trace: https://jaeger.dev.storj.io/trace/52a98d2dbac7a51b FAIL (2025-12-26 21:19:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: RN57IIPN3M5M6WOZN7A7YZNNNLYRFHZIUDHRIOUWSLS3FGJVKECA) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: 5IZWBK5B7MQXWCICRR6E36QYK4M63ZJ2B7E45X4IF7W7O4Y4FMJA) context deadline exceeded ======LOGS====== Dec 26 21:18:56.839795227 (509h49m56.710878273s) [upload=447386]: Starting upload Dec 26 21:18:56.839872194 (509h49m56.710955229s) [upload=447386]: Done starting upload Dec 26 21:18:56.839941245 (509h49m56.71102428s) [upload=447386]: Uploading segments... Dec 26 21:18:56.840095990 (509h49m56.711179036s) [upload=447386]: Got next segment. Inline: false Dec 26 21:18:56.840168879 (509h49m56.711251914s) [upload=447386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 21:18:56.840175952 (509h49m56.711258998s) [upload=447386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 21:18:57.087227783 (509h49m56.958310828s) [seg_pos={0 0}, upload=447386]: Begin upload segment... Dec 26 21:18:57.087280504 (509h49m56.958363539s) [seg_pos={0 0}, upload=447386]: Done begin upload segment. Dec 26 21:18:57.087308827 (509h49m56.958391862s) [noise=true , address=145.79.227.39:4013 , piece_id=H7I7UVNY3EH52X63DNL2OWFPZZQO5LURNTSFTZDP5PDLHQBFWTVA, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.090272601 (509h49m56.961355636s) [noise=true , address=77.90.31.21:4878 , piece_id=5IZWBK5B7MQXWCICRR6E36QYK4M63ZJ2B7E45X4IF7W7O4Y4FMJA, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.091104715 (509h49m56.96218775s) [noise=true , address=45.91.116.10:4082 , piece_id=62BXZLLGCGFF2YORSGTVHWFK6GTOOSTR5BTDM7HSNUJL5P3QGTDQ, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.092393389 (509h49m56.963476424s) [noise=true , address=145.79.227.36:4517 , piece_id=PHP2PFEWABX75YRICE3XBBUYN7CBHIHSWURMFU5SU3SWZWALIVEQ, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.093966876 (509h49m56.965049911s) [noise=true , address=45.91.116.6:4973 , piece_id=HNP5WYG7JHR6MCVHF6OVFFTVAQU45INBMIH2WFIV43CNLRUEFAPA, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.094580473 (509h49m56.965663508s) [noise=true , address=145.79.227.30:4762 , piece_id=AFKDGFTGHL2MQK3MBY2PR7XAZCNDR2FFDMZ2I27HRHTO37VSEOZA, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.097678362 (509h49m56.968761407s) [noise=true , address=45.91.116.12:4226 , piece_id=6NRLVJYCQHPNEVXEEXACITPMG73ROWZKRGSIQXR7RDTSQYIMB6HQ, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.099508498 (509h49m56.970591533s) [noise=true , address=145.79.227.35:4482 , piece_id=RIZMSKLNROEITUYIJ7NBZ6OYFMG7KPMWS5SAEWLIT6GJG65YXXPQ, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.100614183 (509h49m56.971697218s) [noise=true , address=145.79.227.37:4565 , piece_id=PPXVNAGKHKNUW5EGUSHFXFYXDW6TAQX3XR2ICUE3KN4Z36SCVNAQ, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.101125126 (509h49m56.972208161s) [noise=true , address=145.79.227.46:4603 , piece_id=4A5LP6BDGBFUIAUHEFBFWYKNGXGPY7BMAR6I72B3VL3P7KOVP7PA, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.102221874 (509h49m56.973304909s) [noise=true , address=77.90.31.15:4700 , piece_id=RN57IIPN3M5M6WOZN7A7YZNNNLYRFHZIUDHRIOUWSLS3FGJVKECA, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.103709337 (509h49m56.974792372s) [noise=true , address=45.91.116.5:4297 , piece_id=IZEKS65I6MH3GUSWOJGOJOY2MON562DPQUHPCC2IW4FZBVBJEKWA, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.105036975 (509h49m56.97612001s) [noise=true , address=145.79.227.45:4230 , piece_id=LF7M7FR3AVO6VTVO5ODB7PHD4BYGKDZGER24M5VHRYENFRK7D5SQ, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.106130256 (509h49m56.977213291s) [noise=true , address=145.79.227.11:4730 , piece_id=BCMYARWGSLUDZ2MOF7DP3JTWXCIJ4MMBULBKVZDQE6OEUTWBSIPQ, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.106541630 (509h49m56.977624675s) [noise=true , address=145.79.227.49:4339 , piece_id=T444IMRG2TRES5AFEUIGULALLDQ67A4G3TYDSECAZ24E4XWS2FEA, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.109591237 (509h49m56.980674282s) [noise=true , address=145.79.227.52:4447 , piece_id=QOP7WPMXARDPUASGMHKPCPIXBM4ZYHZLT6YKQNXCUVIXBOVMXK6Q, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:58.813768592 (509h49m58.684851637s) [upload=447386]: Next returned nil segment Dec 26 21:18:58.813785474 (509h49m58.684868509s) [upload=447386]: Waiting for error group managing segments... Dec 26 21:18:59.097082502 (509h49m58.968165547s) [noise=true , address=145.79.227.52:4447 , piece_id=QOP7WPMXARDPUASGMHKPCPIXBM4ZYHZLT6YKQNXCUVIXBOVMXK6Q, seg_pos={0 0}, upload=447386]: Done uploading piece. err: <nil> Dec 26 21:18:59.103230278 (509h49m58.974313313s) [noise=true , address=145.79.227.35:4482 , piece_id=RIZMSKLNROEITUYIJ7NBZ6OYFMG7KPMWS5SAEWLIT6GJG65YXXPQ, seg_pos={0 0}, upload=447386]: Done uploading piece. err: <nil> Dec 26 21:18:59.106484484 (509h49m58.977567519s) [noise=true , address=145.79.227.45:4230 , piece_id=LF7M7FR3AVO6VTVO5ODB7PHD4BYGKDZGER24M5VHRYENFRK7D5SQ, seg_pos={0 0}, upload=447386]: Done uploading piece. err: <nil> Dec 26 21:18:59.109972736 (509h49m58.981055771s) [noise=true , address=145.79.227.46:4603 , piece_id=4A5LP6BDGBFUIAUHEFBFWYKNGXGPY7BMAR6I72B3VL3P7KOVP7PA, seg_pos={0 0}, upload=447386]: Done uploading piece. err: <nil> Dec 26 21:18:59.113482520 (509h49m58.984565555s) [noise=true , address=145.79.227.37:4565 , piece_id=PPXVNAGKHKNUW5EGUSHFXFYXDW6TAQX3XR2ICUE3KN4Z36SCVNAQ, seg_pos={0 0}, upload=447386]: Done uploading piece. err: <nil> Dec 26 21:18:59.115557742 (509h49m58.986640777s) [noise=true , address=145.79.227.39:4013 , piece_id=H7I7UVNY3EH52X63DNL2OWFPZZQO5LURNTSFTZDP5PDLHQBFWTVA, seg_pos={0 0}, upload=447386]: Done uploading piece. err: <nil> Dec 26 21:18:59.120013276 (509h49m58.991096311s) [noise=true , address=145.79.227.36:4517 , piece_id=PHP2PFEWABX75YRICE3XBBUYN7CBHIHSWURMFU5SU3SWZWALIVEQ, seg_pos={0 0}, upload=447386]: Done uploading piece. err: <nil> Dec 26 21:18:59.138557387 (509h49m59.009640422s) [noise=true , address=145.79.227.49:4339 , piece_id=T444IMRG2TRES5AFEUIGULALLDQ67A4G3TYDSECAZ24E4XWS2FEA, seg_pos={0 0}, upload=447386]: Done uploading piece. err: <nil> Dec 26 21:18:59.343914273 (509h49m59.214997318s) [noise=true , address=45.91.116.5:4297 , piece_id=IZEKS65I6MH3GUSWOJGOJOY2MON562DPQUHPCC2IW4FZBVBJEKWA, seg_pos={0 0}, upload=447386]: Done uploading piece. err: <nil> Dec 26 21:18:59.459298776 (509h49m59.330381811s) [noise=true , address=45.91.116.12:4226 , piece_id=6NRLVJYCQHPNEVXEEXACITPMG73ROWZKRGSIQXR7RDTSQYIMB6HQ, seg_pos={0 0}, upload=447386]: Done uploading piece. err: <nil> Dec 26 21:18:59.465685226 (509h49m59.336768261s) [noise=true , address=45.91.116.10:4082 , piece_id=62BXZLLGCGFF2YORSGTVHWFK6GTOOSTR5BTDM7HSNUJL5P3QGTDQ, seg_pos={0 0}, upload=447386]: Done uploading piece. err: <nil> Dec 26 21:18:59.494136463 (509h49m59.365219498s) [noise=true , address=77.90.31.21:4878 , piece_id=5IZWBK5B7MQXWCICRR6E36QYK4M63ZJ2B7E45X4IF7W7O4Y4FMJA, seg_pos={0 0}, upload=447386]: Done uploading piece. err: <nil> Dec 26 21:18:59.739373698 (509h49m59.610456733s) [noise=true , address=77.90.31.15:4700 , piece_id=RN57IIPN3M5M6WOZN7A7YZNNNLYRFHZIUDHRIOUWSLS3FGJVKECA, seg_pos={0 0}, upload=447386]: Done uploading piece. err: <nil> Dec 26 21:18:59.739397604 (509h49m59.610480649s) [seg_pos={0 0}, upload=447386]: Segment reached optimal threshold of 13 pieces. Dec 26 21:18:59.739638092 (509h49m59.610721147s) [noise=true , address=45.91.116.6:4973 , piece_id=HNP5WYG7JHR6MCVHF6OVFFTVAQU45INBMIH2WFIV43CNLRUEFAPA, seg_pos={0 0}, upload=447386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 26 21:18:59.739652338 (509h49m59.610735384s) [noise=true , address=45.91.116.6:4973 , piece_id=HNP5WYG7JHR6MCVHF6OVFFTVAQU45INBMIH2WFIV43CNLRUEFAPA, seg_pos={0 0}, upload=447386]: Long tail cancellation due to optimal threshold Dec 26 21:18:59.739737371 (509h49m59.610820406s) [noise=true , address=145.79.227.30:4762 , piece_id=AFKDGFTGHL2MQK3MBY2PR7XAZCNDR2FFDMZ2I27HRHTO37VSEOZA, seg_pos={0 0}, upload=447386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 26 21:18:59.739751397 (509h49m59.610834432s) [noise=true , address=145.79.227.30:4762 , piece_id=AFKDGFTGHL2MQK3MBY2PR7XAZCNDR2FFDMZ2I27HRHTO37VSEOZA, seg_pos={0 0}, upload=447386]: Long tail cancellation due to optimal threshold Dec 26 21:18:59.739953262 (509h49m59.611036297s) [noise=true , address=145.79.227.11:4730 , piece_id=BCMYARWGSLUDZ2MOF7DP3JTWXCIJ4MMBULBKVZDQE6OEUTWBSIPQ, seg_pos={0 0}, upload=447386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: expected piece hash; context canceled; EOF Dec 26 21:18:59.739960696 (509h49m59.611043731s) [noise=true , address=145.79.227.11:4730 , piece_id=BCMYARWGSLUDZ2MOF7DP3JTWXCIJ4MMBULBKVZDQE6OEUTWBSIPQ, seg_pos={0 0}, upload=447386]: Long tail cancellation due to optimal threshold Dec 26 21:18:59.739979582 (509h49m59.611062617s) [seg_pos={0 0}, upload=447386]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 21:18:59.740071196 (509h49m59.611154231s) [upload=447386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:18:59.740076255 (509h49m59.611159291s) [upload=447386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:19:00.037299622 (509h49m59.908382657s) [upload=447386]: Done uploading segments... Dec 26 21:19:00.037322675 (509h49m59.90840572s) [upload=447386]: Upload finished. err: <nil> Duration: 6.165224348s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [...X.XX..XXX...............X....................XX.XX.X..X.X] duration: 30.002558019s trace: https://jaeger.dev.storj.io/trace/52a98d2dbac7a51b FAIL (2025-12-26 21:19:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: RN57IIPN3M5M6WOZN7A7YZNNNLYRFHZIUDHRIOUWSLS3FGJVKECA) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: 5IZWBK5B7MQXWCICRR6E36QYK4M63ZJ2B7E45X4IF7W7O4Y4FMJA) context deadline exceeded ======LOGS====== Dec 26 21:18:56.839795227 (509h49m56.710878273s) [upload=447386]: Starting upload Dec 26 21:18:56.839872194 (509h49m56.710955229s) [upload=447386]: Done starting upload Dec 26 21:18:56.839941245 (509h49m56.71102428s) [upload=447386]: Uploading segments... Dec 26 21:18:56.840095990 (509h49m56.711179036s) [upload=447386]: Got next segment. Inline: false Dec 26 21:18:56.840168879 (509h49m56.711251914s) [upload=447386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 21:18:56.840175952 (509h49m56.711258998s) [upload=447386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 21:18:57.087227783 (509h49m56.958310828s) [seg_pos={0 0}, upload=447386]: Begin upload segment... Dec 26 21:18:57.087280504 (509h49m56.958363539s) [seg_pos={0 0}, upload=447386]: Done begin upload segment. Dec 26 21:18:57.087308827 (509h49m56.958391862s) [noise=true , address=145.79.227.39:4013 , piece_id=H7I7UVNY3EH52X63DNL2OWFPZZQO5LURNTSFTZDP5PDLHQBFWTVA, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.090272601 (509h49m56.961355636s) [noise=true , address=77.90.31.21:4878 , piece_id=5IZWBK5B7MQXWCICRR6E36QYK4M63ZJ2B7E45X4IF7W7O4Y4FMJA, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.091104715 (509h49m56.96218775s) [noise=true , address=45.91.116.10:4082 , piece_id=62BXZLLGCGFF2YORSGTVHWFK6GTOOSTR5BTDM7HSNUJL5P3QGTDQ, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.092393389 (509h49m56.963476424s) [noise=true , address=145.79.227.36:4517 , piece_id=PHP2PFEWABX75YRICE3XBBUYN7CBHIHSWURMFU5SU3SWZWALIVEQ, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.093966876 (509h49m56.965049911s) [noise=true , address=45.91.116.6:4973 , piece_id=HNP5WYG7JHR6MCVHF6OVFFTVAQU45INBMIH2WFIV43CNLRUEFAPA, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.094580473 (509h49m56.965663508s) [noise=true , address=145.79.227.30:4762 , piece_id=AFKDGFTGHL2MQK3MBY2PR7XAZCNDR2FFDMZ2I27HRHTO37VSEOZA, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.097678362 (509h49m56.968761407s) [noise=true , address=45.91.116.12:4226 , piece_id=6NRLVJYCQHPNEVXEEXACITPMG73ROWZKRGSIQXR7RDTSQYIMB6HQ, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.099508498 (509h49m56.970591533s) [noise=true , address=145.79.227.35:4482 , piece_id=RIZMSKLNROEITUYIJ7NBZ6OYFMG7KPMWS5SAEWLIT6GJG65YXXPQ, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.100614183 (509h49m56.971697218s) [noise=true , address=145.79.227.37:4565 , piece_id=PPXVNAGKHKNUW5EGUSHFXFYXDW6TAQX3XR2ICUE3KN4Z36SCVNAQ, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.101125126 (509h49m56.972208161s) [noise=true , address=145.79.227.46:4603 , piece_id=4A5LP6BDGBFUIAUHEFBFWYKNGXGPY7BMAR6I72B3VL3P7KOVP7PA, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.102221874 (509h49m56.973304909s) [noise=true , address=77.90.31.15:4700 , piece_id=RN57IIPN3M5M6WOZN7A7YZNNNLYRFHZIUDHRIOUWSLS3FGJVKECA, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.103709337 (509h49m56.974792372s) [noise=true , address=45.91.116.5:4297 , piece_id=IZEKS65I6MH3GUSWOJGOJOY2MON562DPQUHPCC2IW4FZBVBJEKWA, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.105036975 (509h49m56.97612001s) [noise=true , address=145.79.227.45:4230 , piece_id=LF7M7FR3AVO6VTVO5ODB7PHD4BYGKDZGER24M5VHRYENFRK7D5SQ, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.106130256 (509h49m56.977213291s) [noise=true , address=145.79.227.11:4730 , piece_id=BCMYARWGSLUDZ2MOF7DP3JTWXCIJ4MMBULBKVZDQE6OEUTWBSIPQ, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.106541630 (509h49m56.977624675s) [noise=true , address=145.79.227.49:4339 , piece_id=T444IMRG2TRES5AFEUIGULALLDQ67A4G3TYDSECAZ24E4XWS2FEA, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:57.109591237 (509h49m56.980674282s) [noise=true , address=145.79.227.52:4447 , piece_id=QOP7WPMXARDPUASGMHKPCPIXBM4ZYHZLT6YKQNXCUVIXBOVMXK6Q, seg_pos={0 0}, upload=447386]: Uploading piece... Dec 26 21:18:58.813768592 (509h49m58.684851637s) [upload=447386]: Next returned nil segment Dec 26 21:18:58.813785474 (509h49m58.684868509s) [upload=447386]: Waiting for error group managing segments... Dec 26 21:18:59.097082502 (509h49m58.968165547s) [noise=true , address=145.79.227.52:4447 , piece_id=QOP7WPMXARDPUASGMHKPCPIXBM4ZYHZLT6YKQNXCUVIXBOVMXK6Q, seg_pos={0 0}, upload=447386]: Done uploading piece. err: <nil> Dec 26 21:18:59.103230278 (509h49m58.974313313s) [noise=true , address=145.79.227.35:4482 , piece_id=RIZMSKLNROEITUYIJ7NBZ6OYFMG7KPMWS5SAEWLIT6GJG65YXXPQ, seg_pos={0 0}, upload=447386]: Done uploading piece. err: <nil> Dec 26 21:18:59.106484484 (509h49m58.977567519s) [noise=true , address=145.79.227.45:4230 , piece_id=LF7M7FR3AVO6VTVO5ODB7PHD4BYGKDZGER24M5VHRYENFRK7D5SQ, seg_pos={0 0}, upload=447386]: Done uploading piece. err: <nil> Dec 26 21:18:59.109972736 (509h49m58.981055771s) [noise=true , address=145.79.227.46:4603 , piece_id=4A5LP6BDGBFUIAUHEFBFWYKNGXGPY7BMAR6I72B3VL3P7KOVP7PA, seg_pos={0 0}, upload=447386]: Done uploading piece. err: <nil> Dec 26 21:18:59.113482520 (509h49m58.984565555s) [noise=true , address=145.79.227.37:4565 , piece_id=PPXVNAGKHKNUW5EGUSHFXFYXDW6TAQX3XR2ICUE3KN4Z36SCVNAQ, seg_pos={0 0}, upload=447386]: Done uploading piece. err: <nil> Dec 26 21:18:59.115557742 (509h49m58.986640777s) [noise=true , address=145.79.227.39:4013 , piece_id=H7I7UVNY3EH52X63DNL2OWFPZZQO5LURNTSFTZDP5PDLHQBFWTVA, seg_pos={0 0}, upload=447386]: Done uploading piece. err: <nil> Dec 26 21:18:59.120013276 (509h49m58.991096311s) [noise=true , address=145.79.227.36:4517 , piece_id=PHP2PFEWABX75YRICE3XBBUYN7CBHIHSWURMFU5SU3SWZWALIVEQ, seg_pos={0 0}, upload=447386]: Done uploading piece. err: <nil> Dec 26 21:18:59.138557387 (509h49m59.009640422s) [noise=true , address=145.79.227.49:4339 , piece_id=T444IMRG2TRES5AFEUIGULALLDQ67A4G3TYDSECAZ24E4XWS2FEA, seg_pos={0 0}, upload=447386]: Done uploading piece. err: <nil> Dec 26 21:18:59.343914273 (509h49m59.214997318s) [noise=true , address=45.91.116.5:4297 , piece_id=IZEKS65I6MH3GUSWOJGOJOY2MON562DPQUHPCC2IW4FZBVBJEKWA, seg_pos={0 0}, upload=447386]: Done uploading piece. err: <nil> Dec 26 21:18:59.459298776 (509h49m59.330381811s) [noise=true , address=45.91.116.12:4226 , piece_id=6NRLVJYCQHPNEVXEEXACITPMG73ROWZKRGSIQXR7RDTSQYIMB6HQ, seg_pos={0 0}, upload=447386]: Done uploading piece. err: <nil> Dec 26 21:18:59.465685226 (509h49m59.336768261s) [noise=true , address=45.91.116.10:4082 , piece_id=62BXZLLGCGFF2YORSGTVHWFK6GTOOSTR5BTDM7HSNUJL5P3QGTDQ, seg_pos={0 0}, upload=447386]: Done uploading piece. err: <nil> Dec 26 21:18:59.494136463 (509h49m59.365219498s) [noise=true , address=77.90.31.21:4878 , piece_id=5IZWBK5B7MQXWCICRR6E36QYK4M63ZJ2B7E45X4IF7W7O4Y4FMJA, seg_pos={0 0}, upload=447386]: Done uploading piece. err: <nil> Dec 26 21:18:59.739373698 (509h49m59.610456733s) [noise=true , address=77.90.31.15:4700 , piece_id=RN57IIPN3M5M6WOZN7A7YZNNNLYRFHZIUDHRIOUWSLS3FGJVKECA, seg_pos={0 0}, upload=447386]: Done uploading piece. err: <nil> Dec 26 21:18:59.739397604 (509h49m59.610480649s) [seg_pos={0 0}, upload=447386]: Segment reached optimal threshold of 13 pieces. Dec 26 21:18:59.739638092 (509h49m59.610721147s) [noise=true , address=45.91.116.6:4973 , piece_id=HNP5WYG7JHR6MCVHF6OVFFTVAQU45INBMIH2WFIV43CNLRUEFAPA, seg_pos={0 0}, upload=447386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 26 21:18:59.739652338 (509h49m59.610735384s) [noise=true , address=45.91.116.6:4973 , piece_id=HNP5WYG7JHR6MCVHF6OVFFTVAQU45INBMIH2WFIV43CNLRUEFAPA, seg_pos={0 0}, upload=447386]: Long tail cancellation due to optimal threshold Dec 26 21:18:59.739737371 (509h49m59.610820406s) [noise=true , address=145.79.227.30:4762 , piece_id=AFKDGFTGHL2MQK3MBY2PR7XAZCNDR2FFDMZ2I27HRHTO37VSEOZA, seg_pos={0 0}, upload=447386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 26 21:18:59.739751397 (509h49m59.610834432s) [noise=true , address=145.79.227.30:4762 , piece_id=AFKDGFTGHL2MQK3MBY2PR7XAZCNDR2FFDMZ2I27HRHTO37VSEOZA, seg_pos={0 0}, upload=447386]: Long tail cancellation due to optimal threshold Dec 26 21:18:59.739953262 (509h49m59.611036297s) [noise=true , address=145.79.227.11:4730 , piece_id=BCMYARWGSLUDZ2MOF7DP3JTWXCIJ4MMBULBKVZDQE6OEUTWBSIPQ, seg_pos={0 0}, upload=447386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: expected piece hash; context canceled; EOF Dec 26 21:18:59.739960696 (509h49m59.611043731s) [noise=true , address=145.79.227.11:4730 , piece_id=BCMYARWGSLUDZ2MOF7DP3JTWXCIJ4MMBULBKVZDQE6OEUTWBSIPQ, seg_pos={0 0}, upload=447386]: Long tail cancellation due to optimal threshold Dec 26 21:18:59.739979582 (509h49m59.611062617s) [seg_pos={0 0}, upload=447386]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 21:18:59.740071196 (509h49m59.611154231s) [upload=447386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:18:59.740076255 (509h49m59.611159291s) [upload=447386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:19:00.037299622 (509h49m59.908382657s) [upload=447386]: Done uploading segments... Dec 26 21:19:00.037322675 (509h49m59.90840572s) [upload=447386]: Upload finished. err: <nil> Duration: 30.002576803s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 15.406413115s trace: https://jaeger.dev.storj.io/trace/6d2950019117adaf FAIL (2025-12-26 21:19:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Rj4hUGWvUPuU49QDsHYhibiJhyY5KDGus1rzNeP7KusgyvTsU, Piece ID: NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A) context deadline exceeded; piecestore: (Node ID: 12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, Piece ID: N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ) context deadline exceeded; piecestore: (Node ID: 12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, Piece ID: IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ) context deadline exceeded ======LOGS====== ng piece... Dec 26 20:54:08.087336805 (509h25m7.95841984s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087649520 (509h25m7.958732565s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087988717 (509h25m7.959071752s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088323233 (509h25m7.959406268s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088877028 (509h25m7.959960063s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089247203 (509h25m7.960330238s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089543066 (509h25m7.960626101s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089914133 (509h25m7.960997168s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090327259 (509h25m7.961410294s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090645345 (509h25m7.96172838s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091004318 (509h25m7.962087353s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091338976 (509h25m7.962422021s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091732054 (509h25m7.962815099s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:09.565187143 (509h25m9.436270188s) [upload=447022]: Next returned nil segment Dec 26 20:54:09.565201321 (509h25m9.436284356s) [upload=447022]: Waiting for error group managing segments... Dec 26 20:54:10.137065972 (509h25m10.008149007s) [noise=true , address=77.90.31.10:4523 , piece_id=GQYT3Z53Y6HVTY3IBUNFCIWBEE7KVAUWEJYO7QJFRRAJCHTJMYFA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.137147197 (509h25m10.008230232s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.149325196 (509h25m10.020408242s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.159611673 (509h25m10.030694718s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.164411764 (509h25m10.035494809s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.190924278 (509h25m10.062007313s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.232931086 (509h25m10.104014131s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.241658245 (509h25m10.11274129s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.472574370 (509h25m10.343657405s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.866910566 (509h25m10.737993611s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.886583507 (509h25m10.757666542s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.127773215 (509h25m10.99885626s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.192133217 (509h25m11.063216263s) [noise=true , address=145.79.227.13:4819 , piece_id=BQDRL2QJZ75TAMUM2VSGTF4U3NEMZC4FFQLN6QV6UOAFHWB56MZA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256146219 (509h25m11.127229264s) [noise=true , address=145.79.227.40:4062 , piece_id=IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256176597 (509h25m11.127259632s) [seg_pos={0 0}, upload=447022]: Segment reached optimal threshold of 14 pieces. Dec 26 20:54:11.256428137 (509h25m11.127511192s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98): protocol: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection Dec 26 20:54:11.256439549 (509h25m11.127522584s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256458915 (509h25m11.12754198s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF): protocol: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection Dec 26 20:54:11.256490705 (509h25m11.127573741s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256698171 (509h25m11.127781216s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: rpc: dial tcp 145.79.227.30:4796: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:54:11.256706457 (509h25m11.127789502s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256917309 (509h25m11.128000364s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection Dec 26 20:54:11.256926666 (509h25m11.128009711s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256964869 (509h25m11.128047914s) [seg_pos={0 0}, upload=447022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:54:11.257064138 (509h25m11.128147173s) [upload=447022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:11.257070690 (509h25m11.128153725s) [upload=447022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:11.561962810 (509h25m11.433045875s) [upload=447022]: Done uploading segments... Dec 26 20:54:11.561987427 (509h25m11.433070472s) [upload=447022]: Upload finished. err: <nil> Duration: 15.406428876s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 32.959408281s trace: https://jaeger.dev.storj.io/trace/6d2950019117adaf FAIL (2025-12-26 21:19:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Rj4hUGWvUPuU49QDsHYhibiJhyY5KDGus1rzNeP7KusgyvTsU, Piece ID: NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A) context deadline exceeded; piecestore: (Node ID: 12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, Piece ID: N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ) context deadline exceeded; piecestore: (Node ID: 12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, Piece ID: IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ) context deadline exceeded ======LOGS====== ng piece... Dec 26 20:54:08.087336805 (509h25m7.95841984s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087649520 (509h25m7.958732565s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087988717 (509h25m7.959071752s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088323233 (509h25m7.959406268s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088877028 (509h25m7.959960063s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089247203 (509h25m7.960330238s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089543066 (509h25m7.960626101s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089914133 (509h25m7.960997168s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090327259 (509h25m7.961410294s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090645345 (509h25m7.96172838s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091004318 (509h25m7.962087353s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091338976 (509h25m7.962422021s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091732054 (509h25m7.962815099s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:09.565187143 (509h25m9.436270188s) [upload=447022]: Next returned nil segment Dec 26 20:54:09.565201321 (509h25m9.436284356s) [upload=447022]: Waiting for error group managing segments... Dec 26 20:54:10.137065972 (509h25m10.008149007s) [noise=true , address=77.90.31.10:4523 , piece_id=GQYT3Z53Y6HVTY3IBUNFCIWBEE7KVAUWEJYO7QJFRRAJCHTJMYFA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.137147197 (509h25m10.008230232s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.149325196 (509h25m10.020408242s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.159611673 (509h25m10.030694718s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.164411764 (509h25m10.035494809s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.190924278 (509h25m10.062007313s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.232931086 (509h25m10.104014131s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.241658245 (509h25m10.11274129s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.472574370 (509h25m10.343657405s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.866910566 (509h25m10.737993611s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.886583507 (509h25m10.757666542s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.127773215 (509h25m10.99885626s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.192133217 (509h25m11.063216263s) [noise=true , address=145.79.227.13:4819 , piece_id=BQDRL2QJZ75TAMUM2VSGTF4U3NEMZC4FFQLN6QV6UOAFHWB56MZA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256146219 (509h25m11.127229264s) [noise=true , address=145.79.227.40:4062 , piece_id=IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256176597 (509h25m11.127259632s) [seg_pos={0 0}, upload=447022]: Segment reached optimal threshold of 14 pieces. Dec 26 20:54:11.256428137 (509h25m11.127511192s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98): protocol: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection Dec 26 20:54:11.256439549 (509h25m11.127522584s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256458915 (509h25m11.12754198s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF): protocol: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection Dec 26 20:54:11.256490705 (509h25m11.127573741s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256698171 (509h25m11.127781216s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: rpc: dial tcp 145.79.227.30:4796: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:54:11.256706457 (509h25m11.127789502s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256917309 (509h25m11.128000364s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection Dec 26 20:54:11.256926666 (509h25m11.128009711s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256964869 (509h25m11.128047914s) [seg_pos={0 0}, upload=447022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:54:11.257064138 (509h25m11.128147173s) [upload=447022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:11.257070690 (509h25m11.128153725s) [upload=447022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:11.561962810 (509h25m11.433045875s) [upload=447022]: Done uploading segments... Dec 26 20:54:11.561987427 (509h25m11.433070472s) [upload=447022]: Upload finished. err: <nil> Duration: 32.959434333s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X...X.XX..XXX...............X....................XX.XX.X..X.] duration: 17.324705044s trace: https://jaeger.dev.storj.io/trace/357cebafa5b09ba9 FAIL (2025-12-26 21:18:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: MFKDELJDSDTNLBMWWKLPS4FQ6RYXO55L4HQWXXG4XDNEPBGUKSJQ) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: 4FR6WGMAMUP7PNBNA4RWKWNXAF4OEK2TVCBYED5CYBKNC2IFOMPQ) context deadline exceeded ======LOGS====== Dec 26 21:17:05.153761762 (509h48m5.024844807s) [upload=447357]: Starting upload Dec 26 21:17:05.153890929 (509h48m5.024973974s) [upload=447357]: Done starting upload Dec 26 21:17:05.154032779 (509h48m5.025115824s) [upload=447357]: Uploading segments... Dec 26 21:17:05.154232349 (509h48m5.025315414s) [upload=447357]: Got next segment. Inline: false Dec 26 21:17:05.154286522 (509h48m5.025369557s) [upload=447357]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 21:17:05.154291972 (509h48m5.025375007s) [upload=447357]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 21:17:05.401990875 (509h48m5.27307392s) [seg_pos={0 0}, upload=447357]: Begin upload segment... Dec 26 21:17:05.402108108 (509h48m5.273191153s) [noise=true , address=145.79.227.46:4603 , piece_id=RQ3SEIZT64UPDU4Q4NOHFPE4YMNQ2OSIZ2MWBZVQQDUB4V56VG7A, seg_pos={0 0}, upload=447357]: Uploading piece... Dec 26 21:17:05.402134137 (509h48m5.273217172s) [seg_pos={0 0}, upload=447357]: Done begin upload segment. Dec 26 21:17:05.402174043 (509h48m5.273257078s) [noise=true , address=145.79.227.41:4087 , piece_id=OZTSZ5GK5R3WQDTX47V6HJ74VSTY2EH5LQCZPQZYEK2PCP22KSYQ, seg_pos={0 0}, upload=447357]: Uploading piece... Dec 26 21:17:05.402352934 (509h48m5.273435979s) [noise=true , address=45.91.116.13:4156 , piece_id=MFKDELJDSDTNLBMWWKLPS4FQ6RYXO55L4HQWXXG4XDNEPBGUKSJQ, seg_pos={0 0}, upload=447357]: Uploading piece... Dec 26 21:17:05.403004515 (509h48m5.27408755s) [noise=true , address=145.79.227.52:4445 , piece_id=2OHOIT7NOEAVG7DSHSDEZ47JMYLEGPFSRUPCQAU4C2HW4UGZYXVA, seg_pos={0 0}, upload=447357]: Uploading piece... Dec 26 21:17:05.403344132 (509h48m5.274427167s) [noise=true , address=145.79.227.48:4303 , piece_id=YFZURTO6NTETTHHWMAZYKHTEU7IVSU6AVBYPWBLQRA6THJNEJCGQ, seg_pos={0 0}, upload=447357]: Uploading piece... Dec 26 21:17:05.403632530 (509h48m5.274715566s) [noise=true , address=45.91.116.5:4298 , piece_id=7M5D2TP64GPPANNRH5K56HNFBLILB2KRRTP4KE75GRQXR37U4DEA, seg_pos={0 0}, upload=447357]: Uploading piece... Dec 26 21:17:05.403911181 (509h48m5.274994216s) [noise=true , address=145.79.227.49:4339 , piece_id=GAHZVSEGBGZROHH465BTQO6D3KATQRJVMX4IMX3KSNF3J2YZQ6CA, seg_pos={0 0}, upload=447357]: Uploading piece... Dec 26 21:17:05.404241009 (509h48m5.275324044s) [noise=true , address=45.91.116.6:4973 , piece_id=IBQPL5PBNARFJD56B6XLLLIMGHP5H43CWA4JP2XGSYCUK237YPQA, seg_pos={0 0}, upload=447357]: Uploading piece... Dec 26 21:17:05.404711986 (509h48m5.275795031s) [noise=true , address=77.90.31.19:4842 , piece_id=4FR6WGMAMUP7PNBNA4RWKWNXAF4OEK2TVCBYED5CYBKNC2IFOMPQ, seg_pos={0 0}, upload=447357]: Uploading piece... Dec 26 21:17:05.405175749 (509h48m5.276258784s) [noise=true , address=145.79.227.42:4122 , piece_id=6QNXLZIDE64HUJP3DTGSNKECT6FHLSYV2T35FTYBSQBM6EAL3UOQ, seg_pos={0 0}, upload=447357]: Uploading piece... Dec 26 21:17:05.406253140 (509h48m5.277336175s) [noise=true , address=45.91.116.4:4937 , piece_id=PKTWTIDNUQQOVLRN6SG2P32IU7J4CW6RVWA4D7GPBVAZPV2GG6BA, seg_pos={0 0}, upload=447357]: Uploading piece... Dec 26 21:17:05.407664067 (509h48m5.278747112s) [noise=true , address=77.90.31.10:4519 , piece_id=7UJDJCYUMS653CUUDA3VY4OOMSXBCCH5KZZKNMT4BYYQBU34IJRQ, seg_pos={0 0}, upload=447357]: Uploading piece... Dec 26 21:17:05.409312587 (509h48m5.280395632s) [noise=true , address=145.79.227.50:4375 , piece_id=DXBEXYLEN65M7T4FXNPWZGXLVUEZUGVHUBSLQTJQRTSLVWNC757A, seg_pos={0 0}, upload=447357]: Uploading piece... Dec 26 21:17:05.411007794 (509h48m5.282090829s) [noise=true , address=45.91.116.7:4011 , piece_id=JZYIIAWUKFUFI7UNTQVJQYNOXVCROYYD664YVRCZKCZSRQPUV5OA, seg_pos={0 0}, upload=447357]: Uploading piece... Dec 26 21:17:05.412591150 (509h48m5.283674195s) [noise=true , address=145.79.227.39:4015 , piece_id=6JUXI2KWZXUL546LDDBK4WJEDO2AMYDRUKKFJJUUFUWMZDSRJZHQ, seg_pos={0 0}, upload=447357]: Uploading piece... Dec 26 21:17:05.412787213 (509h48m5.283870258s) [noise=true , address=145.79.227.15:4876 , piece_id=QMNQHYQCBW7EHAMOIJGTWBQHKY5N2FFYQYTKEWXVMBGXEUFPV34A, seg_pos={0 0}, upload=447357]: Uploading piece... Dec 26 21:17:07.074532640 (509h48m6.945615675s) [upload=447357]: Next returned nil segment Dec 26 21:17:07.074797154 (509h48m6.945880189s) [upload=447357]: Waiting for error group managing segments... Dec 26 21:17:07.411789217 (509h48m7.282872252s) [noise=true , address=145.79.227.41:4087 , piece_id=OZTSZ5GK5R3WQDTX47V6HJ74VSTY2EH5LQCZPQZYEK2PCP22KSYQ, seg_pos={0 0}, upload=447357]: Done uploading piece. err: <nil> Dec 26 21:17:07.412024525 (509h48m7.28310758s) [noise=true , address=145.79.227.46:4603 , piece_id=RQ3SEIZT64UPDU4Q4NOHFPE4YMNQ2OSIZ2MWBZVQQDUB4V56VG7A, seg_pos={0 0}, upload=447357]: Done uploading piece. err: <nil> Dec 26 21:17:07.412269491 (509h48m7.283352536s) [noise=true , address=145.79.227.15:4876 , piece_id=QMNQHYQCBW7EHAMOIJGTWBQHKY5N2FFYQYTKEWXVMBGXEUFPV34A, seg_pos={0 0}, upload=447357]: Done uploading piece. err: <nil> Dec 26 21:17:07.414395620 (509h48m7.285478665s) [noise=true , address=145.79.227.42:4122 , piece_id=6QNXLZIDE64HUJP3DTGSNKECT6FHLSYV2T35FTYBSQBM6EAL3UOQ, seg_pos={0 0}, upload=447357]: Done uploading piece. err: <nil> Dec 26 21:17:07.418905266 (509h48m7.289988311s) [noise=true , address=145.79.227.50:4375 , piece_id=DXBEXYLEN65M7T4FXNPWZGXLVUEZUGVHUBSLQTJQRTSLVWNC757A, seg_pos={0 0}, upload=447357]: Done uploading piece. err: <nil> Dec 26 21:17:07.423287040 (509h48m7.294370075s) [noise=true , address=145.79.227.48:4303 , piece_id=YFZURTO6NTETTHHWMAZYKHTEU7IVSU6AVBYPWBLQRA6THJNEJCGQ, seg_pos={0 0}, upload=447357]: Done uploading piece. err: <nil> Dec 26 21:17:07.441433414 (509h48m7.312516459s) [noise=true , address=145.79.227.49:4339 , piece_id=GAHZVSEGBGZROHH465BTQO6D3KATQRJVMX4IMX3KSNF3J2YZQ6CA, seg_pos={0 0}, upload=447357]: Done uploading piece. err: <nil> Dec 26 21:17:07.442218309 (509h48m7.313301354s) [noise=true , address=145.79.227.52:4445 , piece_id=2OHOIT7NOEAVG7DSHSDEZ47JMYLEGPFSRUPCQAU4C2HW4UGZYXVA, seg_pos={0 0}, upload=447357]: Done uploading piece. err: <nil> Dec 26 21:17:07.446872772 (509h48m7.317955807s) [noise=true , address=145.79.227.39:4015 , piece_id=6JUXI2KWZXUL546LDDBK4WJEDO2AMYDRUKKFJJUUFUWMZDSRJZHQ, seg_pos={0 0}, upload=447357]: Done uploading piece. err: <nil> Dec 26 21:17:07.595975941 (509h48m7.467059006s) [noise=true , address=77.90.31.10:4519 , piece_id=7UJDJCYUMS653CUUDA3VY4OOMSXBCCH5KZZKNMT4BYYQBU34IJRQ, seg_pos={0 0}, upload=447357]: Done uploading piece. err: <nil> Dec 26 21:17:07.619956945 (509h48m7.49103998s) [noise=true , address=45.91.116.5:4298 , piece_id=7M5D2TP64GPPANNRH5K56HNFBLILB2KRRTP4KE75GRQXR37U4DEA, seg_pos={0 0}, upload=447357]: Done uploading piece. err: <nil> Dec 26 21:17:07.639974191 (509h48m7.511057236s) [noise=true , address=45.91.116.13:4156 , piece_id=MFKDELJDSDTNLBMWWKLPS4FQ6RYXO55L4HQWXXG4XDNEPBGUKSJQ, seg_pos={0 0}, upload=447357]: Done uploading piece. err: <nil> Dec 26 21:17:07.676015600 (509h48m7.547098635s) [noise=true , address=77.90.31.19:4842 , piece_id=4FR6WGMAMUP7PNBNA4RWKWNXAF4OEK2TVCBYED5CYBKNC2IFOMPQ, seg_pos={0 0}, upload=447357]: Done uploading piece. err: <nil> Dec 26 21:17:07.676034566 (509h48m7.547117612s) [seg_pos={0 0}, upload=447357]: Segment reached optimal threshold of 13 pieces. Dec 26 21:17:07.676215731 (509h48m7.547298766s) [noise=true , address=45.91.116.7:4011 , piece_id=JZYIIAWUKFUFI7UNTQVJQYNOXVCROYYD664YVRCZKCZSRQPUV5OA, seg_pos={0 0}, upload=447357]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:46556->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:46556->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46556->45.91.116.7:4011: use of closed network connection Dec 26 21:17:07.676221502 (509h48m7.547304557s) [noise=true , address=45.91.116.7:4011 , piece_id=JZYIIAWUKFUFI7UNTQVJQYNOXVCROYYD664YVRCZKCZSRQPUV5OA, seg_pos={0 0}, upload=447357]: Long tail cancellation due to optimal threshold Dec 26 21:17:07.676322214 (509h48m7.547405249s) [noise=true , address=45.91.116.4:4937 , piece_id=PKTWTIDNUQQOVLRN6SG2P32IU7J4CW6RVWA4D7GPBVAZPV2GG6BA, seg_pos={0 0}, upload=447357]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 21:17:07.676337643 (509h48m7.547420678s) [noise=true , address=45.91.116.4:4937 , piece_id=PKTWTIDNUQQOVLRN6SG2P32IU7J4CW6RVWA4D7GPBVAZPV2GG6BA, seg_pos={0 0}, upload=447357]: Long tail cancellation due to optimal threshold Dec 26 21:17:07.676380485 (509h48m7.54746352s) [noise=true , address=45.91.116.6:4973 , piece_id=IBQPL5PBNARFJD56B6XLLLIMGHP5H43CWA4JP2XGSYCUK237YPQA, seg_pos={0 0}, upload=447357]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 26 21:17:07.676386887 (509h48m7.547469922s) [noise=true , address=45.91.116.6:4973 , piece_id=IBQPL5PBNARFJD56B6XLLLIMGHP5H43CWA4JP2XGSYCUK237YPQA, seg_pos={0 0}, upload=447357]: Long tail cancellation due to optimal threshold Dec 26 21:17:07.676459135 (509h48m7.54754217s) [seg_pos={0 0}, upload=447357]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 21:17:07.676557693 (509h48m7.547640728s) [upload=447357]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:17:07.676562622 (509h48m7.547645667s) [upload=447357]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:17:07.966761171 (509h48m7.837844217s) [upload=447357]: Done uploading segments... Dec 26 21:17:07.966792121 (509h48m7.837875156s) [upload=447357]: Upload finished. err: <nil> Duration: 17.324729882s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 15.222134419s trace: https://jaeger.dev.storj.io/trace/6d2950019117adaf FAIL (2025-12-26 21:18:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Rj4hUGWvUPuU49QDsHYhibiJhyY5KDGus1rzNeP7KusgyvTsU, Piece ID: NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A) context deadline exceeded; piecestore: (Node ID: 12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, Piece ID: N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ) context deadline exceeded; piecestore: (Node ID: 12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, Piece ID: IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ) context deadline exceeded ======LOGS====== ng piece... Dec 26 20:54:08.087336805 (509h25m7.95841984s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087649520 (509h25m7.958732565s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087988717 (509h25m7.959071752s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088323233 (509h25m7.959406268s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088877028 (509h25m7.959960063s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089247203 (509h25m7.960330238s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089543066 (509h25m7.960626101s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089914133 (509h25m7.960997168s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090327259 (509h25m7.961410294s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090645345 (509h25m7.96172838s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091004318 (509h25m7.962087353s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091338976 (509h25m7.962422021s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091732054 (509h25m7.962815099s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:09.565187143 (509h25m9.436270188s) [upload=447022]: Next returned nil segment Dec 26 20:54:09.565201321 (509h25m9.436284356s) [upload=447022]: Waiting for error group managing segments... Dec 26 20:54:10.137065972 (509h25m10.008149007s) [noise=true , address=77.90.31.10:4523 , piece_id=GQYT3Z53Y6HVTY3IBUNFCIWBEE7KVAUWEJYO7QJFRRAJCHTJMYFA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.137147197 (509h25m10.008230232s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.149325196 (509h25m10.020408242s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.159611673 (509h25m10.030694718s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.164411764 (509h25m10.035494809s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.190924278 (509h25m10.062007313s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.232931086 (509h25m10.104014131s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.241658245 (509h25m10.11274129s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.472574370 (509h25m10.343657405s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.866910566 (509h25m10.737993611s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.886583507 (509h25m10.757666542s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.127773215 (509h25m10.99885626s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.192133217 (509h25m11.063216263s) [noise=true , address=145.79.227.13:4819 , piece_id=BQDRL2QJZ75TAMUM2VSGTF4U3NEMZC4FFQLN6QV6UOAFHWB56MZA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256146219 (509h25m11.127229264s) [noise=true , address=145.79.227.40:4062 , piece_id=IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256176597 (509h25m11.127259632s) [seg_pos={0 0}, upload=447022]: Segment reached optimal threshold of 14 pieces. Dec 26 20:54:11.256428137 (509h25m11.127511192s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98): protocol: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection Dec 26 20:54:11.256439549 (509h25m11.127522584s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256458915 (509h25m11.12754198s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF): protocol: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection Dec 26 20:54:11.256490705 (509h25m11.127573741s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256698171 (509h25m11.127781216s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: rpc: dial tcp 145.79.227.30:4796: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:54:11.256706457 (509h25m11.127789502s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256917309 (509h25m11.128000364s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection Dec 26 20:54:11.256926666 (509h25m11.128009711s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256964869 (509h25m11.128047914s) [seg_pos={0 0}, upload=447022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:54:11.257064138 (509h25m11.128147173s) [upload=447022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:11.257070690 (509h25m11.128153725s) [upload=447022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:11.561962810 (509h25m11.433045875s) [upload=447022]: Done uploading segments... Dec 26 20:54:11.561987427 (509h25m11.433070472s) [upload=447022]: Upload finished. err: <nil> Duration: 15.222162463s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [XX...X.XX..XXX...............X....................XX.XX.X..X] duration: 30.001868057s trace: https://jaeger.dev.storj.io/trace/357cebafa5b09ba9 FAIL (2025-12-26 21:17:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: MFKDELJDSDTNLBMWWKLPS4FQ6RYXO55L4HQWXXG4XDNEPBGUKSJQ) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: 4FR6WGMAMUP7PNBNA4RWKWNXAF4OEK2TVCBYED5CYBKNC2IFOMPQ) context deadline exceeded ======LOGS====== Dec 26 21:17:05.153761762 (509h48m5.024844807s) [upload=447357]: Starting upload Dec 26 21:17:05.153890929 (509h48m5.024973974s) [upload=447357]: Done starting upload Dec 26 21:17:05.154032779 (509h48m5.025115824s) [upload=447357]: Uploading segments... Dec 26 21:17:05.154232349 (509h48m5.025315414s) [upload=447357]: Got next segment. Inline: false Dec 26 21:17:05.154286522 (509h48m5.025369557s) [upload=447357]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 21:17:05.154291972 (509h48m5.025375007s) [upload=447357]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 21:17:05.401990875 (509h48m5.27307392s) [seg_pos={0 0}, upload=447357]: Begin upload segment... Dec 26 21:17:05.402108108 (509h48m5.273191153s) [noise=true , address=145.79.227.46:4603 , piece_id=RQ3SEIZT64UPDU4Q4NOHFPE4YMNQ2OSIZ2MWBZVQQDUB4V56VG7A, seg_pos={0 0}, upload=447357]: Uploading piece... Dec 26 21:17:05.402134137 (509h48m5.273217172s) [seg_pos={0 0}, upload=447357]: Done begin upload segment. Dec 26 21:17:05.402174043 (509h48m5.273257078s) [noise=true , address=145.79.227.41:4087 , piece_id=OZTSZ5GK5R3WQDTX47V6HJ74VSTY2EH5LQCZPQZYEK2PCP22KSYQ, seg_pos={0 0}, upload=447357]: Uploading piece... Dec 26 21:17:05.402352934 (509h48m5.273435979s) [noise=true , address=45.91.116.13:4156 , piece_id=MFKDELJDSDTNLBMWWKLPS4FQ6RYXO55L4HQWXXG4XDNEPBGUKSJQ, seg_pos={0 0}, upload=447357]: Uploading piece... Dec 26 21:17:05.403004515 (509h48m5.27408755s) [noise=true , address=145.79.227.52:4445 , piece_id=2OHOIT7NOEAVG7DSHSDEZ47JMYLEGPFSRUPCQAU4C2HW4UGZYXVA, seg_pos={0 0}, upload=447357]: Uploading piece... Dec 26 21:17:05.403344132 (509h48m5.274427167s) [noise=true , address=145.79.227.48:4303 , piece_id=YFZURTO6NTETTHHWMAZYKHTEU7IVSU6AVBYPWBLQRA6THJNEJCGQ, seg_pos={0 0}, upload=447357]: Uploading piece... Dec 26 21:17:05.403632530 (509h48m5.274715566s) [noise=true , address=45.91.116.5:4298 , piece_id=7M5D2TP64GPPANNRH5K56HNFBLILB2KRRTP4KE75GRQXR37U4DEA, seg_pos={0 0}, upload=447357]: Uploading piece... Dec 26 21:17:05.403911181 (509h48m5.274994216s) [noise=true , address=145.79.227.49:4339 , piece_id=GAHZVSEGBGZROHH465BTQO6D3KATQRJVMX4IMX3KSNF3J2YZQ6CA, seg_pos={0 0}, upload=447357]: Uploading piece... Dec 26 21:17:05.404241009 (509h48m5.275324044s) [noise=true , address=45.91.116.6:4973 , piece_id=IBQPL5PBNARFJD56B6XLLLIMGHP5H43CWA4JP2XGSYCUK237YPQA, seg_pos={0 0}, upload=447357]: Uploading piece... Dec 26 21:17:05.404711986 (509h48m5.275795031s) [noise=true , address=77.90.31.19:4842 , piece_id=4FR6WGMAMUP7PNBNA4RWKWNXAF4OEK2TVCBYED5CYBKNC2IFOMPQ, seg_pos={0 0}, upload=447357]: Uploading piece... Dec 26 21:17:05.405175749 (509h48m5.276258784s) [noise=true , address=145.79.227.42:4122 , piece_id=6QNXLZIDE64HUJP3DTGSNKECT6FHLSYV2T35FTYBSQBM6EAL3UOQ, seg_pos={0 0}, upload=447357]: Uploading piece... Dec 26 21:17:05.406253140 (509h48m5.277336175s) [noise=true , address=45.91.116.4:4937 , piece_id=PKTWTIDNUQQOVLRN6SG2P32IU7J4CW6RVWA4D7GPBVAZPV2GG6BA, seg_pos={0 0}, upload=447357]: Uploading piece... Dec 26 21:17:05.407664067 (509h48m5.278747112s) [noise=true , address=77.90.31.10:4519 , piece_id=7UJDJCYUMS653CUUDA3VY4OOMSXBCCH5KZZKNMT4BYYQBU34IJRQ, seg_pos={0 0}, upload=447357]: Uploading piece... Dec 26 21:17:05.409312587 (509h48m5.280395632s) [noise=true , address=145.79.227.50:4375 , piece_id=DXBEXYLEN65M7T4FXNPWZGXLVUEZUGVHUBSLQTJQRTSLVWNC757A, seg_pos={0 0}, upload=447357]: Uploading piece... Dec 26 21:17:05.411007794 (509h48m5.282090829s) [noise=true , address=45.91.116.7:4011 , piece_id=JZYIIAWUKFUFI7UNTQVJQYNOXVCROYYD664YVRCZKCZSRQPUV5OA, seg_pos={0 0}, upload=447357]: Uploading piece... Dec 26 21:17:05.412591150 (509h48m5.283674195s) [noise=true , address=145.79.227.39:4015 , piece_id=6JUXI2KWZXUL546LDDBK4WJEDO2AMYDRUKKFJJUUFUWMZDSRJZHQ, seg_pos={0 0}, upload=447357]: Uploading piece... Dec 26 21:17:05.412787213 (509h48m5.283870258s) [noise=true , address=145.79.227.15:4876 , piece_id=QMNQHYQCBW7EHAMOIJGTWBQHKY5N2FFYQYTKEWXVMBGXEUFPV34A, seg_pos={0 0}, upload=447357]: Uploading piece... Dec 26 21:17:07.074532640 (509h48m6.945615675s) [upload=447357]: Next returned nil segment Dec 26 21:17:07.074797154 (509h48m6.945880189s) [upload=447357]: Waiting for error group managing segments... Dec 26 21:17:07.411789217 (509h48m7.282872252s) [noise=true , address=145.79.227.41:4087 , piece_id=OZTSZ5GK5R3WQDTX47V6HJ74VSTY2EH5LQCZPQZYEK2PCP22KSYQ, seg_pos={0 0}, upload=447357]: Done uploading piece. err: <nil> Dec 26 21:17:07.412024525 (509h48m7.28310758s) [noise=true , address=145.79.227.46:4603 , piece_id=RQ3SEIZT64UPDU4Q4NOHFPE4YMNQ2OSIZ2MWBZVQQDUB4V56VG7A, seg_pos={0 0}, upload=447357]: Done uploading piece. err: <nil> Dec 26 21:17:07.412269491 (509h48m7.283352536s) [noise=true , address=145.79.227.15:4876 , piece_id=QMNQHYQCBW7EHAMOIJGTWBQHKY5N2FFYQYTKEWXVMBGXEUFPV34A, seg_pos={0 0}, upload=447357]: Done uploading piece. err: <nil> Dec 26 21:17:07.414395620 (509h48m7.285478665s) [noise=true , address=145.79.227.42:4122 , piece_id=6QNXLZIDE64HUJP3DTGSNKECT6FHLSYV2T35FTYBSQBM6EAL3UOQ, seg_pos={0 0}, upload=447357]: Done uploading piece. err: <nil> Dec 26 21:17:07.418905266 (509h48m7.289988311s) [noise=true , address=145.79.227.50:4375 , piece_id=DXBEXYLEN65M7T4FXNPWZGXLVUEZUGVHUBSLQTJQRTSLVWNC757A, seg_pos={0 0}, upload=447357]: Done uploading piece. err: <nil> Dec 26 21:17:07.423287040 (509h48m7.294370075s) [noise=true , address=145.79.227.48:4303 , piece_id=YFZURTO6NTETTHHWMAZYKHTEU7IVSU6AVBYPWBLQRA6THJNEJCGQ, seg_pos={0 0}, upload=447357]: Done uploading piece. err: <nil> Dec 26 21:17:07.441433414 (509h48m7.312516459s) [noise=true , address=145.79.227.49:4339 , piece_id=GAHZVSEGBGZROHH465BTQO6D3KATQRJVMX4IMX3KSNF3J2YZQ6CA, seg_pos={0 0}, upload=447357]: Done uploading piece. err: <nil> Dec 26 21:17:07.442218309 (509h48m7.313301354s) [noise=true , address=145.79.227.52:4445 , piece_id=2OHOIT7NOEAVG7DSHSDEZ47JMYLEGPFSRUPCQAU4C2HW4UGZYXVA, seg_pos={0 0}, upload=447357]: Done uploading piece. err: <nil> Dec 26 21:17:07.446872772 (509h48m7.317955807s) [noise=true , address=145.79.227.39:4015 , piece_id=6JUXI2KWZXUL546LDDBK4WJEDO2AMYDRUKKFJJUUFUWMZDSRJZHQ, seg_pos={0 0}, upload=447357]: Done uploading piece. err: <nil> Dec 26 21:17:07.595975941 (509h48m7.467059006s) [noise=true , address=77.90.31.10:4519 , piece_id=7UJDJCYUMS653CUUDA3VY4OOMSXBCCH5KZZKNMT4BYYQBU34IJRQ, seg_pos={0 0}, upload=447357]: Done uploading piece. err: <nil> Dec 26 21:17:07.619956945 (509h48m7.49103998s) [noise=true , address=45.91.116.5:4298 , piece_id=7M5D2TP64GPPANNRH5K56HNFBLILB2KRRTP4KE75GRQXR37U4DEA, seg_pos={0 0}, upload=447357]: Done uploading piece. err: <nil> Dec 26 21:17:07.639974191 (509h48m7.511057236s) [noise=true , address=45.91.116.13:4156 , piece_id=MFKDELJDSDTNLBMWWKLPS4FQ6RYXO55L4HQWXXG4XDNEPBGUKSJQ, seg_pos={0 0}, upload=447357]: Done uploading piece. err: <nil> Dec 26 21:17:07.676015600 (509h48m7.547098635s) [noise=true , address=77.90.31.19:4842 , piece_id=4FR6WGMAMUP7PNBNA4RWKWNXAF4OEK2TVCBYED5CYBKNC2IFOMPQ, seg_pos={0 0}, upload=447357]: Done uploading piece. err: <nil> Dec 26 21:17:07.676034566 (509h48m7.547117612s) [seg_pos={0 0}, upload=447357]: Segment reached optimal threshold of 13 pieces. Dec 26 21:17:07.676215731 (509h48m7.547298766s) [noise=true , address=45.91.116.7:4011 , piece_id=JZYIIAWUKFUFI7UNTQVJQYNOXVCROYYD664YVRCZKCZSRQPUV5OA, seg_pos={0 0}, upload=447357]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:46556->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:46556->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46556->45.91.116.7:4011: use of closed network connection Dec 26 21:17:07.676221502 (509h48m7.547304557s) [noise=true , address=45.91.116.7:4011 , piece_id=JZYIIAWUKFUFI7UNTQVJQYNOXVCROYYD664YVRCZKCZSRQPUV5OA, seg_pos={0 0}, upload=447357]: Long tail cancellation due to optimal threshold Dec 26 21:17:07.676322214 (509h48m7.547405249s) [noise=true , address=45.91.116.4:4937 , piece_id=PKTWTIDNUQQOVLRN6SG2P32IU7J4CW6RVWA4D7GPBVAZPV2GG6BA, seg_pos={0 0}, upload=447357]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 21:17:07.676337643 (509h48m7.547420678s) [noise=true , address=45.91.116.4:4937 , piece_id=PKTWTIDNUQQOVLRN6SG2P32IU7J4CW6RVWA4D7GPBVAZPV2GG6BA, seg_pos={0 0}, upload=447357]: Long tail cancellation due to optimal threshold Dec 26 21:17:07.676380485 (509h48m7.54746352s) [noise=true , address=45.91.116.6:4973 , piece_id=IBQPL5PBNARFJD56B6XLLLIMGHP5H43CWA4JP2XGSYCUK237YPQA, seg_pos={0 0}, upload=447357]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 26 21:17:07.676386887 (509h48m7.547469922s) [noise=true , address=45.91.116.6:4973 , piece_id=IBQPL5PBNARFJD56B6XLLLIMGHP5H43CWA4JP2XGSYCUK237YPQA, seg_pos={0 0}, upload=447357]: Long tail cancellation due to optimal threshold Dec 26 21:17:07.676459135 (509h48m7.54754217s) [seg_pos={0 0}, upload=447357]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 21:17:07.676557693 (509h48m7.547640728s) [upload=447357]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:17:07.676562622 (509h48m7.547645667s) [upload=447357]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:17:07.966761171 (509h48m7.837844217s) [upload=447357]: Done uploading segments... Dec 26 21:17:07.966792121 (509h48m7.837875156s) [upload=447357]: Upload finished. err: <nil> Duration: 30.001882824s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 17.016960013s trace: https://jaeger.dev.storj.io/trace/6d2950019117adaf FAIL (2025-12-26 21:17:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Rj4hUGWvUPuU49QDsHYhibiJhyY5KDGus1rzNeP7KusgyvTsU, Piece ID: NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A) context deadline exceeded; piecestore: (Node ID: 12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, Piece ID: N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ) context deadline exceeded; piecestore: (Node ID: 12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, Piece ID: IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ) context deadline exceeded ======LOGS====== ng piece... Dec 26 20:54:08.087336805 (509h25m7.95841984s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087649520 (509h25m7.958732565s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087988717 (509h25m7.959071752s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088323233 (509h25m7.959406268s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088877028 (509h25m7.959960063s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089247203 (509h25m7.960330238s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089543066 (509h25m7.960626101s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089914133 (509h25m7.960997168s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090327259 (509h25m7.961410294s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090645345 (509h25m7.96172838s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091004318 (509h25m7.962087353s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091338976 (509h25m7.962422021s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091732054 (509h25m7.962815099s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:09.565187143 (509h25m9.436270188s) [upload=447022]: Next returned nil segment Dec 26 20:54:09.565201321 (509h25m9.436284356s) [upload=447022]: Waiting for error group managing segments... Dec 26 20:54:10.137065972 (509h25m10.008149007s) [noise=true , address=77.90.31.10:4523 , piece_id=GQYT3Z53Y6HVTY3IBUNFCIWBEE7KVAUWEJYO7QJFRRAJCHTJMYFA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.137147197 (509h25m10.008230232s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.149325196 (509h25m10.020408242s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.159611673 (509h25m10.030694718s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.164411764 (509h25m10.035494809s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.190924278 (509h25m10.062007313s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.232931086 (509h25m10.104014131s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.241658245 (509h25m10.11274129s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.472574370 (509h25m10.343657405s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.866910566 (509h25m10.737993611s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.886583507 (509h25m10.757666542s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.127773215 (509h25m10.99885626s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.192133217 (509h25m11.063216263s) [noise=true , address=145.79.227.13:4819 , piece_id=BQDRL2QJZ75TAMUM2VSGTF4U3NEMZC4FFQLN6QV6UOAFHWB56MZA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256146219 (509h25m11.127229264s) [noise=true , address=145.79.227.40:4062 , piece_id=IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256176597 (509h25m11.127259632s) [seg_pos={0 0}, upload=447022]: Segment reached optimal threshold of 14 pieces. Dec 26 20:54:11.256428137 (509h25m11.127511192s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98): protocol: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection Dec 26 20:54:11.256439549 (509h25m11.127522584s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256458915 (509h25m11.12754198s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF): protocol: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection Dec 26 20:54:11.256490705 (509h25m11.127573741s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256698171 (509h25m11.127781216s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: rpc: dial tcp 145.79.227.30:4796: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:54:11.256706457 (509h25m11.127789502s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256917309 (509h25m11.128000364s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection Dec 26 20:54:11.256926666 (509h25m11.128009711s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256964869 (509h25m11.128047914s) [seg_pos={0 0}, upload=447022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:54:11.257064138 (509h25m11.128147173s) [upload=447022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:11.257070690 (509h25m11.128153725s) [upload=447022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:11.561962810 (509h25m11.433045875s) [upload=447022]: Done uploading segments... Dec 26 20:54:11.561987427 (509h25m11.433070472s) [upload=447022]: Upload finished. err: <nil> Duration: 17.017015628s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 26.881623477s trace: https://jaeger.dev.storj.io/trace/6d2950019117adaf FAIL (2025-12-26 21:16:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Rj4hUGWvUPuU49QDsHYhibiJhyY5KDGus1rzNeP7KusgyvTsU, Piece ID: NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A) context deadline exceeded; piecestore: (Node ID: 12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, Piece ID: N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ) context deadline exceeded; piecestore: (Node ID: 12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, Piece ID: IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ) context deadline exceeded ======LOGS====== ng piece... Dec 26 20:54:08.087336805 (509h25m7.95841984s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087649520 (509h25m7.958732565s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087988717 (509h25m7.959071752s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088323233 (509h25m7.959406268s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088877028 (509h25m7.959960063s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089247203 (509h25m7.960330238s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089543066 (509h25m7.960626101s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089914133 (509h25m7.960997168s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090327259 (509h25m7.961410294s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090645345 (509h25m7.96172838s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091004318 (509h25m7.962087353s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091338976 (509h25m7.962422021s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091732054 (509h25m7.962815099s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:09.565187143 (509h25m9.436270188s) [upload=447022]: Next returned nil segment Dec 26 20:54:09.565201321 (509h25m9.436284356s) [upload=447022]: Waiting for error group managing segments... Dec 26 20:54:10.137065972 (509h25m10.008149007s) [noise=true , address=77.90.31.10:4523 , piece_id=GQYT3Z53Y6HVTY3IBUNFCIWBEE7KVAUWEJYO7QJFRRAJCHTJMYFA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.137147197 (509h25m10.008230232s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.149325196 (509h25m10.020408242s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.159611673 (509h25m10.030694718s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.164411764 (509h25m10.035494809s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.190924278 (509h25m10.062007313s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.232931086 (509h25m10.104014131s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.241658245 (509h25m10.11274129s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.472574370 (509h25m10.343657405s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.866910566 (509h25m10.737993611s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.886583507 (509h25m10.757666542s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.127773215 (509h25m10.99885626s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.192133217 (509h25m11.063216263s) [noise=true , address=145.79.227.13:4819 , piece_id=BQDRL2QJZ75TAMUM2VSGTF4U3NEMZC4FFQLN6QV6UOAFHWB56MZA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256146219 (509h25m11.127229264s) [noise=true , address=145.79.227.40:4062 , piece_id=IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256176597 (509h25m11.127259632s) [seg_pos={0 0}, upload=447022]: Segment reached optimal threshold of 14 pieces. Dec 26 20:54:11.256428137 (509h25m11.127511192s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98): protocol: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection Dec 26 20:54:11.256439549 (509h25m11.127522584s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256458915 (509h25m11.12754198s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF): protocol: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection Dec 26 20:54:11.256490705 (509h25m11.127573741s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256698171 (509h25m11.127781216s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: rpc: dial tcp 145.79.227.30:4796: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:54:11.256706457 (509h25m11.127789502s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256917309 (509h25m11.128000364s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection Dec 26 20:54:11.256926666 (509h25m11.128009711s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256964869 (509h25m11.128047914s) [seg_pos={0 0}, upload=447022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:54:11.257064138 (509h25m11.128147173s) [upload=447022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:11.257070690 (509h25m11.128153725s) [upload=447022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:11.561962810 (509h25m11.433045875s) [upload=447022]: Done uploading segments... Dec 26 20:54:11.561987427 (509h25m11.433070472s) [upload=447022]: Upload finished. err: <nil> Duration: 26.881649155s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [XXX...X.XX..XXX...............X....................XX.XX.X..] duration: 14.547307898s trace: https://jaeger.dev.storj.io/trace/e1dc9001c283d0f FAIL (2025-12-26 21:16:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: GHEVUBPZYS3SMSUM4PNR36IEXPXZCLFGWTFUWM6BQ2QKVMGLMPRA) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: 6BKBICK2ESYZKHZRJ3RMTT7HHOXJ2IIPWIBWGEZ27VQA2DVS7T7A) context deadline exceeded ======LOGS====== Dec 26 21:14:21.701071402 (509h45m21.572154437s) [upload=447317]: Starting upload Dec 26 21:14:21.701221497 (509h45m21.572304542s) [upload=447317]: Done starting upload Dec 26 21:14:21.701347367 (509h45m21.572430432s) [upload=447317]: Uploading segments... Dec 26 21:14:21.701521769 (509h45m21.572604805s) [upload=447317]: Got next segment. Inline: false Dec 26 21:14:21.701657448 (509h45m21.572740493s) [upload=447317]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 21:14:21.701665814 (509h45m21.572748859s) [upload=447317]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 21:14:21.945615572 (509h45m21.816698637s) [seg_pos={0 0}, upload=447317]: Begin upload segment... Dec 26 21:14:21.945684473 (509h45m21.816767518s) [seg_pos={0 0}, upload=447317]: Done begin upload segment. Dec 26 21:14:21.945742925 (509h45m21.81682596s) [noise=true , address=145.79.227.15:4876 , piece_id=JPOYCRWQ3YEC22JVO4LK6EEEOIXNO3NMDUPJJFQVQA4KI6AART7A, seg_pos={0 0}, upload=447317]: Uploading piece... Dec 26 21:14:21.945755729 (509h45m21.816838774s) [noise=true , address=45.91.116.6:4974 , piece_id=PKHOVEREEYILOICNAOVUXUPXU2HWPMU2MV5HEMXLXK4BGSEYRW4A, seg_pos={0 0}, upload=447317]: Uploading piece... Dec 26 21:14:21.945833889 (509h45m21.816916924s) [noise=true , address=45.91.116.11:4118 , piece_id=GHEVUBPZYS3SMSUM4PNR36IEXPXZCLFGWTFUWM6BQ2QKVMGLMPRA, seg_pos={0 0}, upload=447317]: Uploading piece... Dec 26 21:14:21.946386711 (509h45m21.817469756s) [noise=true , address=145.79.227.51:4409 , piece_id=JAWYLPDQREAGCOFBINPCAXTABJ6VCEIINTNUUPHUBY6T62QPT3LQ, seg_pos={0 0}, upload=447317]: Uploading piece... Dec 26 21:14:21.946744042 (509h45m21.817827077s) [noise=true , address=145.79.227.49:4339 , piece_id=CI5H6XSHFRJ4ULHJ5I6KMJYAA7EJRKAVP5EKTUVMQ4ZTHFMINEUA, seg_pos={0 0}, upload=447317]: Uploading piece... Dec 26 21:14:21.947185513 (509h45m21.818268548s) [noise=true , address=77.90.31.18:4808 , piece_id=PIKGXUJXLBZQUGBWWCQJITQGSDZV6XNGM2RL72GC7LGFUF5IQ3WQ, seg_pos={0 0}, upload=447317]: Uploading piece... Dec 26 21:14:21.947585944 (509h45m21.81866899s) [noise=true , address=45.91.116.5:4300 , piece_id=LSUMIXQXHZAZNCWPNHW3L6B6QNJIGVH2OZGNOFZQ6WGIOFARYE7A, seg_pos={0 0}, upload=447317]: Uploading piece... Dec 26 21:14:21.947861099 (509h45m21.818944134s) [noise=true , address=45.91.116.14:4189 , piece_id=ATLBKYEJXPIO5X3JIA7OFSQTVIWKXZIIQZIYGOT5JABBML2JFE5Q, seg_pos={0 0}, upload=447317]: Uploading piece... Dec 26 21:14:21.948146372 (509h45m21.819229417s) [noise=true , address=145.79.227.41:4087 , piece_id=2WLKJ7YOIYSJQHLV4XPKNJV5626NT2GMTHSSGQQFBBXJ5J3RDWOQ, seg_pos={0 0}, upload=447317]: Uploading piece... Dec 26 21:14:21.948473385 (509h45m21.81955642s) [noise=true , address=145.79.227.52:4445 , piece_id=YIJPAKGBHO37SQRX624GZ5EF5QUR4H472MRVMNQVGS2MPCN2IDMQ, seg_pos={0 0}, upload=447317]: Uploading piece... Dec 26 21:14:21.948780068 (509h45m21.819863103s) [noise=true , address=145.79.227.3:4149 , piece_id=XW5GQOFPCYGZZIJEEU5XQC74C64UCNKOT5ASSYN6KT7QOHZNCJDQ, seg_pos={0 0}, upload=447317]: Uploading piece... Dec 26 21:14:21.949068738 (509h45m21.820151783s) [noise=true , address=145.79.227.45:4231 , piece_id=RIVKLAAVK2W7H5CQM3ER6RD5WQMXKD5FJYSOYBTIHP3YXRUTLWSA, seg_pos={0 0}, upload=447317]: Uploading piece... Dec 26 21:14:21.949868251 (509h45m21.820951296s) [noise=true , address=145.79.227.50:4373 , piece_id=O6XKMGZSEPOUL7G57KPSKGBSJFD76FXIEDC3YMFZ3F7WIFV2N76Q, seg_pos={0 0}, upload=447317]: Uploading piece... Dec 26 21:14:21.950670298 (509h45m21.821753333s) [noise=true , address=77.90.31.14:4664 , piece_id=6BKBICK2ESYZKHZRJ3RMTT7HHOXJ2IIPWIBWGEZ27VQA2DVS7T7A, seg_pos={0 0}, upload=447317]: Uploading piece... Dec 26 21:14:21.951248368 (509h45m21.822331413s) [noise=true , address=45.91.116.7:4011 , piece_id=XXT4256PQI3JDBLP62QTMDRM5FCYO55QHFBJ74BNWVUS2MZHMHIQ, seg_pos={0 0}, upload=447317]: Uploading piece... Dec 26 21:14:21.951866836 (509h45m21.822949871s) [noise=true , address=145.79.227.37:4565 , piece_id=IOAE2CERI4NPTPAS6HEZXCAIFABZPKQNEKDNWI7I3Y4LH7SDH4XQ, seg_pos={0 0}, upload=447317]: Uploading piece... Dec 26 21:14:23.658453334 (509h45m23.529536369s) [upload=447317]: Next returned nil segment Dec 26 21:14:23.658467390 (509h45m23.529550425s) [upload=447317]: Waiting for error group managing segments... Dec 26 21:14:23.945957448 (509h45m23.817040483s) [noise=true , address=145.79.227.50:4373 , piece_id=O6XKMGZSEPOUL7G57KPSKGBSJFD76FXIEDC3YMFZ3F7WIFV2N76Q, seg_pos={0 0}, upload=447317]: Done uploading piece. err: <nil> Dec 26 21:14:23.947571752 (509h45m23.818654787s) [noise=true , address=145.79.227.49:4339 , piece_id=CI5H6XSHFRJ4ULHJ5I6KMJYAA7EJRKAVP5EKTUVMQ4ZTHFMINEUA, seg_pos={0 0}, upload=447317]: Done uploading piece. err: <nil> Dec 26 21:14:23.951076015 (509h45m23.82215906s) [noise=true , address=145.79.227.51:4409 , piece_id=JAWYLPDQREAGCOFBINPCAXTABJ6VCEIINTNUUPHUBY6T62QPT3LQ, seg_pos={0 0}, upload=447317]: Done uploading piece. err: <nil> Dec 26 21:14:23.956390795 (509h45m23.82747383s) [noise=true , address=145.79.227.15:4876 , piece_id=JPOYCRWQ3YEC22JVO4LK6EEEOIXNO3NMDUPJJFQVQA4KI6AART7A, seg_pos={0 0}, upload=447317]: Done uploading piece. err: <nil> Dec 26 21:14:23.967483197 (509h45m23.838566242s) [noise=true , address=145.79.227.41:4087 , piece_id=2WLKJ7YOIYSJQHLV4XPKNJV5626NT2GMTHSSGQQFBBXJ5J3RDWOQ, seg_pos={0 0}, upload=447317]: Done uploading piece. err: <nil> Dec 26 21:14:23.969921921 (509h45m23.841004956s) [noise=true , address=145.79.227.52:4445 , piece_id=YIJPAKGBHO37SQRX624GZ5EF5QUR4H472MRVMNQVGS2MPCN2IDMQ, seg_pos={0 0}, upload=447317]: Done uploading piece. err: <nil> Dec 26 21:14:23.979221749 (509h45m23.850304794s) [noise=true , address=145.79.227.45:4231 , piece_id=RIVKLAAVK2W7H5CQM3ER6RD5WQMXKD5FJYSOYBTIHP3YXRUTLWSA, seg_pos={0 0}, upload=447317]: Done uploading piece. err: <nil> Dec 26 21:14:23.980514381 (509h45m23.851597426s) [noise=true , address=145.79.227.37:4565 , piece_id=IOAE2CERI4NPTPAS6HEZXCAIFABZPKQNEKDNWI7I3Y4LH7SDH4XQ, seg_pos={0 0}, upload=447317]: Done uploading piece. err: <nil> Dec 26 21:14:24.142582120 (509h45m24.013665155s) [noise=true , address=77.90.31.14:4664 , piece_id=6BKBICK2ESYZKHZRJ3RMTT7HHOXJ2IIPWIBWGEZ27VQA2DVS7T7A, seg_pos={0 0}, upload=447317]: Done uploading piece. err: <nil> Dec 26 21:14:24.174440284 (509h45m24.045523329s) [noise=true , address=77.90.31.18:4808 , piece_id=PIKGXUJXLBZQUGBWWCQJITQGSDZV6XNGM2RL72GC7LGFUF5IQ3WQ, seg_pos={0 0}, upload=447317]: Done uploading piece. err: <nil> Dec 26 21:14:24.255083445 (509h45m24.12616648s) [noise=true , address=45.91.116.7:4011 , piece_id=XXT4256PQI3JDBLP62QTMDRM5FCYO55QHFBJ74BNWVUS2MZHMHIQ, seg_pos={0 0}, upload=447317]: Done uploading piece. err: <nil> Dec 26 21:14:24.273742405 (509h45m24.14482545s) [noise=true , address=45.91.116.14:4189 , piece_id=ATLBKYEJXPIO5X3JIA7OFSQTVIWKXZIIQZIYGOT5JABBML2JFE5Q, seg_pos={0 0}, upload=447317]: Done uploading piece. err: <nil> Dec 26 21:14:24.292559737 (509h45m24.163642782s) [noise=true , address=45.91.116.11:4118 , piece_id=GHEVUBPZYS3SMSUM4PNR36IEXPXZCLFGWTFUWM6BQ2QKVMGLMPRA, seg_pos={0 0}, upload=447317]: Done uploading piece. err: <nil> Dec 26 21:14:24.292579244 (509h45m24.163662289s) [seg_pos={0 0}, upload=447317]: Segment reached optimal threshold of 13 pieces. Dec 26 21:14:24.292729099 (509h45m24.163812155s) [noise=true , address=45.91.116.5:4300 , piece_id=LSUMIXQXHZAZNCWPNHW3L6B6QNJIGVH2OZGNOFZQ6WGIOFARYE7A, seg_pos={0 0}, upload=447317]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 26 21:14:24.292734660 (509h45m24.163817705s) [noise=true , address=45.91.116.5:4300 , piece_id=LSUMIXQXHZAZNCWPNHW3L6B6QNJIGVH2OZGNOFZQ6WGIOFARYE7A, seg_pos={0 0}, upload=447317]: Long tail cancellation due to optimal threshold Dec 26 21:14:24.292977262 (509h45m24.164060307s) [noise=true , address=45.91.116.6:4974 , piece_id=PKHOVEREEYILOICNAOVUXUPXU2HWPMU2MV5HEMXLXK4BGSEYRW4A, seg_pos={0 0}, upload=447317]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 26 21:14:24.292983704 (509h45m24.164066749s) [noise=true , address=45.91.116.6:4974 , piece_id=PKHOVEREEYILOICNAOVUXUPXU2HWPMU2MV5HEMXLXK4BGSEYRW4A, seg_pos={0 0}, upload=447317]: Long tail cancellation due to optimal threshold Dec 26 21:14:24.293073796 (509h45m24.164156831s) [noise=true , address=145.79.227.3:4149 , piece_id=XW5GQOFPCYGZZIJEEU5XQC74C64UCNKOT5ASSYN6KT7QOHZNCJDQ, seg_pos={0 0}, upload=447317]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm): protocol: expected piece hash; context canceled; EOF Dec 26 21:14:24.293078815 (509h45m24.16416185s) [noise=true , address=145.79.227.3:4149 , piece_id=XW5GQOFPCYGZZIJEEU5XQC74C64UCNKOT5ASSYN6KT7QOHZNCJDQ, seg_pos={0 0}, upload=447317]: Long tail cancellation due to optimal threshold Dec 26 21:14:24.293092943 (509h45m24.164175978s) [seg_pos={0 0}, upload=447317]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 21:14:24.293159289 (509h45m24.164242334s) [upload=447317]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:14:24.293163637 (509h45m24.164246672s) [upload=447317]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:14:24.580326893 (509h45m24.451409928s) [upload=447317]: Done uploading segments... Dec 26 21:14:24.580353343 (509h45m24.451436378s) [upload=447317]: Upload finished. err: <nil> Duration: 14.547336933s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 27.340291883s trace: https://jaeger.dev.storj.io/trace/6d2950019117adaf FAIL (2025-12-26 21:15:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Rj4hUGWvUPuU49QDsHYhibiJhyY5KDGus1rzNeP7KusgyvTsU, Piece ID: NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A) context deadline exceeded; piecestore: (Node ID: 12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, Piece ID: N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ) context deadline exceeded; piecestore: (Node ID: 12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, Piece ID: IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ) context deadline exceeded ======LOGS====== ng piece... Dec 26 20:54:08.087336805 (509h25m7.95841984s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087649520 (509h25m7.958732565s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087988717 (509h25m7.959071752s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088323233 (509h25m7.959406268s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088877028 (509h25m7.959960063s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089247203 (509h25m7.960330238s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089543066 (509h25m7.960626101s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089914133 (509h25m7.960997168s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090327259 (509h25m7.961410294s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090645345 (509h25m7.96172838s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091004318 (509h25m7.962087353s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091338976 (509h25m7.962422021s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091732054 (509h25m7.962815099s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:09.565187143 (509h25m9.436270188s) [upload=447022]: Next returned nil segment Dec 26 20:54:09.565201321 (509h25m9.436284356s) [upload=447022]: Waiting for error group managing segments... Dec 26 20:54:10.137065972 (509h25m10.008149007s) [noise=true , address=77.90.31.10:4523 , piece_id=GQYT3Z53Y6HVTY3IBUNFCIWBEE7KVAUWEJYO7QJFRRAJCHTJMYFA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.137147197 (509h25m10.008230232s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.149325196 (509h25m10.020408242s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.159611673 (509h25m10.030694718s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.164411764 (509h25m10.035494809s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.190924278 (509h25m10.062007313s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.232931086 (509h25m10.104014131s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.241658245 (509h25m10.11274129s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.472574370 (509h25m10.343657405s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.866910566 (509h25m10.737993611s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.886583507 (509h25m10.757666542s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.127773215 (509h25m10.99885626s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.192133217 (509h25m11.063216263s) [noise=true , address=145.79.227.13:4819 , piece_id=BQDRL2QJZ75TAMUM2VSGTF4U3NEMZC4FFQLN6QV6UOAFHWB56MZA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256146219 (509h25m11.127229264s) [noise=true , address=145.79.227.40:4062 , piece_id=IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256176597 (509h25m11.127259632s) [seg_pos={0 0}, upload=447022]: Segment reached optimal threshold of 14 pieces. Dec 26 20:54:11.256428137 (509h25m11.127511192s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98): protocol: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection Dec 26 20:54:11.256439549 (509h25m11.127522584s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256458915 (509h25m11.12754198s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF): protocol: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection Dec 26 20:54:11.256490705 (509h25m11.127573741s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256698171 (509h25m11.127781216s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: rpc: dial tcp 145.79.227.30:4796: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:54:11.256706457 (509h25m11.127789502s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256917309 (509h25m11.128000364s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection Dec 26 20:54:11.256926666 (509h25m11.128009711s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256964869 (509h25m11.128047914s) [seg_pos={0 0}, upload=447022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:54:11.257064138 (509h25m11.128147173s) [upload=447022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:11.257070690 (509h25m11.128153725s) [upload=447022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:11.561962810 (509h25m11.433045875s) [upload=447022]: Done uploading segments... Dec 26 20:54:11.561987427 (509h25m11.433070472s) [upload=447022]: Upload finished. err: <nil> Duration: 27.340320347s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.XXX...X.XX..XXX...............X....................XX.XX.X.] duration: 6.332171966s trace: https://jaeger.dev.storj.io/trace/e1dc9001c283d0f FAIL (2025-12-26 21:15:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: GHEVUBPZYS3SMSUM4PNR36IEXPXZCLFGWTFUWM6BQ2QKVMGLMPRA) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: 6BKBICK2ESYZKHZRJ3RMTT7HHOXJ2IIPWIBWGEZ27VQA2DVS7T7A) context deadline exceeded ======LOGS====== Dec 26 21:14:21.701071402 (509h45m21.572154437s) [upload=447317]: Starting upload Dec 26 21:14:21.701221497 (509h45m21.572304542s) [upload=447317]: Done starting upload Dec 26 21:14:21.701347367 (509h45m21.572430432s) [upload=447317]: Uploading segments... Dec 26 21:14:21.701521769 (509h45m21.572604805s) [upload=447317]: Got next segment. Inline: false Dec 26 21:14:21.701657448 (509h45m21.572740493s) [upload=447317]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 21:14:21.701665814 (509h45m21.572748859s) [upload=447317]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 21:14:21.945615572 (509h45m21.816698637s) [seg_pos={0 0}, upload=447317]: Begin upload segment... Dec 26 21:14:21.945684473 (509h45m21.816767518s) [seg_pos={0 0}, upload=447317]: Done begin upload segment. Dec 26 21:14:21.945742925 (509h45m21.81682596s) [noise=true , address=145.79.227.15:4876 , piece_id=JPOYCRWQ3YEC22JVO4LK6EEEOIXNO3NMDUPJJFQVQA4KI6AART7A, seg_pos={0 0}, upload=447317]: Uploading piece... Dec 26 21:14:21.945755729 (509h45m21.816838774s) [noise=true , address=45.91.116.6:4974 , piece_id=PKHOVEREEYILOICNAOVUXUPXU2HWPMU2MV5HEMXLXK4BGSEYRW4A, seg_pos={0 0}, upload=447317]: Uploading piece... Dec 26 21:14:21.945833889 (509h45m21.816916924s) [noise=true , address=45.91.116.11:4118 , piece_id=GHEVUBPZYS3SMSUM4PNR36IEXPXZCLFGWTFUWM6BQ2QKVMGLMPRA, seg_pos={0 0}, upload=447317]: Uploading piece... Dec 26 21:14:21.946386711 (509h45m21.817469756s) [noise=true , address=145.79.227.51:4409 , piece_id=JAWYLPDQREAGCOFBINPCAXTABJ6VCEIINTNUUPHUBY6T62QPT3LQ, seg_pos={0 0}, upload=447317]: Uploading piece... Dec 26 21:14:21.946744042 (509h45m21.817827077s) [noise=true , address=145.79.227.49:4339 , piece_id=CI5H6XSHFRJ4ULHJ5I6KMJYAA7EJRKAVP5EKTUVMQ4ZTHFMINEUA, seg_pos={0 0}, upload=447317]: Uploading piece... Dec 26 21:14:21.947185513 (509h45m21.818268548s) [noise=true , address=77.90.31.18:4808 , piece_id=PIKGXUJXLBZQUGBWWCQJITQGSDZV6XNGM2RL72GC7LGFUF5IQ3WQ, seg_pos={0 0}, upload=447317]: Uploading piece... Dec 26 21:14:21.947585944 (509h45m21.81866899s) [noise=true , address=45.91.116.5:4300 , piece_id=LSUMIXQXHZAZNCWPNHW3L6B6QNJIGVH2OZGNOFZQ6WGIOFARYE7A, seg_pos={0 0}, upload=447317]: Uploading piece... Dec 26 21:14:21.947861099 (509h45m21.818944134s) [noise=true , address=45.91.116.14:4189 , piece_id=ATLBKYEJXPIO5X3JIA7OFSQTVIWKXZIIQZIYGOT5JABBML2JFE5Q, seg_pos={0 0}, upload=447317]: Uploading piece... Dec 26 21:14:21.948146372 (509h45m21.819229417s) [noise=true , address=145.79.227.41:4087 , piece_id=2WLKJ7YOIYSJQHLV4XPKNJV5626NT2GMTHSSGQQFBBXJ5J3RDWOQ, seg_pos={0 0}, upload=447317]: Uploading piece... Dec 26 21:14:21.948473385 (509h45m21.81955642s) [noise=true , address=145.79.227.52:4445 , piece_id=YIJPAKGBHO37SQRX624GZ5EF5QUR4H472MRVMNQVGS2MPCN2IDMQ, seg_pos={0 0}, upload=447317]: Uploading piece... Dec 26 21:14:21.948780068 (509h45m21.819863103s) [noise=true , address=145.79.227.3:4149 , piece_id=XW5GQOFPCYGZZIJEEU5XQC74C64UCNKOT5ASSYN6KT7QOHZNCJDQ, seg_pos={0 0}, upload=447317]: Uploading piece... Dec 26 21:14:21.949068738 (509h45m21.820151783s) [noise=true , address=145.79.227.45:4231 , piece_id=RIVKLAAVK2W7H5CQM3ER6RD5WQMXKD5FJYSOYBTIHP3YXRUTLWSA, seg_pos={0 0}, upload=447317]: Uploading piece... Dec 26 21:14:21.949868251 (509h45m21.820951296s) [noise=true , address=145.79.227.50:4373 , piece_id=O6XKMGZSEPOUL7G57KPSKGBSJFD76FXIEDC3YMFZ3F7WIFV2N76Q, seg_pos={0 0}, upload=447317]: Uploading piece... Dec 26 21:14:21.950670298 (509h45m21.821753333s) [noise=true , address=77.90.31.14:4664 , piece_id=6BKBICK2ESYZKHZRJ3RMTT7HHOXJ2IIPWIBWGEZ27VQA2DVS7T7A, seg_pos={0 0}, upload=447317]: Uploading piece... Dec 26 21:14:21.951248368 (509h45m21.822331413s) [noise=true , address=45.91.116.7:4011 , piece_id=XXT4256PQI3JDBLP62QTMDRM5FCYO55QHFBJ74BNWVUS2MZHMHIQ, seg_pos={0 0}, upload=447317]: Uploading piece... Dec 26 21:14:21.951866836 (509h45m21.822949871s) [noise=true , address=145.79.227.37:4565 , piece_id=IOAE2CERI4NPTPAS6HEZXCAIFABZPKQNEKDNWI7I3Y4LH7SDH4XQ, seg_pos={0 0}, upload=447317]: Uploading piece... Dec 26 21:14:23.658453334 (509h45m23.529536369s) [upload=447317]: Next returned nil segment Dec 26 21:14:23.658467390 (509h45m23.529550425s) [upload=447317]: Waiting for error group managing segments... Dec 26 21:14:23.945957448 (509h45m23.817040483s) [noise=true , address=145.79.227.50:4373 , piece_id=O6XKMGZSEPOUL7G57KPSKGBSJFD76FXIEDC3YMFZ3F7WIFV2N76Q, seg_pos={0 0}, upload=447317]: Done uploading piece. err: <nil> Dec 26 21:14:23.947571752 (509h45m23.818654787s) [noise=true , address=145.79.227.49:4339 , piece_id=CI5H6XSHFRJ4ULHJ5I6KMJYAA7EJRKAVP5EKTUVMQ4ZTHFMINEUA, seg_pos={0 0}, upload=447317]: Done uploading piece. err: <nil> Dec 26 21:14:23.951076015 (509h45m23.82215906s) [noise=true , address=145.79.227.51:4409 , piece_id=JAWYLPDQREAGCOFBINPCAXTABJ6VCEIINTNUUPHUBY6T62QPT3LQ, seg_pos={0 0}, upload=447317]: Done uploading piece. err: <nil> Dec 26 21:14:23.956390795 (509h45m23.82747383s) [noise=true , address=145.79.227.15:4876 , piece_id=JPOYCRWQ3YEC22JVO4LK6EEEOIXNO3NMDUPJJFQVQA4KI6AART7A, seg_pos={0 0}, upload=447317]: Done uploading piece. err: <nil> Dec 26 21:14:23.967483197 (509h45m23.838566242s) [noise=true , address=145.79.227.41:4087 , piece_id=2WLKJ7YOIYSJQHLV4XPKNJV5626NT2GMTHSSGQQFBBXJ5J3RDWOQ, seg_pos={0 0}, upload=447317]: Done uploading piece. err: <nil> Dec 26 21:14:23.969921921 (509h45m23.841004956s) [noise=true , address=145.79.227.52:4445 , piece_id=YIJPAKGBHO37SQRX624GZ5EF5QUR4H472MRVMNQVGS2MPCN2IDMQ, seg_pos={0 0}, upload=447317]: Done uploading piece. err: <nil> Dec 26 21:14:23.979221749 (509h45m23.850304794s) [noise=true , address=145.79.227.45:4231 , piece_id=RIVKLAAVK2W7H5CQM3ER6RD5WQMXKD5FJYSOYBTIHP3YXRUTLWSA, seg_pos={0 0}, upload=447317]: Done uploading piece. err: <nil> Dec 26 21:14:23.980514381 (509h45m23.851597426s) [noise=true , address=145.79.227.37:4565 , piece_id=IOAE2CERI4NPTPAS6HEZXCAIFABZPKQNEKDNWI7I3Y4LH7SDH4XQ, seg_pos={0 0}, upload=447317]: Done uploading piece. err: <nil> Dec 26 21:14:24.142582120 (509h45m24.013665155s) [noise=true , address=77.90.31.14:4664 , piece_id=6BKBICK2ESYZKHZRJ3RMTT7HHOXJ2IIPWIBWGEZ27VQA2DVS7T7A, seg_pos={0 0}, upload=447317]: Done uploading piece. err: <nil> Dec 26 21:14:24.174440284 (509h45m24.045523329s) [noise=true , address=77.90.31.18:4808 , piece_id=PIKGXUJXLBZQUGBWWCQJITQGSDZV6XNGM2RL72GC7LGFUF5IQ3WQ, seg_pos={0 0}, upload=447317]: Done uploading piece. err: <nil> Dec 26 21:14:24.255083445 (509h45m24.12616648s) [noise=true , address=45.91.116.7:4011 , piece_id=XXT4256PQI3JDBLP62QTMDRM5FCYO55QHFBJ74BNWVUS2MZHMHIQ, seg_pos={0 0}, upload=447317]: Done uploading piece. err: <nil> Dec 26 21:14:24.273742405 (509h45m24.14482545s) [noise=true , address=45.91.116.14:4189 , piece_id=ATLBKYEJXPIO5X3JIA7OFSQTVIWKXZIIQZIYGOT5JABBML2JFE5Q, seg_pos={0 0}, upload=447317]: Done uploading piece. err: <nil> Dec 26 21:14:24.292559737 (509h45m24.163642782s) [noise=true , address=45.91.116.11:4118 , piece_id=GHEVUBPZYS3SMSUM4PNR36IEXPXZCLFGWTFUWM6BQ2QKVMGLMPRA, seg_pos={0 0}, upload=447317]: Done uploading piece. err: <nil> Dec 26 21:14:24.292579244 (509h45m24.163662289s) [seg_pos={0 0}, upload=447317]: Segment reached optimal threshold of 13 pieces. Dec 26 21:14:24.292729099 (509h45m24.163812155s) [noise=true , address=45.91.116.5:4300 , piece_id=LSUMIXQXHZAZNCWPNHW3L6B6QNJIGVH2OZGNOFZQ6WGIOFARYE7A, seg_pos={0 0}, upload=447317]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 26 21:14:24.292734660 (509h45m24.163817705s) [noise=true , address=45.91.116.5:4300 , piece_id=LSUMIXQXHZAZNCWPNHW3L6B6QNJIGVH2OZGNOFZQ6WGIOFARYE7A, seg_pos={0 0}, upload=447317]: Long tail cancellation due to optimal threshold Dec 26 21:14:24.292977262 (509h45m24.164060307s) [noise=true , address=45.91.116.6:4974 , piece_id=PKHOVEREEYILOICNAOVUXUPXU2HWPMU2MV5HEMXLXK4BGSEYRW4A, seg_pos={0 0}, upload=447317]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 26 21:14:24.292983704 (509h45m24.164066749s) [noise=true , address=45.91.116.6:4974 , piece_id=PKHOVEREEYILOICNAOVUXUPXU2HWPMU2MV5HEMXLXK4BGSEYRW4A, seg_pos={0 0}, upload=447317]: Long tail cancellation due to optimal threshold Dec 26 21:14:24.293073796 (509h45m24.164156831s) [noise=true , address=145.79.227.3:4149 , piece_id=XW5GQOFPCYGZZIJEEU5XQC74C64UCNKOT5ASSYN6KT7QOHZNCJDQ, seg_pos={0 0}, upload=447317]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm): protocol: expected piece hash; context canceled; EOF Dec 26 21:14:24.293078815 (509h45m24.16416185s) [noise=true , address=145.79.227.3:4149 , piece_id=XW5GQOFPCYGZZIJEEU5XQC74C64UCNKOT5ASSYN6KT7QOHZNCJDQ, seg_pos={0 0}, upload=447317]: Long tail cancellation due to optimal threshold Dec 26 21:14:24.293092943 (509h45m24.164175978s) [seg_pos={0 0}, upload=447317]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 21:14:24.293159289 (509h45m24.164242334s) [upload=447317]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:14:24.293163637 (509h45m24.164246672s) [upload=447317]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:14:24.580326893 (509h45m24.451409928s) [upload=447317]: Done uploading segments... Dec 26 21:14:24.580353343 (509h45m24.451436378s) [upload=447317]: Upload finished. err: <nil> Duration: 6.33220055s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)