Orbiter History

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [............X..] duration: 314.541659ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/1fc7c26ef222e679 FAIL (2025-11-14 01:51:49 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:53548->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 314.570012ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 2 out of 15 is failed (threshold=6) [..........X.X..] duration: 48.136341ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/8a3ce894c9564a5 FAIL (2025-11-14 01:51:34 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:43336->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 48.166728ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.912549334s trace: https://jaeger.dev.storj.io/trace/5bc86ccfe33cf550 FAIL (2025-11-14 01:51:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ce_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288397607 (113h30m2.839868078s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288507285 (113h30m2.839977746s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.289077772 (113h30m2.840548233s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290074169 (113h30m2.84154464s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290622854 (113h30m2.842093345s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292062385 (113h30m2.843532866s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292958642 (113h30m2.844429123s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.294563758 (113h30m2.846034229s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.296755000 (113h30m2.848225462s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.298618379 (113h30m2.85008884s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.300188338 (113h30m2.85165881s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.301125483 (113h30m2.852595944s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302545266 (113h30m2.854015727s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302898339 (113h30m2.8543688s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.303825163 (113h30m2.855295624s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.316046324 (113h30m2.867516785s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.318180529 (113h30m2.86965099s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:05.581380111 (113h30m4.132850592s) [upload=99881]: Next returned nil segment Nov 14 01:34:05.581407382 (113h30m4.132877844s) [upload=99881]: Waiting for error group managing segments... Nov 14 01:34:05.921914348 (113h30m4.473384809s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.005586503 (113h30m4.557056964s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.026850744 (113h30m4.578321205s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.121926626 (113h30m4.673397087s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.235634986 (113h30m4.787105437s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.277897963 (113h30m4.829368424s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.283628915 (113h30m4.835099376s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.287808744 (113h30m4.839279205s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.300041498 (113h30m4.851511949s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.304400718 (113h30m4.855871179s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.325040680 (113h30m4.876511141s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.331007762 (113h30m4.882478223s) [noise=true , address=77.90.31.12:4611 , piece_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.481058726 (113h30m5.032529207s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:35:02.598238012 (113h31m1.149708493s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection Nov 14 01:35:02.599889276 (113h31m1.151359737s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:145.79.227.52:4455): protocol: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection Nov 14 01:35:02.600041686 (113h31m1.151512147s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:127WWD5hr1ZBkRLCHz3jb7kpqqnxPS6VuZ1MyQDmitFbhTSbJNK, address:145.79.227.49:4344): protocol: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection Nov 14 01:35:02.600244733 (113h31m1.151715204s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:145.79.227.35:4488): protocol: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection Nov 14 01:35:02.600569551 (113h31m1.152040022s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12h6MND3rBadG4EjUP18S7izyj23cVBLaB5drFsuNGhcxwAYXSA, address:145.79.227.17:4983): protocol: read tcp 188.166.225.170:33042->145.79.227.17:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:35:02.600603536 (113h31m1.152073997s) [seg_pos={0 0}, upload=99881]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 01:35:02.847367167 (113h31m1.398837648s) [upload=99881]: Done uploading segments... Nov 14 01:35:02.847427322 (113h31m1.398897783s) [upload=99881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.912565956s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X.............X...................X........................] duration: 5.883667985s trace: https://jaeger.dev.storj.io/trace/4377cd58781b9764 FAIL (2025-11-14 01:51:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== DXCQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351147808 (113h30m14.902618269s) [noise=true , address=77.90.31.13:4628 , piece_id=WHP562KP7DABPK6XQT5F43QGNNCDFXYYYP775VGZLXIG2UEWKK4A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351381713 (113h30m14.902852174s) [noise=true , address=145.79.227.30:4763 , piece_id=IW3HMSEGIX2TFUY277OSEF4BULPDWA3AO3MNJ7OCUULORPLHNWOQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351673048 (113h30m14.903143509s) [noise=true , address=145.79.227.26:4653 , piece_id=EEKXCCTYRNCRJFVM2QBQVQADQJ2YVDCDOXFIFUP4BYUEVYDGWZQA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351818645 (113h30m14.903289106s) [noise=true , address=145.79.227.14:4840 , piece_id=3DEHNGEWEG4GA5YTLKGDV4BYUJ7Z6KCVLM343ZB5MYZWEMOASS7A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352062729 (113h30m14.903533191s) [noise=true , address=45.91.116.13:4154 , piece_id=DSC3N7KGSUSEVCEII65IQMYLHEUC5U52NNSREC32Y3Q3MDNKWREQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352224859 (113h30m14.90369532s) [noise=true , address=77.90.31.18:4807 , piece_id=TD3XBJUZ6MM5CB7NMYEIZWQC63U2PZZJPYHIM3OOU7PE3Y7KGCIQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352479703 (113h30m14.903950165s) [noise=true , address=77.90.31.17:4771 , piece_id=ARX2MBQVJSPWKWS2AHE2QBBWNREGMUORBQPNFQJHJDWBT2FCDPAA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352614791 (113h30m14.904085342s) [noise=true , address=145.79.227.21:4473 , piece_id=DIGA6L2TCAOGHAGUY5K4T3R7H66CF3JLXBXKMC7TPTXVKBCLDJBA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352904944 (113h30m14.904375405s) [noise=true , address=45.91.116.6:4973 , piece_id=NEGS6AAGBP4WRJJ6TXEBNHCV2FXW2OBMRWAWL4RNY4YZZZSI3ZLQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.353329152 (113h30m14.904799613s) [noise=true , address=77.90.31.12:4592 , piece_id=ET6LSZ6LXAAAPVJLBLL4UOIF6SKOKDCKXEXKIQ4OS3RD5J5QB2UQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.353685219 (113h30m14.90515568s) [noise=true , address=145.79.227.16:4912 , piece_id=SKK6OIUPSWXXD6QS5ZCVVORVSFXXOE5UJH7POHTKYY6PU4LX2JGA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354114446 (113h30m14.905584907s) [noise=true , address=45.91.116.12:4225 , piece_id=RMDFCX4JGDRPQ46PPTYAWJE7NIYVDE3J2LCKREBFXHR7I6U2G52A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354334907 (113h30m14.905805368s) [noise=true , address=145.79.227.13:4804 , piece_id=NZL7G7WFABHCYGW63IPFVWP45AMPZRDYLDF4JNKJHK5PLSHYHLSQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354804771 (113h30m14.906275222s) [noise=true , address=145.79.227.20:4439 , piece_id=FEDAGRFBHNL6ES3XTCRWPEI4NN2CMMDJIVNW5WOG3KYD7VXFXA7Q, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:18.090813414 (113h30m16.642283865s) [upload=99885]: Next returned nil segment Nov 14 01:34:18.090832882 (113h30m16.642303333s) [upload=99885]: Waiting for error group managing segments... Nov 14 01:34:18.376699271 (113h30m16.928169732s) [noise=true , address=45.91.116.12:4225 , piece_id=RMDFCX4JGDRPQ46PPTYAWJE7NIYVDE3J2LCKREBFXHR7I6U2G52A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.449067032 (113h30m17.000537493s) [noise=true , address=77.90.31.18:4807 , piece_id=TD3XBJUZ6MM5CB7NMYEIZWQC63U2PZZJPYHIM3OOU7PE3Y7KGCIQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.476540927 (113h30m17.028011378s) [noise=true , address=77.90.31.13:4628 , piece_id=WHP562KP7DABPK6XQT5F43QGNNCDFXYYYP775VGZLXIG2UEWKK4A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.502738892 (113h30m17.054209343s) [noise=true , address=77.90.31.12:4592 , piece_id=ET6LSZ6LXAAAPVJLBLL4UOIF6SKOKDCKXEXKIQ4OS3RD5J5QB2UQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.504520516 (113h30m17.055990977s) [noise=true , address=45.91.116.6:4973 , piece_id=NEGS6AAGBP4WRJJ6TXEBNHCV2FXW2OBMRWAWL4RNY4YZZZSI3ZLQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.504778116 (113h30m17.056248577s) [noise=true , address=45.91.116.13:4154 , piece_id=DSC3N7KGSUSEVCEII65IQMYLHEUC5U52NNSREC32Y3Q3MDNKWREQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.511639541 (113h30m17.063110002s) [noise=true , address=77.90.31.10:4517 , piece_id=GDOCT6DE4NL56DWKOS3JNSQVIVE342SDIZ5YR7XPNRXYPOQYDXCQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.529697226 (113h30m17.081167687s) [noise=true , address=77.90.31.17:4771 , piece_id=ARX2MBQVJSPWKWS2AHE2QBBWNREGMUORBQPNFQJHJDWBT2FCDPAA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:45.367038063 (113h30m43.918508514s) [noise=true , address=145.79.227.26:4653 , piece_id=EEKXCCTYRNCRJFVM2QBQVQADQJ2YVDCDOXFIFUP4BYUEVYDGWZQA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection Nov 14 01:34:45.367162339 (113h30m43.91863279s) [noise=true , address=145.79.227.14:4840 , piece_id=3DEHNGEWEG4GA5YTLKGDV4BYUJ7Z6KCVLM343ZB5MYZWEMOASS7A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection Nov 14 01:34:45.367201353 (113h30m43.918671804s) [noise=true , address=145.79.227.9:4366 , piece_id=GXQSDBFPFZPZ352A5KBNQJS5A32C7ZO522AHCHO3A27P4LAMUFZQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection Nov 14 01:34:45.367247291 (113h30m43.918717742s) [noise=true , address=145.79.227.20:4439 , piece_id=FEDAGRFBHNL6ES3XTCRWPEI4NN2CMMDJIVNW5WOG3KYD7VXFXA7Q, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection; write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection Nov 14 01:34:45.367425070 (113h30m43.918895531s) [noise=true , address=145.79.227.13:4804 , piece_id=NZL7G7WFABHCYGW63IPFVWP45AMPZRDYLDF4JNKJHK5PLSHYHLSQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection Nov 14 01:34:45.367779144 (113h30m43.919249605s) [noise=true , address=145.79.227.30:4763 , piece_id=IW3HMSEGIX2TFUY277OSEF4BULPDWA3AO3MNJ7OCUULORPLHNWOQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:50566->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:34:45.367825933 (113h30m43.919296394s) [noise=true , address=145.79.227.16:4912 , piece_id=SKK6OIUPSWXXD6QS5ZCVVORVSFXXOE5UJH7POHTKYY6PU4LX2JGA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection Nov 14 01:34:45.368025623 (113h30m43.919496114s) [noise=true , address=145.79.227.21:4473 , piece_id=DIGA6L2TCAOGHAGUY5K4T3R7H66CF3JLXBXKMC7TPTXVKBCLDJBA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: read tcp 188.166.225.170:49576->145.79.227.21:4473: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:34:45.368068755 (113h30m43.919539206s) [seg_pos={0 0}, upload=99885]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 14 01:34:45.634359530 (113h30m44.185829991s) [upload=99885]: Done uploading segments... Nov 14 01:34:45.634431096 (113h30m44.185901557s) [upload=99885]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.883695356s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 2 out of 15 is failed (threshold=6) [...........X.X.] duration: 53.048044ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/8a3ce894c9564a5 FAIL (2025-11-14 01:51:10 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:43336->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 53.080295ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X.............X...................X.......................] duration: 5.855423804s trace: https://jaeger.dev.storj.io/trace/4377cd58781b9764 FAIL (2025-11-14 01:51:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== DXCQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351147808 (113h30m14.902618269s) [noise=true , address=77.90.31.13:4628 , piece_id=WHP562KP7DABPK6XQT5F43QGNNCDFXYYYP775VGZLXIG2UEWKK4A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351381713 (113h30m14.902852174s) [noise=true , address=145.79.227.30:4763 , piece_id=IW3HMSEGIX2TFUY277OSEF4BULPDWA3AO3MNJ7OCUULORPLHNWOQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351673048 (113h30m14.903143509s) [noise=true , address=145.79.227.26:4653 , piece_id=EEKXCCTYRNCRJFVM2QBQVQADQJ2YVDCDOXFIFUP4BYUEVYDGWZQA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351818645 (113h30m14.903289106s) [noise=true , address=145.79.227.14:4840 , piece_id=3DEHNGEWEG4GA5YTLKGDV4BYUJ7Z6KCVLM343ZB5MYZWEMOASS7A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352062729 (113h30m14.903533191s) [noise=true , address=45.91.116.13:4154 , piece_id=DSC3N7KGSUSEVCEII65IQMYLHEUC5U52NNSREC32Y3Q3MDNKWREQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352224859 (113h30m14.90369532s) [noise=true , address=77.90.31.18:4807 , piece_id=TD3XBJUZ6MM5CB7NMYEIZWQC63U2PZZJPYHIM3OOU7PE3Y7KGCIQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352479703 (113h30m14.903950165s) [noise=true , address=77.90.31.17:4771 , piece_id=ARX2MBQVJSPWKWS2AHE2QBBWNREGMUORBQPNFQJHJDWBT2FCDPAA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352614791 (113h30m14.904085342s) [noise=true , address=145.79.227.21:4473 , piece_id=DIGA6L2TCAOGHAGUY5K4T3R7H66CF3JLXBXKMC7TPTXVKBCLDJBA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352904944 (113h30m14.904375405s) [noise=true , address=45.91.116.6:4973 , piece_id=NEGS6AAGBP4WRJJ6TXEBNHCV2FXW2OBMRWAWL4RNY4YZZZSI3ZLQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.353329152 (113h30m14.904799613s) [noise=true , address=77.90.31.12:4592 , piece_id=ET6LSZ6LXAAAPVJLBLL4UOIF6SKOKDCKXEXKIQ4OS3RD5J5QB2UQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.353685219 (113h30m14.90515568s) [noise=true , address=145.79.227.16:4912 , piece_id=SKK6OIUPSWXXD6QS5ZCVVORVSFXXOE5UJH7POHTKYY6PU4LX2JGA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354114446 (113h30m14.905584907s) [noise=true , address=45.91.116.12:4225 , piece_id=RMDFCX4JGDRPQ46PPTYAWJE7NIYVDE3J2LCKREBFXHR7I6U2G52A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354334907 (113h30m14.905805368s) [noise=true , address=145.79.227.13:4804 , piece_id=NZL7G7WFABHCYGW63IPFVWP45AMPZRDYLDF4JNKJHK5PLSHYHLSQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354804771 (113h30m14.906275222s) [noise=true , address=145.79.227.20:4439 , piece_id=FEDAGRFBHNL6ES3XTCRWPEI4NN2CMMDJIVNW5WOG3KYD7VXFXA7Q, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:18.090813414 (113h30m16.642283865s) [upload=99885]: Next returned nil segment Nov 14 01:34:18.090832882 (113h30m16.642303333s) [upload=99885]: Waiting for error group managing segments... Nov 14 01:34:18.376699271 (113h30m16.928169732s) [noise=true , address=45.91.116.12:4225 , piece_id=RMDFCX4JGDRPQ46PPTYAWJE7NIYVDE3J2LCKREBFXHR7I6U2G52A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.449067032 (113h30m17.000537493s) [noise=true , address=77.90.31.18:4807 , piece_id=TD3XBJUZ6MM5CB7NMYEIZWQC63U2PZZJPYHIM3OOU7PE3Y7KGCIQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.476540927 (113h30m17.028011378s) [noise=true , address=77.90.31.13:4628 , piece_id=WHP562KP7DABPK6XQT5F43QGNNCDFXYYYP775VGZLXIG2UEWKK4A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.502738892 (113h30m17.054209343s) [noise=true , address=77.90.31.12:4592 , piece_id=ET6LSZ6LXAAAPVJLBLL4UOIF6SKOKDCKXEXKIQ4OS3RD5J5QB2UQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.504520516 (113h30m17.055990977s) [noise=true , address=45.91.116.6:4973 , piece_id=NEGS6AAGBP4WRJJ6TXEBNHCV2FXW2OBMRWAWL4RNY4YZZZSI3ZLQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.504778116 (113h30m17.056248577s) [noise=true , address=45.91.116.13:4154 , piece_id=DSC3N7KGSUSEVCEII65IQMYLHEUC5U52NNSREC32Y3Q3MDNKWREQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.511639541 (113h30m17.063110002s) [noise=true , address=77.90.31.10:4517 , piece_id=GDOCT6DE4NL56DWKOS3JNSQVIVE342SDIZ5YR7XPNRXYPOQYDXCQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.529697226 (113h30m17.081167687s) [noise=true , address=77.90.31.17:4771 , piece_id=ARX2MBQVJSPWKWS2AHE2QBBWNREGMUORBQPNFQJHJDWBT2FCDPAA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:45.367038063 (113h30m43.918508514s) [noise=true , address=145.79.227.26:4653 , piece_id=EEKXCCTYRNCRJFVM2QBQVQADQJ2YVDCDOXFIFUP4BYUEVYDGWZQA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection Nov 14 01:34:45.367162339 (113h30m43.91863279s) [noise=true , address=145.79.227.14:4840 , piece_id=3DEHNGEWEG4GA5YTLKGDV4BYUJ7Z6KCVLM343ZB5MYZWEMOASS7A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection Nov 14 01:34:45.367201353 (113h30m43.918671804s) [noise=true , address=145.79.227.9:4366 , piece_id=GXQSDBFPFZPZ352A5KBNQJS5A32C7ZO522AHCHO3A27P4LAMUFZQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection Nov 14 01:34:45.367247291 (113h30m43.918717742s) [noise=true , address=145.79.227.20:4439 , piece_id=FEDAGRFBHNL6ES3XTCRWPEI4NN2CMMDJIVNW5WOG3KYD7VXFXA7Q, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection; write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection Nov 14 01:34:45.367425070 (113h30m43.918895531s) [noise=true , address=145.79.227.13:4804 , piece_id=NZL7G7WFABHCYGW63IPFVWP45AMPZRDYLDF4JNKJHK5PLSHYHLSQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection Nov 14 01:34:45.367779144 (113h30m43.919249605s) [noise=true , address=145.79.227.30:4763 , piece_id=IW3HMSEGIX2TFUY277OSEF4BULPDWA3AO3MNJ7OCUULORPLHNWOQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:50566->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:34:45.367825933 (113h30m43.919296394s) [noise=true , address=145.79.227.16:4912 , piece_id=SKK6OIUPSWXXD6QS5ZCVVORVSFXXOE5UJH7POHTKYY6PU4LX2JGA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection Nov 14 01:34:45.368025623 (113h30m43.919496114s) [noise=true , address=145.79.227.21:4473 , piece_id=DIGA6L2TCAOGHAGUY5K4T3R7H66CF3JLXBXKMC7TPTXVKBCLDJBA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: read tcp 188.166.225.170:49576->145.79.227.21:4473: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:34:45.368068755 (113h30m43.919539206s) [seg_pos={0 0}, upload=99885]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 14 01:34:45.634359530 (113h30m44.185829991s) [upload=99885]: Done uploading segments... Nov 14 01:34:45.634431096 (113h30m44.185901557s) [upload=99885]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.855445094s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.............X.] duration: 348.976096ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/1fc7c26ef222e679 FAIL (2025-11-14 01:50:49 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:53548->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 349.002195ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X.............X...................X......................] duration: 6.066668973s trace: https://jaeger.dev.storj.io/trace/4377cd58781b9764 FAIL (2025-11-14 01:50:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== DXCQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351147808 (113h30m14.902618269s) [noise=true , address=77.90.31.13:4628 , piece_id=WHP562KP7DABPK6XQT5F43QGNNCDFXYYYP775VGZLXIG2UEWKK4A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351381713 (113h30m14.902852174s) [noise=true , address=145.79.227.30:4763 , piece_id=IW3HMSEGIX2TFUY277OSEF4BULPDWA3AO3MNJ7OCUULORPLHNWOQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351673048 (113h30m14.903143509s) [noise=true , address=145.79.227.26:4653 , piece_id=EEKXCCTYRNCRJFVM2QBQVQADQJ2YVDCDOXFIFUP4BYUEVYDGWZQA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351818645 (113h30m14.903289106s) [noise=true , address=145.79.227.14:4840 , piece_id=3DEHNGEWEG4GA5YTLKGDV4BYUJ7Z6KCVLM343ZB5MYZWEMOASS7A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352062729 (113h30m14.903533191s) [noise=true , address=45.91.116.13:4154 , piece_id=DSC3N7KGSUSEVCEII65IQMYLHEUC5U52NNSREC32Y3Q3MDNKWREQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352224859 (113h30m14.90369532s) [noise=true , address=77.90.31.18:4807 , piece_id=TD3XBJUZ6MM5CB7NMYEIZWQC63U2PZZJPYHIM3OOU7PE3Y7KGCIQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352479703 (113h30m14.903950165s) [noise=true , address=77.90.31.17:4771 , piece_id=ARX2MBQVJSPWKWS2AHE2QBBWNREGMUORBQPNFQJHJDWBT2FCDPAA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352614791 (113h30m14.904085342s) [noise=true , address=145.79.227.21:4473 , piece_id=DIGA6L2TCAOGHAGUY5K4T3R7H66CF3JLXBXKMC7TPTXVKBCLDJBA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352904944 (113h30m14.904375405s) [noise=true , address=45.91.116.6:4973 , piece_id=NEGS6AAGBP4WRJJ6TXEBNHCV2FXW2OBMRWAWL4RNY4YZZZSI3ZLQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.353329152 (113h30m14.904799613s) [noise=true , address=77.90.31.12:4592 , piece_id=ET6LSZ6LXAAAPVJLBLL4UOIF6SKOKDCKXEXKIQ4OS3RD5J5QB2UQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.353685219 (113h30m14.90515568s) [noise=true , address=145.79.227.16:4912 , piece_id=SKK6OIUPSWXXD6QS5ZCVVORVSFXXOE5UJH7POHTKYY6PU4LX2JGA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354114446 (113h30m14.905584907s) [noise=true , address=45.91.116.12:4225 , piece_id=RMDFCX4JGDRPQ46PPTYAWJE7NIYVDE3J2LCKREBFXHR7I6U2G52A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354334907 (113h30m14.905805368s) [noise=true , address=145.79.227.13:4804 , piece_id=NZL7G7WFABHCYGW63IPFVWP45AMPZRDYLDF4JNKJHK5PLSHYHLSQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354804771 (113h30m14.906275222s) [noise=true , address=145.79.227.20:4439 , piece_id=FEDAGRFBHNL6ES3XTCRWPEI4NN2CMMDJIVNW5WOG3KYD7VXFXA7Q, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:18.090813414 (113h30m16.642283865s) [upload=99885]: Next returned nil segment Nov 14 01:34:18.090832882 (113h30m16.642303333s) [upload=99885]: Waiting for error group managing segments... Nov 14 01:34:18.376699271 (113h30m16.928169732s) [noise=true , address=45.91.116.12:4225 , piece_id=RMDFCX4JGDRPQ46PPTYAWJE7NIYVDE3J2LCKREBFXHR7I6U2G52A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.449067032 (113h30m17.000537493s) [noise=true , address=77.90.31.18:4807 , piece_id=TD3XBJUZ6MM5CB7NMYEIZWQC63U2PZZJPYHIM3OOU7PE3Y7KGCIQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.476540927 (113h30m17.028011378s) [noise=true , address=77.90.31.13:4628 , piece_id=WHP562KP7DABPK6XQT5F43QGNNCDFXYYYP775VGZLXIG2UEWKK4A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.502738892 (113h30m17.054209343s) [noise=true , address=77.90.31.12:4592 , piece_id=ET6LSZ6LXAAAPVJLBLL4UOIF6SKOKDCKXEXKIQ4OS3RD5J5QB2UQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.504520516 (113h30m17.055990977s) [noise=true , address=45.91.116.6:4973 , piece_id=NEGS6AAGBP4WRJJ6TXEBNHCV2FXW2OBMRWAWL4RNY4YZZZSI3ZLQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.504778116 (113h30m17.056248577s) [noise=true , address=45.91.116.13:4154 , piece_id=DSC3N7KGSUSEVCEII65IQMYLHEUC5U52NNSREC32Y3Q3MDNKWREQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.511639541 (113h30m17.063110002s) [noise=true , address=77.90.31.10:4517 , piece_id=GDOCT6DE4NL56DWKOS3JNSQVIVE342SDIZ5YR7XPNRXYPOQYDXCQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.529697226 (113h30m17.081167687s) [noise=true , address=77.90.31.17:4771 , piece_id=ARX2MBQVJSPWKWS2AHE2QBBWNREGMUORBQPNFQJHJDWBT2FCDPAA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:45.367038063 (113h30m43.918508514s) [noise=true , address=145.79.227.26:4653 , piece_id=EEKXCCTYRNCRJFVM2QBQVQADQJ2YVDCDOXFIFUP4BYUEVYDGWZQA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection Nov 14 01:34:45.367162339 (113h30m43.91863279s) [noise=true , address=145.79.227.14:4840 , piece_id=3DEHNGEWEG4GA5YTLKGDV4BYUJ7Z6KCVLM343ZB5MYZWEMOASS7A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection Nov 14 01:34:45.367201353 (113h30m43.918671804s) [noise=true , address=145.79.227.9:4366 , piece_id=GXQSDBFPFZPZ352A5KBNQJS5A32C7ZO522AHCHO3A27P4LAMUFZQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection Nov 14 01:34:45.367247291 (113h30m43.918717742s) [noise=true , address=145.79.227.20:4439 , piece_id=FEDAGRFBHNL6ES3XTCRWPEI4NN2CMMDJIVNW5WOG3KYD7VXFXA7Q, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection; write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection Nov 14 01:34:45.367425070 (113h30m43.918895531s) [noise=true , address=145.79.227.13:4804 , piece_id=NZL7G7WFABHCYGW63IPFVWP45AMPZRDYLDF4JNKJHK5PLSHYHLSQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection Nov 14 01:34:45.367779144 (113h30m43.919249605s) [noise=true , address=145.79.227.30:4763 , piece_id=IW3HMSEGIX2TFUY277OSEF4BULPDWA3AO3MNJ7OCUULORPLHNWOQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:50566->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:34:45.367825933 (113h30m43.919296394s) [noise=true , address=145.79.227.16:4912 , piece_id=SKK6OIUPSWXXD6QS5ZCVVORVSFXXOE5UJH7POHTKYY6PU4LX2JGA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection Nov 14 01:34:45.368025623 (113h30m43.919496114s) [noise=true , address=145.79.227.21:4473 , piece_id=DIGA6L2TCAOGHAGUY5K4T3R7H66CF3JLXBXKMC7TPTXVKBCLDJBA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: read tcp 188.166.225.170:49576->145.79.227.21:4473: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:34:45.368068755 (113h30m43.919539206s) [seg_pos={0 0}, upload=99885]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 14 01:34:45.634359530 (113h30m44.185829991s) [upload=99885]: Done uploading segments... Nov 14 01:34:45.634431096 (113h30m44.185901557s) [upload=99885]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.066700634s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.823307244s trace: https://jaeger.dev.storj.io/trace/5bc86ccfe33cf550 FAIL (2025-11-14 01:50:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ce_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288397607 (113h30m2.839868078s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288507285 (113h30m2.839977746s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.289077772 (113h30m2.840548233s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290074169 (113h30m2.84154464s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290622854 (113h30m2.842093345s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292062385 (113h30m2.843532866s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292958642 (113h30m2.844429123s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.294563758 (113h30m2.846034229s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.296755000 (113h30m2.848225462s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.298618379 (113h30m2.85008884s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.300188338 (113h30m2.85165881s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.301125483 (113h30m2.852595944s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302545266 (113h30m2.854015727s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302898339 (113h30m2.8543688s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.303825163 (113h30m2.855295624s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.316046324 (113h30m2.867516785s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.318180529 (113h30m2.86965099s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:05.581380111 (113h30m4.132850592s) [upload=99881]: Next returned nil segment Nov 14 01:34:05.581407382 (113h30m4.132877844s) [upload=99881]: Waiting for error group managing segments... Nov 14 01:34:05.921914348 (113h30m4.473384809s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.005586503 (113h30m4.557056964s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.026850744 (113h30m4.578321205s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.121926626 (113h30m4.673397087s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.235634986 (113h30m4.787105437s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.277897963 (113h30m4.829368424s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.283628915 (113h30m4.835099376s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.287808744 (113h30m4.839279205s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.300041498 (113h30m4.851511949s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.304400718 (113h30m4.855871179s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.325040680 (113h30m4.876511141s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.331007762 (113h30m4.882478223s) [noise=true , address=77.90.31.12:4611 , piece_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.481058726 (113h30m5.032529207s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:35:02.598238012 (113h31m1.149708493s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection Nov 14 01:35:02.599889276 (113h31m1.151359737s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:145.79.227.52:4455): protocol: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection Nov 14 01:35:02.600041686 (113h31m1.151512147s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:127WWD5hr1ZBkRLCHz3jb7kpqqnxPS6VuZ1MyQDmitFbhTSbJNK, address:145.79.227.49:4344): protocol: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection Nov 14 01:35:02.600244733 (113h31m1.151715204s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:145.79.227.35:4488): protocol: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection Nov 14 01:35:02.600569551 (113h31m1.152040022s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12h6MND3rBadG4EjUP18S7izyj23cVBLaB5drFsuNGhcxwAYXSA, address:145.79.227.17:4983): protocol: read tcp 188.166.225.170:33042->145.79.227.17:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:35:02.600603536 (113h31m1.152073997s) [seg_pos={0 0}, upload=99881]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 01:35:02.847367167 (113h31m1.398837648s) [upload=99881]: Done uploading segments... Nov 14 01:35:02.847427322 (113h31m1.398897783s) [upload=99881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.823335589s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X.............X...................X.....................] duration: 5.933194511s trace: https://jaeger.dev.storj.io/trace/4377cd58781b9764 FAIL (2025-11-14 01:50:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== DXCQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351147808 (113h30m14.902618269s) [noise=true , address=77.90.31.13:4628 , piece_id=WHP562KP7DABPK6XQT5F43QGNNCDFXYYYP775VGZLXIG2UEWKK4A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351381713 (113h30m14.902852174s) [noise=true , address=145.79.227.30:4763 , piece_id=IW3HMSEGIX2TFUY277OSEF4BULPDWA3AO3MNJ7OCUULORPLHNWOQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351673048 (113h30m14.903143509s) [noise=true , address=145.79.227.26:4653 , piece_id=EEKXCCTYRNCRJFVM2QBQVQADQJ2YVDCDOXFIFUP4BYUEVYDGWZQA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351818645 (113h30m14.903289106s) [noise=true , address=145.79.227.14:4840 , piece_id=3DEHNGEWEG4GA5YTLKGDV4BYUJ7Z6KCVLM343ZB5MYZWEMOASS7A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352062729 (113h30m14.903533191s) [noise=true , address=45.91.116.13:4154 , piece_id=DSC3N7KGSUSEVCEII65IQMYLHEUC5U52NNSREC32Y3Q3MDNKWREQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352224859 (113h30m14.90369532s) [noise=true , address=77.90.31.18:4807 , piece_id=TD3XBJUZ6MM5CB7NMYEIZWQC63U2PZZJPYHIM3OOU7PE3Y7KGCIQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352479703 (113h30m14.903950165s) [noise=true , address=77.90.31.17:4771 , piece_id=ARX2MBQVJSPWKWS2AHE2QBBWNREGMUORBQPNFQJHJDWBT2FCDPAA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352614791 (113h30m14.904085342s) [noise=true , address=145.79.227.21:4473 , piece_id=DIGA6L2TCAOGHAGUY5K4T3R7H66CF3JLXBXKMC7TPTXVKBCLDJBA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352904944 (113h30m14.904375405s) [noise=true , address=45.91.116.6:4973 , piece_id=NEGS6AAGBP4WRJJ6TXEBNHCV2FXW2OBMRWAWL4RNY4YZZZSI3ZLQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.353329152 (113h30m14.904799613s) [noise=true , address=77.90.31.12:4592 , piece_id=ET6LSZ6LXAAAPVJLBLL4UOIF6SKOKDCKXEXKIQ4OS3RD5J5QB2UQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.353685219 (113h30m14.90515568s) [noise=true , address=145.79.227.16:4912 , piece_id=SKK6OIUPSWXXD6QS5ZCVVORVSFXXOE5UJH7POHTKYY6PU4LX2JGA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354114446 (113h30m14.905584907s) [noise=true , address=45.91.116.12:4225 , piece_id=RMDFCX4JGDRPQ46PPTYAWJE7NIYVDE3J2LCKREBFXHR7I6U2G52A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354334907 (113h30m14.905805368s) [noise=true , address=145.79.227.13:4804 , piece_id=NZL7G7WFABHCYGW63IPFVWP45AMPZRDYLDF4JNKJHK5PLSHYHLSQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354804771 (113h30m14.906275222s) [noise=true , address=145.79.227.20:4439 , piece_id=FEDAGRFBHNL6ES3XTCRWPEI4NN2CMMDJIVNW5WOG3KYD7VXFXA7Q, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:18.090813414 (113h30m16.642283865s) [upload=99885]: Next returned nil segment Nov 14 01:34:18.090832882 (113h30m16.642303333s) [upload=99885]: Waiting for error group managing segments... Nov 14 01:34:18.376699271 (113h30m16.928169732s) [noise=true , address=45.91.116.12:4225 , piece_id=RMDFCX4JGDRPQ46PPTYAWJE7NIYVDE3J2LCKREBFXHR7I6U2G52A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.449067032 (113h30m17.000537493s) [noise=true , address=77.90.31.18:4807 , piece_id=TD3XBJUZ6MM5CB7NMYEIZWQC63U2PZZJPYHIM3OOU7PE3Y7KGCIQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.476540927 (113h30m17.028011378s) [noise=true , address=77.90.31.13:4628 , piece_id=WHP562KP7DABPK6XQT5F43QGNNCDFXYYYP775VGZLXIG2UEWKK4A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.502738892 (113h30m17.054209343s) [noise=true , address=77.90.31.12:4592 , piece_id=ET6LSZ6LXAAAPVJLBLL4UOIF6SKOKDCKXEXKIQ4OS3RD5J5QB2UQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.504520516 (113h30m17.055990977s) [noise=true , address=45.91.116.6:4973 , piece_id=NEGS6AAGBP4WRJJ6TXEBNHCV2FXW2OBMRWAWL4RNY4YZZZSI3ZLQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.504778116 (113h30m17.056248577s) [noise=true , address=45.91.116.13:4154 , piece_id=DSC3N7KGSUSEVCEII65IQMYLHEUC5U52NNSREC32Y3Q3MDNKWREQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.511639541 (113h30m17.063110002s) [noise=true , address=77.90.31.10:4517 , piece_id=GDOCT6DE4NL56DWKOS3JNSQVIVE342SDIZ5YR7XPNRXYPOQYDXCQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.529697226 (113h30m17.081167687s) [noise=true , address=77.90.31.17:4771 , piece_id=ARX2MBQVJSPWKWS2AHE2QBBWNREGMUORBQPNFQJHJDWBT2FCDPAA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:45.367038063 (113h30m43.918508514s) [noise=true , address=145.79.227.26:4653 , piece_id=EEKXCCTYRNCRJFVM2QBQVQADQJ2YVDCDOXFIFUP4BYUEVYDGWZQA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection Nov 14 01:34:45.367162339 (113h30m43.91863279s) [noise=true , address=145.79.227.14:4840 , piece_id=3DEHNGEWEG4GA5YTLKGDV4BYUJ7Z6KCVLM343ZB5MYZWEMOASS7A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection Nov 14 01:34:45.367201353 (113h30m43.918671804s) [noise=true , address=145.79.227.9:4366 , piece_id=GXQSDBFPFZPZ352A5KBNQJS5A32C7ZO522AHCHO3A27P4LAMUFZQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection Nov 14 01:34:45.367247291 (113h30m43.918717742s) [noise=true , address=145.79.227.20:4439 , piece_id=FEDAGRFBHNL6ES3XTCRWPEI4NN2CMMDJIVNW5WOG3KYD7VXFXA7Q, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection; write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection Nov 14 01:34:45.367425070 (113h30m43.918895531s) [noise=true , address=145.79.227.13:4804 , piece_id=NZL7G7WFABHCYGW63IPFVWP45AMPZRDYLDF4JNKJHK5PLSHYHLSQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection Nov 14 01:34:45.367779144 (113h30m43.919249605s) [noise=true , address=145.79.227.30:4763 , piece_id=IW3HMSEGIX2TFUY277OSEF4BULPDWA3AO3MNJ7OCUULORPLHNWOQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:50566->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:34:45.367825933 (113h30m43.919296394s) [noise=true , address=145.79.227.16:4912 , piece_id=SKK6OIUPSWXXD6QS5ZCVVORVSFXXOE5UJH7POHTKYY6PU4LX2JGA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection Nov 14 01:34:45.368025623 (113h30m43.919496114s) [noise=true , address=145.79.227.21:4473 , piece_id=DIGA6L2TCAOGHAGUY5K4T3R7H66CF3JLXBXKMC7TPTXVKBCLDJBA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: read tcp 188.166.225.170:49576->145.79.227.21:4473: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:34:45.368068755 (113h30m43.919539206s) [seg_pos={0 0}, upload=99885]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 14 01:34:45.634359530 (113h30m44.185829991s) [upload=99885]: Done uploading segments... Nov 14 01:34:45.634431096 (113h30m44.185901557s) [upload=99885]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.933220661s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [..............X] duration: 435.936173ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/1fc7c26ef222e679 FAIL (2025-11-14 01:50:16 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:53548->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 435.965019ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 2 out of 15 is failed (threshold=6) [............X.X] duration: 223.21296ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/8a3ce894c9564a5 FAIL (2025-11-14 01:49:59 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:43336->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 223.241974ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.523681684s trace: https://jaeger.dev.storj.io/trace/5bc86ccfe33cf550 FAIL (2025-11-14 01:49:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ce_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288397607 (113h30m2.839868078s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288507285 (113h30m2.839977746s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.289077772 (113h30m2.840548233s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290074169 (113h30m2.84154464s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290622854 (113h30m2.842093345s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292062385 (113h30m2.843532866s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292958642 (113h30m2.844429123s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.294563758 (113h30m2.846034229s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.296755000 (113h30m2.848225462s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.298618379 (113h30m2.85008884s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.300188338 (113h30m2.85165881s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.301125483 (113h30m2.852595944s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302545266 (113h30m2.854015727s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302898339 (113h30m2.8543688s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.303825163 (113h30m2.855295624s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.316046324 (113h30m2.867516785s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.318180529 (113h30m2.86965099s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:05.581380111 (113h30m4.132850592s) [upload=99881]: Next returned nil segment Nov 14 01:34:05.581407382 (113h30m4.132877844s) [upload=99881]: Waiting for error group managing segments... Nov 14 01:34:05.921914348 (113h30m4.473384809s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.005586503 (113h30m4.557056964s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.026850744 (113h30m4.578321205s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.121926626 (113h30m4.673397087s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.235634986 (113h30m4.787105437s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.277897963 (113h30m4.829368424s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.283628915 (113h30m4.835099376s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.287808744 (113h30m4.839279205s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.300041498 (113h30m4.851511949s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.304400718 (113h30m4.855871179s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.325040680 (113h30m4.876511141s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.331007762 (113h30m4.882478223s) [noise=true , address=77.90.31.12:4611 , piece_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.481058726 (113h30m5.032529207s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:35:02.598238012 (113h31m1.149708493s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection Nov 14 01:35:02.599889276 (113h31m1.151359737s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:145.79.227.52:4455): protocol: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection Nov 14 01:35:02.600041686 (113h31m1.151512147s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:127WWD5hr1ZBkRLCHz3jb7kpqqnxPS6VuZ1MyQDmitFbhTSbJNK, address:145.79.227.49:4344): protocol: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection Nov 14 01:35:02.600244733 (113h31m1.151715204s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:145.79.227.35:4488): protocol: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection Nov 14 01:35:02.600569551 (113h31m1.152040022s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12h6MND3rBadG4EjUP18S7izyj23cVBLaB5drFsuNGhcxwAYXSA, address:145.79.227.17:4983): protocol: read tcp 188.166.225.170:33042->145.79.227.17:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:35:02.600603536 (113h31m1.152073997s) [seg_pos={0 0}, upload=99881]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 01:35:02.847367167 (113h31m1.398837648s) [upload=99881]: Done uploading segments... Nov 14 01:35:02.847427322 (113h31m1.398897783s) [upload=99881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.523702544s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X.............X...................X....................] duration: 6.056521089s trace: https://jaeger.dev.storj.io/trace/4377cd58781b9764 FAIL (2025-11-14 01:49:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== DXCQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351147808 (113h30m14.902618269s) [noise=true , address=77.90.31.13:4628 , piece_id=WHP562KP7DABPK6XQT5F43QGNNCDFXYYYP775VGZLXIG2UEWKK4A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351381713 (113h30m14.902852174s) [noise=true , address=145.79.227.30:4763 , piece_id=IW3HMSEGIX2TFUY277OSEF4BULPDWA3AO3MNJ7OCUULORPLHNWOQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351673048 (113h30m14.903143509s) [noise=true , address=145.79.227.26:4653 , piece_id=EEKXCCTYRNCRJFVM2QBQVQADQJ2YVDCDOXFIFUP4BYUEVYDGWZQA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351818645 (113h30m14.903289106s) [noise=true , address=145.79.227.14:4840 , piece_id=3DEHNGEWEG4GA5YTLKGDV4BYUJ7Z6KCVLM343ZB5MYZWEMOASS7A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352062729 (113h30m14.903533191s) [noise=true , address=45.91.116.13:4154 , piece_id=DSC3N7KGSUSEVCEII65IQMYLHEUC5U52NNSREC32Y3Q3MDNKWREQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352224859 (113h30m14.90369532s) [noise=true , address=77.90.31.18:4807 , piece_id=TD3XBJUZ6MM5CB7NMYEIZWQC63U2PZZJPYHIM3OOU7PE3Y7KGCIQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352479703 (113h30m14.903950165s) [noise=true , address=77.90.31.17:4771 , piece_id=ARX2MBQVJSPWKWS2AHE2QBBWNREGMUORBQPNFQJHJDWBT2FCDPAA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352614791 (113h30m14.904085342s) [noise=true , address=145.79.227.21:4473 , piece_id=DIGA6L2TCAOGHAGUY5K4T3R7H66CF3JLXBXKMC7TPTXVKBCLDJBA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352904944 (113h30m14.904375405s) [noise=true , address=45.91.116.6:4973 , piece_id=NEGS6AAGBP4WRJJ6TXEBNHCV2FXW2OBMRWAWL4RNY4YZZZSI3ZLQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.353329152 (113h30m14.904799613s) [noise=true , address=77.90.31.12:4592 , piece_id=ET6LSZ6LXAAAPVJLBLL4UOIF6SKOKDCKXEXKIQ4OS3RD5J5QB2UQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.353685219 (113h30m14.90515568s) [noise=true , address=145.79.227.16:4912 , piece_id=SKK6OIUPSWXXD6QS5ZCVVORVSFXXOE5UJH7POHTKYY6PU4LX2JGA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354114446 (113h30m14.905584907s) [noise=true , address=45.91.116.12:4225 , piece_id=RMDFCX4JGDRPQ46PPTYAWJE7NIYVDE3J2LCKREBFXHR7I6U2G52A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354334907 (113h30m14.905805368s) [noise=true , address=145.79.227.13:4804 , piece_id=NZL7G7WFABHCYGW63IPFVWP45AMPZRDYLDF4JNKJHK5PLSHYHLSQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354804771 (113h30m14.906275222s) [noise=true , address=145.79.227.20:4439 , piece_id=FEDAGRFBHNL6ES3XTCRWPEI4NN2CMMDJIVNW5WOG3KYD7VXFXA7Q, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:18.090813414 (113h30m16.642283865s) [upload=99885]: Next returned nil segment Nov 14 01:34:18.090832882 (113h30m16.642303333s) [upload=99885]: Waiting for error group managing segments... Nov 14 01:34:18.376699271 (113h30m16.928169732s) [noise=true , address=45.91.116.12:4225 , piece_id=RMDFCX4JGDRPQ46PPTYAWJE7NIYVDE3J2LCKREBFXHR7I6U2G52A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.449067032 (113h30m17.000537493s) [noise=true , address=77.90.31.18:4807 , piece_id=TD3XBJUZ6MM5CB7NMYEIZWQC63U2PZZJPYHIM3OOU7PE3Y7KGCIQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.476540927 (113h30m17.028011378s) [noise=true , address=77.90.31.13:4628 , piece_id=WHP562KP7DABPK6XQT5F43QGNNCDFXYYYP775VGZLXIG2UEWKK4A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.502738892 (113h30m17.054209343s) [noise=true , address=77.90.31.12:4592 , piece_id=ET6LSZ6LXAAAPVJLBLL4UOIF6SKOKDCKXEXKIQ4OS3RD5J5QB2UQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.504520516 (113h30m17.055990977s) [noise=true , address=45.91.116.6:4973 , piece_id=NEGS6AAGBP4WRJJ6TXEBNHCV2FXW2OBMRWAWL4RNY4YZZZSI3ZLQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.504778116 (113h30m17.056248577s) [noise=true , address=45.91.116.13:4154 , piece_id=DSC3N7KGSUSEVCEII65IQMYLHEUC5U52NNSREC32Y3Q3MDNKWREQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.511639541 (113h30m17.063110002s) [noise=true , address=77.90.31.10:4517 , piece_id=GDOCT6DE4NL56DWKOS3JNSQVIVE342SDIZ5YR7XPNRXYPOQYDXCQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.529697226 (113h30m17.081167687s) [noise=true , address=77.90.31.17:4771 , piece_id=ARX2MBQVJSPWKWS2AHE2QBBWNREGMUORBQPNFQJHJDWBT2FCDPAA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:45.367038063 (113h30m43.918508514s) [noise=true , address=145.79.227.26:4653 , piece_id=EEKXCCTYRNCRJFVM2QBQVQADQJ2YVDCDOXFIFUP4BYUEVYDGWZQA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection Nov 14 01:34:45.367162339 (113h30m43.91863279s) [noise=true , address=145.79.227.14:4840 , piece_id=3DEHNGEWEG4GA5YTLKGDV4BYUJ7Z6KCVLM343ZB5MYZWEMOASS7A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection Nov 14 01:34:45.367201353 (113h30m43.918671804s) [noise=true , address=145.79.227.9:4366 , piece_id=GXQSDBFPFZPZ352A5KBNQJS5A32C7ZO522AHCHO3A27P4LAMUFZQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection Nov 14 01:34:45.367247291 (113h30m43.918717742s) [noise=true , address=145.79.227.20:4439 , piece_id=FEDAGRFBHNL6ES3XTCRWPEI4NN2CMMDJIVNW5WOG3KYD7VXFXA7Q, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection; write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection Nov 14 01:34:45.367425070 (113h30m43.918895531s) [noise=true , address=145.79.227.13:4804 , piece_id=NZL7G7WFABHCYGW63IPFVWP45AMPZRDYLDF4JNKJHK5PLSHYHLSQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection Nov 14 01:34:45.367779144 (113h30m43.919249605s) [noise=true , address=145.79.227.30:4763 , piece_id=IW3HMSEGIX2TFUY277OSEF4BULPDWA3AO3MNJ7OCUULORPLHNWOQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:50566->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:34:45.367825933 (113h30m43.919296394s) [noise=true , address=145.79.227.16:4912 , piece_id=SKK6OIUPSWXXD6QS5ZCVVORVSFXXOE5UJH7POHTKYY6PU4LX2JGA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection Nov 14 01:34:45.368025623 (113h30m43.919496114s) [noise=true , address=145.79.227.21:4473 , piece_id=DIGA6L2TCAOGHAGUY5K4T3R7H66CF3JLXBXKMC7TPTXVKBCLDJBA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: read tcp 188.166.225.170:49576->145.79.227.21:4473: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:34:45.368068755 (113h30m43.919539206s) [seg_pos={0 0}, upload=99885]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 14 01:34:45.634359530 (113h30m44.185829991s) [upload=99885]: Done uploading segments... Nov 14 01:34:45.634431096 (113h30m44.185901557s) [upload=99885]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.056654854s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.............X.] duration: 48.868284ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/63134d41945be4fd FAIL (2025-11-14 01:49:16 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:37376->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 48.893923ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.893341559s trace: https://jaeger.dev.storj.io/trace/5bc86ccfe33cf550 FAIL (2025-11-14 01:49:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ce_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288397607 (113h30m2.839868078s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288507285 (113h30m2.839977746s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.289077772 (113h30m2.840548233s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290074169 (113h30m2.84154464s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290622854 (113h30m2.842093345s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292062385 (113h30m2.843532866s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292958642 (113h30m2.844429123s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.294563758 (113h30m2.846034229s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.296755000 (113h30m2.848225462s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.298618379 (113h30m2.85008884s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.300188338 (113h30m2.85165881s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.301125483 (113h30m2.852595944s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302545266 (113h30m2.854015727s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302898339 (113h30m2.8543688s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.303825163 (113h30m2.855295624s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.316046324 (113h30m2.867516785s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.318180529 (113h30m2.86965099s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:05.581380111 (113h30m4.132850592s) [upload=99881]: Next returned nil segment Nov 14 01:34:05.581407382 (113h30m4.132877844s) [upload=99881]: Waiting for error group managing segments... Nov 14 01:34:05.921914348 (113h30m4.473384809s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.005586503 (113h30m4.557056964s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.026850744 (113h30m4.578321205s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.121926626 (113h30m4.673397087s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.235634986 (113h30m4.787105437s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.277897963 (113h30m4.829368424s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.283628915 (113h30m4.835099376s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.287808744 (113h30m4.839279205s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.300041498 (113h30m4.851511949s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.304400718 (113h30m4.855871179s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.325040680 (113h30m4.876511141s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.331007762 (113h30m4.882478223s) [noise=true , address=77.90.31.12:4611 , piece_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.481058726 (113h30m5.032529207s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:35:02.598238012 (113h31m1.149708493s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection Nov 14 01:35:02.599889276 (113h31m1.151359737s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:145.79.227.52:4455): protocol: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection Nov 14 01:35:02.600041686 (113h31m1.151512147s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:127WWD5hr1ZBkRLCHz3jb7kpqqnxPS6VuZ1MyQDmitFbhTSbJNK, address:145.79.227.49:4344): protocol: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection Nov 14 01:35:02.600244733 (113h31m1.151715204s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:145.79.227.35:4488): protocol: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection Nov 14 01:35:02.600569551 (113h31m1.152040022s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12h6MND3rBadG4EjUP18S7izyj23cVBLaB5drFsuNGhcxwAYXSA, address:145.79.227.17:4983): protocol: read tcp 188.166.225.170:33042->145.79.227.17:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:35:02.600603536 (113h31m1.152073997s) [seg_pos={0 0}, upload=99881]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 01:35:02.847367167 (113h31m1.398837648s) [upload=99881]: Done uploading segments... Nov 14 01:35:02.847427322 (113h31m1.398897783s) [upload=99881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.893375384s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X.............X...................X...................] duration: 6.148060551s trace: https://jaeger.dev.storj.io/trace/4377cd58781b9764 FAIL (2025-11-14 01:48:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== DXCQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351147808 (113h30m14.902618269s) [noise=true , address=77.90.31.13:4628 , piece_id=WHP562KP7DABPK6XQT5F43QGNNCDFXYYYP775VGZLXIG2UEWKK4A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351381713 (113h30m14.902852174s) [noise=true , address=145.79.227.30:4763 , piece_id=IW3HMSEGIX2TFUY277OSEF4BULPDWA3AO3MNJ7OCUULORPLHNWOQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351673048 (113h30m14.903143509s) [noise=true , address=145.79.227.26:4653 , piece_id=EEKXCCTYRNCRJFVM2QBQVQADQJ2YVDCDOXFIFUP4BYUEVYDGWZQA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351818645 (113h30m14.903289106s) [noise=true , address=145.79.227.14:4840 , piece_id=3DEHNGEWEG4GA5YTLKGDV4BYUJ7Z6KCVLM343ZB5MYZWEMOASS7A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352062729 (113h30m14.903533191s) [noise=true , address=45.91.116.13:4154 , piece_id=DSC3N7KGSUSEVCEII65IQMYLHEUC5U52NNSREC32Y3Q3MDNKWREQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352224859 (113h30m14.90369532s) [noise=true , address=77.90.31.18:4807 , piece_id=TD3XBJUZ6MM5CB7NMYEIZWQC63U2PZZJPYHIM3OOU7PE3Y7KGCIQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352479703 (113h30m14.903950165s) [noise=true , address=77.90.31.17:4771 , piece_id=ARX2MBQVJSPWKWS2AHE2QBBWNREGMUORBQPNFQJHJDWBT2FCDPAA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352614791 (113h30m14.904085342s) [noise=true , address=145.79.227.21:4473 , piece_id=DIGA6L2TCAOGHAGUY5K4T3R7H66CF3JLXBXKMC7TPTXVKBCLDJBA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352904944 (113h30m14.904375405s) [noise=true , address=45.91.116.6:4973 , piece_id=NEGS6AAGBP4WRJJ6TXEBNHCV2FXW2OBMRWAWL4RNY4YZZZSI3ZLQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.353329152 (113h30m14.904799613s) [noise=true , address=77.90.31.12:4592 , piece_id=ET6LSZ6LXAAAPVJLBLL4UOIF6SKOKDCKXEXKIQ4OS3RD5J5QB2UQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.353685219 (113h30m14.90515568s) [noise=true , address=145.79.227.16:4912 , piece_id=SKK6OIUPSWXXD6QS5ZCVVORVSFXXOE5UJH7POHTKYY6PU4LX2JGA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354114446 (113h30m14.905584907s) [noise=true , address=45.91.116.12:4225 , piece_id=RMDFCX4JGDRPQ46PPTYAWJE7NIYVDE3J2LCKREBFXHR7I6U2G52A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354334907 (113h30m14.905805368s) [noise=true , address=145.79.227.13:4804 , piece_id=NZL7G7WFABHCYGW63IPFVWP45AMPZRDYLDF4JNKJHK5PLSHYHLSQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354804771 (113h30m14.906275222s) [noise=true , address=145.79.227.20:4439 , piece_id=FEDAGRFBHNL6ES3XTCRWPEI4NN2CMMDJIVNW5WOG3KYD7VXFXA7Q, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:18.090813414 (113h30m16.642283865s) [upload=99885]: Next returned nil segment Nov 14 01:34:18.090832882 (113h30m16.642303333s) [upload=99885]: Waiting for error group managing segments... Nov 14 01:34:18.376699271 (113h30m16.928169732s) [noise=true , address=45.91.116.12:4225 , piece_id=RMDFCX4JGDRPQ46PPTYAWJE7NIYVDE3J2LCKREBFXHR7I6U2G52A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.449067032 (113h30m17.000537493s) [noise=true , address=77.90.31.18:4807 , piece_id=TD3XBJUZ6MM5CB7NMYEIZWQC63U2PZZJPYHIM3OOU7PE3Y7KGCIQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.476540927 (113h30m17.028011378s) [noise=true , address=77.90.31.13:4628 , piece_id=WHP562KP7DABPK6XQT5F43QGNNCDFXYYYP775VGZLXIG2UEWKK4A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.502738892 (113h30m17.054209343s) [noise=true , address=77.90.31.12:4592 , piece_id=ET6LSZ6LXAAAPVJLBLL4UOIF6SKOKDCKXEXKIQ4OS3RD5J5QB2UQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.504520516 (113h30m17.055990977s) [noise=true , address=45.91.116.6:4973 , piece_id=NEGS6AAGBP4WRJJ6TXEBNHCV2FXW2OBMRWAWL4RNY4YZZZSI3ZLQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.504778116 (113h30m17.056248577s) [noise=true , address=45.91.116.13:4154 , piece_id=DSC3N7KGSUSEVCEII65IQMYLHEUC5U52NNSREC32Y3Q3MDNKWREQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.511639541 (113h30m17.063110002s) [noise=true , address=77.90.31.10:4517 , piece_id=GDOCT6DE4NL56DWKOS3JNSQVIVE342SDIZ5YR7XPNRXYPOQYDXCQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.529697226 (113h30m17.081167687s) [noise=true , address=77.90.31.17:4771 , piece_id=ARX2MBQVJSPWKWS2AHE2QBBWNREGMUORBQPNFQJHJDWBT2FCDPAA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:45.367038063 (113h30m43.918508514s) [noise=true , address=145.79.227.26:4653 , piece_id=EEKXCCTYRNCRJFVM2QBQVQADQJ2YVDCDOXFIFUP4BYUEVYDGWZQA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection Nov 14 01:34:45.367162339 (113h30m43.91863279s) [noise=true , address=145.79.227.14:4840 , piece_id=3DEHNGEWEG4GA5YTLKGDV4BYUJ7Z6KCVLM343ZB5MYZWEMOASS7A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection Nov 14 01:34:45.367201353 (113h30m43.918671804s) [noise=true , address=145.79.227.9:4366 , piece_id=GXQSDBFPFZPZ352A5KBNQJS5A32C7ZO522AHCHO3A27P4LAMUFZQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection Nov 14 01:34:45.367247291 (113h30m43.918717742s) [noise=true , address=145.79.227.20:4439 , piece_id=FEDAGRFBHNL6ES3XTCRWPEI4NN2CMMDJIVNW5WOG3KYD7VXFXA7Q, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection; write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection Nov 14 01:34:45.367425070 (113h30m43.918895531s) [noise=true , address=145.79.227.13:4804 , piece_id=NZL7G7WFABHCYGW63IPFVWP45AMPZRDYLDF4JNKJHK5PLSHYHLSQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection Nov 14 01:34:45.367779144 (113h30m43.919249605s) [noise=true , address=145.79.227.30:4763 , piece_id=IW3HMSEGIX2TFUY277OSEF4BULPDWA3AO3MNJ7OCUULORPLHNWOQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:50566->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:34:45.367825933 (113h30m43.919296394s) [noise=true , address=145.79.227.16:4912 , piece_id=SKK6OIUPSWXXD6QS5ZCVVORVSFXXOE5UJH7POHTKYY6PU4LX2JGA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection Nov 14 01:34:45.368025623 (113h30m43.919496114s) [noise=true , address=145.79.227.21:4473 , piece_id=DIGA6L2TCAOGHAGUY5K4T3R7H66CF3JLXBXKMC7TPTXVKBCLDJBA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: read tcp 188.166.225.170:49576->145.79.227.21:4473: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:34:45.368068755 (113h30m43.919539206s) [seg_pos={0 0}, upload=99885]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 14 01:34:45.634359530 (113h30m44.185829991s) [upload=99885]: Done uploading segments... Nov 14 01:34:45.634431096 (113h30m44.185901557s) [upload=99885]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.148087161s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [..............X] duration: 223.085784ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/63134d41945be4fd FAIL (2025-11-14 01:48:34 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:37376->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 223.127024ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X.............X...................X..................] duration: 5.907078709s trace: https://jaeger.dev.storj.io/trace/4377cd58781b9764 FAIL (2025-11-14 01:48:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== DXCQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351147808 (113h30m14.902618269s) [noise=true , address=77.90.31.13:4628 , piece_id=WHP562KP7DABPK6XQT5F43QGNNCDFXYYYP775VGZLXIG2UEWKK4A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351381713 (113h30m14.902852174s) [noise=true , address=145.79.227.30:4763 , piece_id=IW3HMSEGIX2TFUY277OSEF4BULPDWA3AO3MNJ7OCUULORPLHNWOQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351673048 (113h30m14.903143509s) [noise=true , address=145.79.227.26:4653 , piece_id=EEKXCCTYRNCRJFVM2QBQVQADQJ2YVDCDOXFIFUP4BYUEVYDGWZQA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351818645 (113h30m14.903289106s) [noise=true , address=145.79.227.14:4840 , piece_id=3DEHNGEWEG4GA5YTLKGDV4BYUJ7Z6KCVLM343ZB5MYZWEMOASS7A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352062729 (113h30m14.903533191s) [noise=true , address=45.91.116.13:4154 , piece_id=DSC3N7KGSUSEVCEII65IQMYLHEUC5U52NNSREC32Y3Q3MDNKWREQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352224859 (113h30m14.90369532s) [noise=true , address=77.90.31.18:4807 , piece_id=TD3XBJUZ6MM5CB7NMYEIZWQC63U2PZZJPYHIM3OOU7PE3Y7KGCIQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352479703 (113h30m14.903950165s) [noise=true , address=77.90.31.17:4771 , piece_id=ARX2MBQVJSPWKWS2AHE2QBBWNREGMUORBQPNFQJHJDWBT2FCDPAA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352614791 (113h30m14.904085342s) [noise=true , address=145.79.227.21:4473 , piece_id=DIGA6L2TCAOGHAGUY5K4T3R7H66CF3JLXBXKMC7TPTXVKBCLDJBA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352904944 (113h30m14.904375405s) [noise=true , address=45.91.116.6:4973 , piece_id=NEGS6AAGBP4WRJJ6TXEBNHCV2FXW2OBMRWAWL4RNY4YZZZSI3ZLQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.353329152 (113h30m14.904799613s) [noise=true , address=77.90.31.12:4592 , piece_id=ET6LSZ6LXAAAPVJLBLL4UOIF6SKOKDCKXEXKIQ4OS3RD5J5QB2UQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.353685219 (113h30m14.90515568s) [noise=true , address=145.79.227.16:4912 , piece_id=SKK6OIUPSWXXD6QS5ZCVVORVSFXXOE5UJH7POHTKYY6PU4LX2JGA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354114446 (113h30m14.905584907s) [noise=true , address=45.91.116.12:4225 , piece_id=RMDFCX4JGDRPQ46PPTYAWJE7NIYVDE3J2LCKREBFXHR7I6U2G52A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354334907 (113h30m14.905805368s) [noise=true , address=145.79.227.13:4804 , piece_id=NZL7G7WFABHCYGW63IPFVWP45AMPZRDYLDF4JNKJHK5PLSHYHLSQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354804771 (113h30m14.906275222s) [noise=true , address=145.79.227.20:4439 , piece_id=FEDAGRFBHNL6ES3XTCRWPEI4NN2CMMDJIVNW5WOG3KYD7VXFXA7Q, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:18.090813414 (113h30m16.642283865s) [upload=99885]: Next returned nil segment Nov 14 01:34:18.090832882 (113h30m16.642303333s) [upload=99885]: Waiting for error group managing segments... Nov 14 01:34:18.376699271 (113h30m16.928169732s) [noise=true , address=45.91.116.12:4225 , piece_id=RMDFCX4JGDRPQ46PPTYAWJE7NIYVDE3J2LCKREBFXHR7I6U2G52A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.449067032 (113h30m17.000537493s) [noise=true , address=77.90.31.18:4807 , piece_id=TD3XBJUZ6MM5CB7NMYEIZWQC63U2PZZJPYHIM3OOU7PE3Y7KGCIQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.476540927 (113h30m17.028011378s) [noise=true , address=77.90.31.13:4628 , piece_id=WHP562KP7DABPK6XQT5F43QGNNCDFXYYYP775VGZLXIG2UEWKK4A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.502738892 (113h30m17.054209343s) [noise=true , address=77.90.31.12:4592 , piece_id=ET6LSZ6LXAAAPVJLBLL4UOIF6SKOKDCKXEXKIQ4OS3RD5J5QB2UQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.504520516 (113h30m17.055990977s) [noise=true , address=45.91.116.6:4973 , piece_id=NEGS6AAGBP4WRJJ6TXEBNHCV2FXW2OBMRWAWL4RNY4YZZZSI3ZLQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.504778116 (113h30m17.056248577s) [noise=true , address=45.91.116.13:4154 , piece_id=DSC3N7KGSUSEVCEII65IQMYLHEUC5U52NNSREC32Y3Q3MDNKWREQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.511639541 (113h30m17.063110002s) [noise=true , address=77.90.31.10:4517 , piece_id=GDOCT6DE4NL56DWKOS3JNSQVIVE342SDIZ5YR7XPNRXYPOQYDXCQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.529697226 (113h30m17.081167687s) [noise=true , address=77.90.31.17:4771 , piece_id=ARX2MBQVJSPWKWS2AHE2QBBWNREGMUORBQPNFQJHJDWBT2FCDPAA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:45.367038063 (113h30m43.918508514s) [noise=true , address=145.79.227.26:4653 , piece_id=EEKXCCTYRNCRJFVM2QBQVQADQJ2YVDCDOXFIFUP4BYUEVYDGWZQA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection Nov 14 01:34:45.367162339 (113h30m43.91863279s) [noise=true , address=145.79.227.14:4840 , piece_id=3DEHNGEWEG4GA5YTLKGDV4BYUJ7Z6KCVLM343ZB5MYZWEMOASS7A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection Nov 14 01:34:45.367201353 (113h30m43.918671804s) [noise=true , address=145.79.227.9:4366 , piece_id=GXQSDBFPFZPZ352A5KBNQJS5A32C7ZO522AHCHO3A27P4LAMUFZQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection Nov 14 01:34:45.367247291 (113h30m43.918717742s) [noise=true , address=145.79.227.20:4439 , piece_id=FEDAGRFBHNL6ES3XTCRWPEI4NN2CMMDJIVNW5WOG3KYD7VXFXA7Q, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection; write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection Nov 14 01:34:45.367425070 (113h30m43.918895531s) [noise=true , address=145.79.227.13:4804 , piece_id=NZL7G7WFABHCYGW63IPFVWP45AMPZRDYLDF4JNKJHK5PLSHYHLSQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection Nov 14 01:34:45.367779144 (113h30m43.919249605s) [noise=true , address=145.79.227.30:4763 , piece_id=IW3HMSEGIX2TFUY277OSEF4BULPDWA3AO3MNJ7OCUULORPLHNWOQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:50566->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:34:45.367825933 (113h30m43.919296394s) [noise=true , address=145.79.227.16:4912 , piece_id=SKK6OIUPSWXXD6QS5ZCVVORVSFXXOE5UJH7POHTKYY6PU4LX2JGA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection Nov 14 01:34:45.368025623 (113h30m43.919496114s) [noise=true , address=145.79.227.21:4473 , piece_id=DIGA6L2TCAOGHAGUY5K4T3R7H66CF3JLXBXKMC7TPTXVKBCLDJBA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: read tcp 188.166.225.170:49576->145.79.227.21:4473: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:34:45.368068755 (113h30m43.919539206s) [seg_pos={0 0}, upload=99885]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 14 01:34:45.634359530 (113h30m44.185829991s) [upload=99885]: Done uploading segments... Nov 14 01:34:45.634431096 (113h30m44.185901557s) [upload=99885]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.90711039s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.82738139s trace: https://jaeger.dev.storj.io/trace/5bc86ccfe33cf550 FAIL (2025-11-14 01:47:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ce_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288397607 (113h30m2.839868078s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288507285 (113h30m2.839977746s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.289077772 (113h30m2.840548233s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290074169 (113h30m2.84154464s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290622854 (113h30m2.842093345s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292062385 (113h30m2.843532866s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292958642 (113h30m2.844429123s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.294563758 (113h30m2.846034229s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.296755000 (113h30m2.848225462s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.298618379 (113h30m2.85008884s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.300188338 (113h30m2.85165881s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.301125483 (113h30m2.852595944s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302545266 (113h30m2.854015727s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302898339 (113h30m2.8543688s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.303825163 (113h30m2.855295624s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.316046324 (113h30m2.867516785s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.318180529 (113h30m2.86965099s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:05.581380111 (113h30m4.132850592s) [upload=99881]: Next returned nil segment Nov 14 01:34:05.581407382 (113h30m4.132877844s) [upload=99881]: Waiting for error group managing segments... Nov 14 01:34:05.921914348 (113h30m4.473384809s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.005586503 (113h30m4.557056964s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.026850744 (113h30m4.578321205s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.121926626 (113h30m4.673397087s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.235634986 (113h30m4.787105437s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.277897963 (113h30m4.829368424s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.283628915 (113h30m4.835099376s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.287808744 (113h30m4.839279205s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.300041498 (113h30m4.851511949s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.304400718 (113h30m4.855871179s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.325040680 (113h30m4.876511141s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.331007762 (113h30m4.882478223s) [noise=true , address=77.90.31.12:4611 , piece_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.481058726 (113h30m5.032529207s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:35:02.598238012 (113h31m1.149708493s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection Nov 14 01:35:02.599889276 (113h31m1.151359737s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:145.79.227.52:4455): protocol: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection Nov 14 01:35:02.600041686 (113h31m1.151512147s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:127WWD5hr1ZBkRLCHz3jb7kpqqnxPS6VuZ1MyQDmitFbhTSbJNK, address:145.79.227.49:4344): protocol: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection Nov 14 01:35:02.600244733 (113h31m1.151715204s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:145.79.227.35:4488): protocol: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection Nov 14 01:35:02.600569551 (113h31m1.152040022s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12h6MND3rBadG4EjUP18S7izyj23cVBLaB5drFsuNGhcxwAYXSA, address:145.79.227.17:4983): protocol: read tcp 188.166.225.170:33042->145.79.227.17:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:35:02.600603536 (113h31m1.152073997s) [seg_pos={0 0}, upload=99881]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 01:35:02.847367167 (113h31m1.398837648s) [upload=99881]: Done uploading segments... Nov 14 01:35:02.847427322 (113h31m1.398897783s) [upload=99881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.827411857s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X.............X...................X.................] duration: 6.071191968s trace: https://jaeger.dev.storj.io/trace/4377cd58781b9764 FAIL (2025-11-14 01:47:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== DXCQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351147808 (113h30m14.902618269s) [noise=true , address=77.90.31.13:4628 , piece_id=WHP562KP7DABPK6XQT5F43QGNNCDFXYYYP775VGZLXIG2UEWKK4A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351381713 (113h30m14.902852174s) [noise=true , address=145.79.227.30:4763 , piece_id=IW3HMSEGIX2TFUY277OSEF4BULPDWA3AO3MNJ7OCUULORPLHNWOQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351673048 (113h30m14.903143509s) [noise=true , address=145.79.227.26:4653 , piece_id=EEKXCCTYRNCRJFVM2QBQVQADQJ2YVDCDOXFIFUP4BYUEVYDGWZQA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351818645 (113h30m14.903289106s) [noise=true , address=145.79.227.14:4840 , piece_id=3DEHNGEWEG4GA5YTLKGDV4BYUJ7Z6KCVLM343ZB5MYZWEMOASS7A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352062729 (113h30m14.903533191s) [noise=true , address=45.91.116.13:4154 , piece_id=DSC3N7KGSUSEVCEII65IQMYLHEUC5U52NNSREC32Y3Q3MDNKWREQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352224859 (113h30m14.90369532s) [noise=true , address=77.90.31.18:4807 , piece_id=TD3XBJUZ6MM5CB7NMYEIZWQC63U2PZZJPYHIM3OOU7PE3Y7KGCIQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352479703 (113h30m14.903950165s) [noise=true , address=77.90.31.17:4771 , piece_id=ARX2MBQVJSPWKWS2AHE2QBBWNREGMUORBQPNFQJHJDWBT2FCDPAA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352614791 (113h30m14.904085342s) [noise=true , address=145.79.227.21:4473 , piece_id=DIGA6L2TCAOGHAGUY5K4T3R7H66CF3JLXBXKMC7TPTXVKBCLDJBA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352904944 (113h30m14.904375405s) [noise=true , address=45.91.116.6:4973 , piece_id=NEGS6AAGBP4WRJJ6TXEBNHCV2FXW2OBMRWAWL4RNY4YZZZSI3ZLQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.353329152 (113h30m14.904799613s) [noise=true , address=77.90.31.12:4592 , piece_id=ET6LSZ6LXAAAPVJLBLL4UOIF6SKOKDCKXEXKIQ4OS3RD5J5QB2UQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.353685219 (113h30m14.90515568s) [noise=true , address=145.79.227.16:4912 , piece_id=SKK6OIUPSWXXD6QS5ZCVVORVSFXXOE5UJH7POHTKYY6PU4LX2JGA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354114446 (113h30m14.905584907s) [noise=true , address=45.91.116.12:4225 , piece_id=RMDFCX4JGDRPQ46PPTYAWJE7NIYVDE3J2LCKREBFXHR7I6U2G52A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354334907 (113h30m14.905805368s) [noise=true , address=145.79.227.13:4804 , piece_id=NZL7G7WFABHCYGW63IPFVWP45AMPZRDYLDF4JNKJHK5PLSHYHLSQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354804771 (113h30m14.906275222s) [noise=true , address=145.79.227.20:4439 , piece_id=FEDAGRFBHNL6ES3XTCRWPEI4NN2CMMDJIVNW5WOG3KYD7VXFXA7Q, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:18.090813414 (113h30m16.642283865s) [upload=99885]: Next returned nil segment Nov 14 01:34:18.090832882 (113h30m16.642303333s) [upload=99885]: Waiting for error group managing segments... Nov 14 01:34:18.376699271 (113h30m16.928169732s) [noise=true , address=45.91.116.12:4225 , piece_id=RMDFCX4JGDRPQ46PPTYAWJE7NIYVDE3J2LCKREBFXHR7I6U2G52A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.449067032 (113h30m17.000537493s) [noise=true , address=77.90.31.18:4807 , piece_id=TD3XBJUZ6MM5CB7NMYEIZWQC63U2PZZJPYHIM3OOU7PE3Y7KGCIQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.476540927 (113h30m17.028011378s) [noise=true , address=77.90.31.13:4628 , piece_id=WHP562KP7DABPK6XQT5F43QGNNCDFXYYYP775VGZLXIG2UEWKK4A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.502738892 (113h30m17.054209343s) [noise=true , address=77.90.31.12:4592 , piece_id=ET6LSZ6LXAAAPVJLBLL4UOIF6SKOKDCKXEXKIQ4OS3RD5J5QB2UQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.504520516 (113h30m17.055990977s) [noise=true , address=45.91.116.6:4973 , piece_id=NEGS6AAGBP4WRJJ6TXEBNHCV2FXW2OBMRWAWL4RNY4YZZZSI3ZLQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.504778116 (113h30m17.056248577s) [noise=true , address=45.91.116.13:4154 , piece_id=DSC3N7KGSUSEVCEII65IQMYLHEUC5U52NNSREC32Y3Q3MDNKWREQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.511639541 (113h30m17.063110002s) [noise=true , address=77.90.31.10:4517 , piece_id=GDOCT6DE4NL56DWKOS3JNSQVIVE342SDIZ5YR7XPNRXYPOQYDXCQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.529697226 (113h30m17.081167687s) [noise=true , address=77.90.31.17:4771 , piece_id=ARX2MBQVJSPWKWS2AHE2QBBWNREGMUORBQPNFQJHJDWBT2FCDPAA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:45.367038063 (113h30m43.918508514s) [noise=true , address=145.79.227.26:4653 , piece_id=EEKXCCTYRNCRJFVM2QBQVQADQJ2YVDCDOXFIFUP4BYUEVYDGWZQA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection Nov 14 01:34:45.367162339 (113h30m43.91863279s) [noise=true , address=145.79.227.14:4840 , piece_id=3DEHNGEWEG4GA5YTLKGDV4BYUJ7Z6KCVLM343ZB5MYZWEMOASS7A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection Nov 14 01:34:45.367201353 (113h30m43.918671804s) [noise=true , address=145.79.227.9:4366 , piece_id=GXQSDBFPFZPZ352A5KBNQJS5A32C7ZO522AHCHO3A27P4LAMUFZQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection Nov 14 01:34:45.367247291 (113h30m43.918717742s) [noise=true , address=145.79.227.20:4439 , piece_id=FEDAGRFBHNL6ES3XTCRWPEI4NN2CMMDJIVNW5WOG3KYD7VXFXA7Q, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection; write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection Nov 14 01:34:45.367425070 (113h30m43.918895531s) [noise=true , address=145.79.227.13:4804 , piece_id=NZL7G7WFABHCYGW63IPFVWP45AMPZRDYLDF4JNKJHK5PLSHYHLSQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection Nov 14 01:34:45.367779144 (113h30m43.919249605s) [noise=true , address=145.79.227.30:4763 , piece_id=IW3HMSEGIX2TFUY277OSEF4BULPDWA3AO3MNJ7OCUULORPLHNWOQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:50566->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:34:45.367825933 (113h30m43.919296394s) [noise=true , address=145.79.227.16:4912 , piece_id=SKK6OIUPSWXXD6QS5ZCVVORVSFXXOE5UJH7POHTKYY6PU4LX2JGA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection Nov 14 01:34:45.368025623 (113h30m43.919496114s) [noise=true , address=145.79.227.21:4473 , piece_id=DIGA6L2TCAOGHAGUY5K4T3R7H66CF3JLXBXKMC7TPTXVKBCLDJBA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: read tcp 188.166.225.170:49576->145.79.227.21:4473: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:34:45.368068755 (113h30m43.919539206s) [seg_pos={0 0}, upload=99885]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 14 01:34:45.634359530 (113h30m44.185829991s) [upload=99885]: Done uploading segments... Nov 14 01:34:45.634431096 (113h30m44.185901557s) [upload=99885]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.071220453s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.718833336s trace: https://jaeger.dev.storj.io/trace/5bc86ccfe33cf550 FAIL (2025-11-14 01:47:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ce_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288397607 (113h30m2.839868078s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288507285 (113h30m2.839977746s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.289077772 (113h30m2.840548233s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290074169 (113h30m2.84154464s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290622854 (113h30m2.842093345s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292062385 (113h30m2.843532866s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292958642 (113h30m2.844429123s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.294563758 (113h30m2.846034229s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.296755000 (113h30m2.848225462s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.298618379 (113h30m2.85008884s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.300188338 (113h30m2.85165881s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.301125483 (113h30m2.852595944s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302545266 (113h30m2.854015727s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302898339 (113h30m2.8543688s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.303825163 (113h30m2.855295624s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.316046324 (113h30m2.867516785s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.318180529 (113h30m2.86965099s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:05.581380111 (113h30m4.132850592s) [upload=99881]: Next returned nil segment Nov 14 01:34:05.581407382 (113h30m4.132877844s) [upload=99881]: Waiting for error group managing segments... Nov 14 01:34:05.921914348 (113h30m4.473384809s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.005586503 (113h30m4.557056964s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.026850744 (113h30m4.578321205s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.121926626 (113h30m4.673397087s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.235634986 (113h30m4.787105437s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.277897963 (113h30m4.829368424s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.283628915 (113h30m4.835099376s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.287808744 (113h30m4.839279205s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.300041498 (113h30m4.851511949s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.304400718 (113h30m4.855871179s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.325040680 (113h30m4.876511141s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.331007762 (113h30m4.882478223s) [noise=true , address=77.90.31.12:4611 , piece_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.481058726 (113h30m5.032529207s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:35:02.598238012 (113h31m1.149708493s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection Nov 14 01:35:02.599889276 (113h31m1.151359737s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:145.79.227.52:4455): protocol: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection Nov 14 01:35:02.600041686 (113h31m1.151512147s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:127WWD5hr1ZBkRLCHz3jb7kpqqnxPS6VuZ1MyQDmitFbhTSbJNK, address:145.79.227.49:4344): protocol: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection Nov 14 01:35:02.600244733 (113h31m1.151715204s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:145.79.227.35:4488): protocol: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection Nov 14 01:35:02.600569551 (113h31m1.152040022s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12h6MND3rBadG4EjUP18S7izyj23cVBLaB5drFsuNGhcxwAYXSA, address:145.79.227.17:4983): protocol: read tcp 188.166.225.170:33042->145.79.227.17:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:35:02.600603536 (113h31m1.152073997s) [seg_pos={0 0}, upload=99881]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 01:35:02.847367167 (113h31m1.398837648s) [upload=99881]: Done uploading segments... Nov 14 01:35:02.847427322 (113h31m1.398897783s) [upload=99881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.718859196s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X.............X...................X................] duration: 5.859317502s trace: https://jaeger.dev.storj.io/trace/4377cd58781b9764 FAIL (2025-11-14 01:47:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== DXCQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351147808 (113h30m14.902618269s) [noise=true , address=77.90.31.13:4628 , piece_id=WHP562KP7DABPK6XQT5F43QGNNCDFXYYYP775VGZLXIG2UEWKK4A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351381713 (113h30m14.902852174s) [noise=true , address=145.79.227.30:4763 , piece_id=IW3HMSEGIX2TFUY277OSEF4BULPDWA3AO3MNJ7OCUULORPLHNWOQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351673048 (113h30m14.903143509s) [noise=true , address=145.79.227.26:4653 , piece_id=EEKXCCTYRNCRJFVM2QBQVQADQJ2YVDCDOXFIFUP4BYUEVYDGWZQA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351818645 (113h30m14.903289106s) [noise=true , address=145.79.227.14:4840 , piece_id=3DEHNGEWEG4GA5YTLKGDV4BYUJ7Z6KCVLM343ZB5MYZWEMOASS7A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352062729 (113h30m14.903533191s) [noise=true , address=45.91.116.13:4154 , piece_id=DSC3N7KGSUSEVCEII65IQMYLHEUC5U52NNSREC32Y3Q3MDNKWREQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352224859 (113h30m14.90369532s) [noise=true , address=77.90.31.18:4807 , piece_id=TD3XBJUZ6MM5CB7NMYEIZWQC63U2PZZJPYHIM3OOU7PE3Y7KGCIQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352479703 (113h30m14.903950165s) [noise=true , address=77.90.31.17:4771 , piece_id=ARX2MBQVJSPWKWS2AHE2QBBWNREGMUORBQPNFQJHJDWBT2FCDPAA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352614791 (113h30m14.904085342s) [noise=true , address=145.79.227.21:4473 , piece_id=DIGA6L2TCAOGHAGUY5K4T3R7H66CF3JLXBXKMC7TPTXVKBCLDJBA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352904944 (113h30m14.904375405s) [noise=true , address=45.91.116.6:4973 , piece_id=NEGS6AAGBP4WRJJ6TXEBNHCV2FXW2OBMRWAWL4RNY4YZZZSI3ZLQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.353329152 (113h30m14.904799613s) [noise=true , address=77.90.31.12:4592 , piece_id=ET6LSZ6LXAAAPVJLBLL4UOIF6SKOKDCKXEXKIQ4OS3RD5J5QB2UQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.353685219 (113h30m14.90515568s) [noise=true , address=145.79.227.16:4912 , piece_id=SKK6OIUPSWXXD6QS5ZCVVORVSFXXOE5UJH7POHTKYY6PU4LX2JGA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354114446 (113h30m14.905584907s) [noise=true , address=45.91.116.12:4225 , piece_id=RMDFCX4JGDRPQ46PPTYAWJE7NIYVDE3J2LCKREBFXHR7I6U2G52A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354334907 (113h30m14.905805368s) [noise=true , address=145.79.227.13:4804 , piece_id=NZL7G7WFABHCYGW63IPFVWP45AMPZRDYLDF4JNKJHK5PLSHYHLSQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354804771 (113h30m14.906275222s) [noise=true , address=145.79.227.20:4439 , piece_id=FEDAGRFBHNL6ES3XTCRWPEI4NN2CMMDJIVNW5WOG3KYD7VXFXA7Q, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:18.090813414 (113h30m16.642283865s) [upload=99885]: Next returned nil segment Nov 14 01:34:18.090832882 (113h30m16.642303333s) [upload=99885]: Waiting for error group managing segments... Nov 14 01:34:18.376699271 (113h30m16.928169732s) [noise=true , address=45.91.116.12:4225 , piece_id=RMDFCX4JGDRPQ46PPTYAWJE7NIYVDE3J2LCKREBFXHR7I6U2G52A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.449067032 (113h30m17.000537493s) [noise=true , address=77.90.31.18:4807 , piece_id=TD3XBJUZ6MM5CB7NMYEIZWQC63U2PZZJPYHIM3OOU7PE3Y7KGCIQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.476540927 (113h30m17.028011378s) [noise=true , address=77.90.31.13:4628 , piece_id=WHP562KP7DABPK6XQT5F43QGNNCDFXYYYP775VGZLXIG2UEWKK4A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.502738892 (113h30m17.054209343s) [noise=true , address=77.90.31.12:4592 , piece_id=ET6LSZ6LXAAAPVJLBLL4UOIF6SKOKDCKXEXKIQ4OS3RD5J5QB2UQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.504520516 (113h30m17.055990977s) [noise=true , address=45.91.116.6:4973 , piece_id=NEGS6AAGBP4WRJJ6TXEBNHCV2FXW2OBMRWAWL4RNY4YZZZSI3ZLQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.504778116 (113h30m17.056248577s) [noise=true , address=45.91.116.13:4154 , piece_id=DSC3N7KGSUSEVCEII65IQMYLHEUC5U52NNSREC32Y3Q3MDNKWREQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.511639541 (113h30m17.063110002s) [noise=true , address=77.90.31.10:4517 , piece_id=GDOCT6DE4NL56DWKOS3JNSQVIVE342SDIZ5YR7XPNRXYPOQYDXCQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.529697226 (113h30m17.081167687s) [noise=true , address=77.90.31.17:4771 , piece_id=ARX2MBQVJSPWKWS2AHE2QBBWNREGMUORBQPNFQJHJDWBT2FCDPAA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:45.367038063 (113h30m43.918508514s) [noise=true , address=145.79.227.26:4653 , piece_id=EEKXCCTYRNCRJFVM2QBQVQADQJ2YVDCDOXFIFUP4BYUEVYDGWZQA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection Nov 14 01:34:45.367162339 (113h30m43.91863279s) [noise=true , address=145.79.227.14:4840 , piece_id=3DEHNGEWEG4GA5YTLKGDV4BYUJ7Z6KCVLM343ZB5MYZWEMOASS7A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection Nov 14 01:34:45.367201353 (113h30m43.918671804s) [noise=true , address=145.79.227.9:4366 , piece_id=GXQSDBFPFZPZ352A5KBNQJS5A32C7ZO522AHCHO3A27P4LAMUFZQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection Nov 14 01:34:45.367247291 (113h30m43.918717742s) [noise=true , address=145.79.227.20:4439 , piece_id=FEDAGRFBHNL6ES3XTCRWPEI4NN2CMMDJIVNW5WOG3KYD7VXFXA7Q, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection; write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection Nov 14 01:34:45.367425070 (113h30m43.918895531s) [noise=true , address=145.79.227.13:4804 , piece_id=NZL7G7WFABHCYGW63IPFVWP45AMPZRDYLDF4JNKJHK5PLSHYHLSQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection Nov 14 01:34:45.367779144 (113h30m43.919249605s) [noise=true , address=145.79.227.30:4763 , piece_id=IW3HMSEGIX2TFUY277OSEF4BULPDWA3AO3MNJ7OCUULORPLHNWOQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:50566->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:34:45.367825933 (113h30m43.919296394s) [noise=true , address=145.79.227.16:4912 , piece_id=SKK6OIUPSWXXD6QS5ZCVVORVSFXXOE5UJH7POHTKYY6PU4LX2JGA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection Nov 14 01:34:45.368025623 (113h30m43.919496114s) [noise=true , address=145.79.227.21:4473 , piece_id=DIGA6L2TCAOGHAGUY5K4T3R7H66CF3JLXBXKMC7TPTXVKBCLDJBA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: read tcp 188.166.225.170:49576->145.79.227.21:4473: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:34:45.368068755 (113h30m43.919539206s) [seg_pos={0 0}, upload=99885]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 14 01:34:45.634359530 (113h30m44.185829991s) [upload=99885]: Done uploading segments... Nov 14 01:34:45.634431096 (113h30m44.185901557s) [upload=99885]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.859337461s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........X.............X...................X...............] duration: 5.869518055s trace: https://jaeger.dev.storj.io/trace/4377cd58781b9764 FAIL (2025-11-14 01:46:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== DXCQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351147808 (113h30m14.902618269s) [noise=true , address=77.90.31.13:4628 , piece_id=WHP562KP7DABPK6XQT5F43QGNNCDFXYYYP775VGZLXIG2UEWKK4A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351381713 (113h30m14.902852174s) [noise=true , address=145.79.227.30:4763 , piece_id=IW3HMSEGIX2TFUY277OSEF4BULPDWA3AO3MNJ7OCUULORPLHNWOQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351673048 (113h30m14.903143509s) [noise=true , address=145.79.227.26:4653 , piece_id=EEKXCCTYRNCRJFVM2QBQVQADQJ2YVDCDOXFIFUP4BYUEVYDGWZQA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351818645 (113h30m14.903289106s) [noise=true , address=145.79.227.14:4840 , piece_id=3DEHNGEWEG4GA5YTLKGDV4BYUJ7Z6KCVLM343ZB5MYZWEMOASS7A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352062729 (113h30m14.903533191s) [noise=true , address=45.91.116.13:4154 , piece_id=DSC3N7KGSUSEVCEII65IQMYLHEUC5U52NNSREC32Y3Q3MDNKWREQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352224859 (113h30m14.90369532s) [noise=true , address=77.90.31.18:4807 , piece_id=TD3XBJUZ6MM5CB7NMYEIZWQC63U2PZZJPYHIM3OOU7PE3Y7KGCIQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352479703 (113h30m14.903950165s) [noise=true , address=77.90.31.17:4771 , piece_id=ARX2MBQVJSPWKWS2AHE2QBBWNREGMUORBQPNFQJHJDWBT2FCDPAA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352614791 (113h30m14.904085342s) [noise=true , address=145.79.227.21:4473 , piece_id=DIGA6L2TCAOGHAGUY5K4T3R7H66CF3JLXBXKMC7TPTXVKBCLDJBA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352904944 (113h30m14.904375405s) [noise=true , address=45.91.116.6:4973 , piece_id=NEGS6AAGBP4WRJJ6TXEBNHCV2FXW2OBMRWAWL4RNY4YZZZSI3ZLQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.353329152 (113h30m14.904799613s) [noise=true , address=77.90.31.12:4592 , piece_id=ET6LSZ6LXAAAPVJLBLL4UOIF6SKOKDCKXEXKIQ4OS3RD5J5QB2UQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.353685219 (113h30m14.90515568s) [noise=true , address=145.79.227.16:4912 , piece_id=SKK6OIUPSWXXD6QS5ZCVVORVSFXXOE5UJH7POHTKYY6PU4LX2JGA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354114446 (113h30m14.905584907s) [noise=true , address=45.91.116.12:4225 , piece_id=RMDFCX4JGDRPQ46PPTYAWJE7NIYVDE3J2LCKREBFXHR7I6U2G52A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354334907 (113h30m14.905805368s) [noise=true , address=145.79.227.13:4804 , piece_id=NZL7G7WFABHCYGW63IPFVWP45AMPZRDYLDF4JNKJHK5PLSHYHLSQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354804771 (113h30m14.906275222s) [noise=true , address=145.79.227.20:4439 , piece_id=FEDAGRFBHNL6ES3XTCRWPEI4NN2CMMDJIVNW5WOG3KYD7VXFXA7Q, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:18.090813414 (113h30m16.642283865s) [upload=99885]: Next returned nil segment Nov 14 01:34:18.090832882 (113h30m16.642303333s) [upload=99885]: Waiting for error group managing segments... Nov 14 01:34:18.376699271 (113h30m16.928169732s) [noise=true , address=45.91.116.12:4225 , piece_id=RMDFCX4JGDRPQ46PPTYAWJE7NIYVDE3J2LCKREBFXHR7I6U2G52A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.449067032 (113h30m17.000537493s) [noise=true , address=77.90.31.18:4807 , piece_id=TD3XBJUZ6MM5CB7NMYEIZWQC63U2PZZJPYHIM3OOU7PE3Y7KGCIQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.476540927 (113h30m17.028011378s) [noise=true , address=77.90.31.13:4628 , piece_id=WHP562KP7DABPK6XQT5F43QGNNCDFXYYYP775VGZLXIG2UEWKK4A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.502738892 (113h30m17.054209343s) [noise=true , address=77.90.31.12:4592 , piece_id=ET6LSZ6LXAAAPVJLBLL4UOIF6SKOKDCKXEXKIQ4OS3RD5J5QB2UQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.504520516 (113h30m17.055990977s) [noise=true , address=45.91.116.6:4973 , piece_id=NEGS6AAGBP4WRJJ6TXEBNHCV2FXW2OBMRWAWL4RNY4YZZZSI3ZLQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.504778116 (113h30m17.056248577s) [noise=true , address=45.91.116.13:4154 , piece_id=DSC3N7KGSUSEVCEII65IQMYLHEUC5U52NNSREC32Y3Q3MDNKWREQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.511639541 (113h30m17.063110002s) [noise=true , address=77.90.31.10:4517 , piece_id=GDOCT6DE4NL56DWKOS3JNSQVIVE342SDIZ5YR7XPNRXYPOQYDXCQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.529697226 (113h30m17.081167687s) [noise=true , address=77.90.31.17:4771 , piece_id=ARX2MBQVJSPWKWS2AHE2QBBWNREGMUORBQPNFQJHJDWBT2FCDPAA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:45.367038063 (113h30m43.918508514s) [noise=true , address=145.79.227.26:4653 , piece_id=EEKXCCTYRNCRJFVM2QBQVQADQJ2YVDCDOXFIFUP4BYUEVYDGWZQA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection Nov 14 01:34:45.367162339 (113h30m43.91863279s) [noise=true , address=145.79.227.14:4840 , piece_id=3DEHNGEWEG4GA5YTLKGDV4BYUJ7Z6KCVLM343ZB5MYZWEMOASS7A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection Nov 14 01:34:45.367201353 (113h30m43.918671804s) [noise=true , address=145.79.227.9:4366 , piece_id=GXQSDBFPFZPZ352A5KBNQJS5A32C7ZO522AHCHO3A27P4LAMUFZQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection Nov 14 01:34:45.367247291 (113h30m43.918717742s) [noise=true , address=145.79.227.20:4439 , piece_id=FEDAGRFBHNL6ES3XTCRWPEI4NN2CMMDJIVNW5WOG3KYD7VXFXA7Q, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection; write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection Nov 14 01:34:45.367425070 (113h30m43.918895531s) [noise=true , address=145.79.227.13:4804 , piece_id=NZL7G7WFABHCYGW63IPFVWP45AMPZRDYLDF4JNKJHK5PLSHYHLSQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection Nov 14 01:34:45.367779144 (113h30m43.919249605s) [noise=true , address=145.79.227.30:4763 , piece_id=IW3HMSEGIX2TFUY277OSEF4BULPDWA3AO3MNJ7OCUULORPLHNWOQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:50566->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:34:45.367825933 (113h30m43.919296394s) [noise=true , address=145.79.227.16:4912 , piece_id=SKK6OIUPSWXXD6QS5ZCVVORVSFXXOE5UJH7POHTKYY6PU4LX2JGA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection Nov 14 01:34:45.368025623 (113h30m43.919496114s) [noise=true , address=145.79.227.21:4473 , piece_id=DIGA6L2TCAOGHAGUY5K4T3R7H66CF3JLXBXKMC7TPTXVKBCLDJBA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: read tcp 188.166.225.170:49576->145.79.227.21:4473: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:34:45.368068755 (113h30m43.919539206s) [seg_pos={0 0}, upload=99885]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 14 01:34:45.634359530 (113h30m44.185829991s) [upload=99885]: Done uploading segments... Nov 14 01:34:45.634431096 (113h30m44.185901557s) [upload=99885]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.869546741s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.797916984s trace: https://jaeger.dev.storj.io/trace/5bc86ccfe33cf550 FAIL (2025-11-14 01:46:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ce_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288397607 (113h30m2.839868078s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288507285 (113h30m2.839977746s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.289077772 (113h30m2.840548233s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290074169 (113h30m2.84154464s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290622854 (113h30m2.842093345s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292062385 (113h30m2.843532866s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292958642 (113h30m2.844429123s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.294563758 (113h30m2.846034229s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.296755000 (113h30m2.848225462s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.298618379 (113h30m2.85008884s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.300188338 (113h30m2.85165881s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.301125483 (113h30m2.852595944s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302545266 (113h30m2.854015727s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302898339 (113h30m2.8543688s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.303825163 (113h30m2.855295624s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.316046324 (113h30m2.867516785s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.318180529 (113h30m2.86965099s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:05.581380111 (113h30m4.132850592s) [upload=99881]: Next returned nil segment Nov 14 01:34:05.581407382 (113h30m4.132877844s) [upload=99881]: Waiting for error group managing segments... Nov 14 01:34:05.921914348 (113h30m4.473384809s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.005586503 (113h30m4.557056964s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.026850744 (113h30m4.578321205s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.121926626 (113h30m4.673397087s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.235634986 (113h30m4.787105437s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.277897963 (113h30m4.829368424s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.283628915 (113h30m4.835099376s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.287808744 (113h30m4.839279205s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.300041498 (113h30m4.851511949s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.304400718 (113h30m4.855871179s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.325040680 (113h30m4.876511141s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.331007762 (113h30m4.882478223s) [noise=true , address=77.90.31.12:4611 , piece_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.481058726 (113h30m5.032529207s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:35:02.598238012 (113h31m1.149708493s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection Nov 14 01:35:02.599889276 (113h31m1.151359737s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:145.79.227.52:4455): protocol: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection Nov 14 01:35:02.600041686 (113h31m1.151512147s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:127WWD5hr1ZBkRLCHz3jb7kpqqnxPS6VuZ1MyQDmitFbhTSbJNK, address:145.79.227.49:4344): protocol: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection Nov 14 01:35:02.600244733 (113h31m1.151715204s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:145.79.227.35:4488): protocol: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection Nov 14 01:35:02.600569551 (113h31m1.152040022s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12h6MND3rBadG4EjUP18S7izyj23cVBLaB5drFsuNGhcxwAYXSA, address:145.79.227.17:4983): protocol: read tcp 188.166.225.170:33042->145.79.227.17:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:35:02.600603536 (113h31m1.152073997s) [seg_pos={0 0}, upload=99881]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 01:35:02.847367167 (113h31m1.398837648s) [upload=99881]: Done uploading segments... Nov 14 01:35:02.847427322 (113h31m1.398897783s) [upload=99881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.797949146s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.634423445s trace: https://jaeger.dev.storj.io/trace/5bc86ccfe33cf550 FAIL (2025-11-14 01:45:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ce_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288397607 (113h30m2.839868078s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288507285 (113h30m2.839977746s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.289077772 (113h30m2.840548233s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290074169 (113h30m2.84154464s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290622854 (113h30m2.842093345s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292062385 (113h30m2.843532866s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292958642 (113h30m2.844429123s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.294563758 (113h30m2.846034229s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.296755000 (113h30m2.848225462s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.298618379 (113h30m2.85008884s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.300188338 (113h30m2.85165881s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.301125483 (113h30m2.852595944s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302545266 (113h30m2.854015727s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302898339 (113h30m2.8543688s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.303825163 (113h30m2.855295624s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.316046324 (113h30m2.867516785s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.318180529 (113h30m2.86965099s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:05.581380111 (113h30m4.132850592s) [upload=99881]: Next returned nil segment Nov 14 01:34:05.581407382 (113h30m4.132877844s) [upload=99881]: Waiting for error group managing segments... Nov 14 01:34:05.921914348 (113h30m4.473384809s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.005586503 (113h30m4.557056964s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.026850744 (113h30m4.578321205s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.121926626 (113h30m4.673397087s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.235634986 (113h30m4.787105437s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.277897963 (113h30m4.829368424s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.283628915 (113h30m4.835099376s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.287808744 (113h30m4.839279205s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.300041498 (113h30m4.851511949s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.304400718 (113h30m4.855871179s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.325040680 (113h30m4.876511141s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.331007762 (113h30m4.882478223s) [noise=true , address=77.90.31.12:4611 , piece_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.481058726 (113h30m5.032529207s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:35:02.598238012 (113h31m1.149708493s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection Nov 14 01:35:02.599889276 (113h31m1.151359737s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:145.79.227.52:4455): protocol: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection Nov 14 01:35:02.600041686 (113h31m1.151512147s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:127WWD5hr1ZBkRLCHz3jb7kpqqnxPS6VuZ1MyQDmitFbhTSbJNK, address:145.79.227.49:4344): protocol: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection Nov 14 01:35:02.600244733 (113h31m1.151715204s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:145.79.227.35:4488): protocol: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection Nov 14 01:35:02.600569551 (113h31m1.152040022s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12h6MND3rBadG4EjUP18S7izyj23cVBLaB5drFsuNGhcxwAYXSA, address:145.79.227.17:4983): protocol: read tcp 188.166.225.170:33042->145.79.227.17:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:35:02.600603536 (113h31m1.152073997s) [seg_pos={0 0}, upload=99881]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 01:35:02.847367167 (113h31m1.398837648s) [upload=99881]: Done uploading segments... Nov 14 01:35:02.847427322 (113h31m1.398897783s) [upload=99881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.634450426s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X.............X...................X..............] duration: 6.195794803s trace: https://jaeger.dev.storj.io/trace/4377cd58781b9764 FAIL (2025-11-14 01:45:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== DXCQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351147808 (113h30m14.902618269s) [noise=true , address=77.90.31.13:4628 , piece_id=WHP562KP7DABPK6XQT5F43QGNNCDFXYYYP775VGZLXIG2UEWKK4A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351381713 (113h30m14.902852174s) [noise=true , address=145.79.227.30:4763 , piece_id=IW3HMSEGIX2TFUY277OSEF4BULPDWA3AO3MNJ7OCUULORPLHNWOQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351673048 (113h30m14.903143509s) [noise=true , address=145.79.227.26:4653 , piece_id=EEKXCCTYRNCRJFVM2QBQVQADQJ2YVDCDOXFIFUP4BYUEVYDGWZQA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351818645 (113h30m14.903289106s) [noise=true , address=145.79.227.14:4840 , piece_id=3DEHNGEWEG4GA5YTLKGDV4BYUJ7Z6KCVLM343ZB5MYZWEMOASS7A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352062729 (113h30m14.903533191s) [noise=true , address=45.91.116.13:4154 , piece_id=DSC3N7KGSUSEVCEII65IQMYLHEUC5U52NNSREC32Y3Q3MDNKWREQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352224859 (113h30m14.90369532s) [noise=true , address=77.90.31.18:4807 , piece_id=TD3XBJUZ6MM5CB7NMYEIZWQC63U2PZZJPYHIM3OOU7PE3Y7KGCIQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352479703 (113h30m14.903950165s) [noise=true , address=77.90.31.17:4771 , piece_id=ARX2MBQVJSPWKWS2AHE2QBBWNREGMUORBQPNFQJHJDWBT2FCDPAA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352614791 (113h30m14.904085342s) [noise=true , address=145.79.227.21:4473 , piece_id=DIGA6L2TCAOGHAGUY5K4T3R7H66CF3JLXBXKMC7TPTXVKBCLDJBA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352904944 (113h30m14.904375405s) [noise=true , address=45.91.116.6:4973 , piece_id=NEGS6AAGBP4WRJJ6TXEBNHCV2FXW2OBMRWAWL4RNY4YZZZSI3ZLQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.353329152 (113h30m14.904799613s) [noise=true , address=77.90.31.12:4592 , piece_id=ET6LSZ6LXAAAPVJLBLL4UOIF6SKOKDCKXEXKIQ4OS3RD5J5QB2UQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.353685219 (113h30m14.90515568s) [noise=true , address=145.79.227.16:4912 , piece_id=SKK6OIUPSWXXD6QS5ZCVVORVSFXXOE5UJH7POHTKYY6PU4LX2JGA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354114446 (113h30m14.905584907s) [noise=true , address=45.91.116.12:4225 , piece_id=RMDFCX4JGDRPQ46PPTYAWJE7NIYVDE3J2LCKREBFXHR7I6U2G52A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354334907 (113h30m14.905805368s) [noise=true , address=145.79.227.13:4804 , piece_id=NZL7G7WFABHCYGW63IPFVWP45AMPZRDYLDF4JNKJHK5PLSHYHLSQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354804771 (113h30m14.906275222s) [noise=true , address=145.79.227.20:4439 , piece_id=FEDAGRFBHNL6ES3XTCRWPEI4NN2CMMDJIVNW5WOG3KYD7VXFXA7Q, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:18.090813414 (113h30m16.642283865s) [upload=99885]: Next returned nil segment Nov 14 01:34:18.090832882 (113h30m16.642303333s) [upload=99885]: Waiting for error group managing segments... Nov 14 01:34:18.376699271 (113h30m16.928169732s) [noise=true , address=45.91.116.12:4225 , piece_id=RMDFCX4JGDRPQ46PPTYAWJE7NIYVDE3J2LCKREBFXHR7I6U2G52A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.449067032 (113h30m17.000537493s) [noise=true , address=77.90.31.18:4807 , piece_id=TD3XBJUZ6MM5CB7NMYEIZWQC63U2PZZJPYHIM3OOU7PE3Y7KGCIQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.476540927 (113h30m17.028011378s) [noise=true , address=77.90.31.13:4628 , piece_id=WHP562KP7DABPK6XQT5F43QGNNCDFXYYYP775VGZLXIG2UEWKK4A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.502738892 (113h30m17.054209343s) [noise=true , address=77.90.31.12:4592 , piece_id=ET6LSZ6LXAAAPVJLBLL4UOIF6SKOKDCKXEXKIQ4OS3RD5J5QB2UQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.504520516 (113h30m17.055990977s) [noise=true , address=45.91.116.6:4973 , piece_id=NEGS6AAGBP4WRJJ6TXEBNHCV2FXW2OBMRWAWL4RNY4YZZZSI3ZLQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.504778116 (113h30m17.056248577s) [noise=true , address=45.91.116.13:4154 , piece_id=DSC3N7KGSUSEVCEII65IQMYLHEUC5U52NNSREC32Y3Q3MDNKWREQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.511639541 (113h30m17.063110002s) [noise=true , address=77.90.31.10:4517 , piece_id=GDOCT6DE4NL56DWKOS3JNSQVIVE342SDIZ5YR7XPNRXYPOQYDXCQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.529697226 (113h30m17.081167687s) [noise=true , address=77.90.31.17:4771 , piece_id=ARX2MBQVJSPWKWS2AHE2QBBWNREGMUORBQPNFQJHJDWBT2FCDPAA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:45.367038063 (113h30m43.918508514s) [noise=true , address=145.79.227.26:4653 , piece_id=EEKXCCTYRNCRJFVM2QBQVQADQJ2YVDCDOXFIFUP4BYUEVYDGWZQA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection Nov 14 01:34:45.367162339 (113h30m43.91863279s) [noise=true , address=145.79.227.14:4840 , piece_id=3DEHNGEWEG4GA5YTLKGDV4BYUJ7Z6KCVLM343ZB5MYZWEMOASS7A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection Nov 14 01:34:45.367201353 (113h30m43.918671804s) [noise=true , address=145.79.227.9:4366 , piece_id=GXQSDBFPFZPZ352A5KBNQJS5A32C7ZO522AHCHO3A27P4LAMUFZQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection Nov 14 01:34:45.367247291 (113h30m43.918717742s) [noise=true , address=145.79.227.20:4439 , piece_id=FEDAGRFBHNL6ES3XTCRWPEI4NN2CMMDJIVNW5WOG3KYD7VXFXA7Q, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection; write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection Nov 14 01:34:45.367425070 (113h30m43.918895531s) [noise=true , address=145.79.227.13:4804 , piece_id=NZL7G7WFABHCYGW63IPFVWP45AMPZRDYLDF4JNKJHK5PLSHYHLSQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection Nov 14 01:34:45.367779144 (113h30m43.919249605s) [noise=true , address=145.79.227.30:4763 , piece_id=IW3HMSEGIX2TFUY277OSEF4BULPDWA3AO3MNJ7OCUULORPLHNWOQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:50566->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:34:45.367825933 (113h30m43.919296394s) [noise=true , address=145.79.227.16:4912 , piece_id=SKK6OIUPSWXXD6QS5ZCVVORVSFXXOE5UJH7POHTKYY6PU4LX2JGA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection Nov 14 01:34:45.368025623 (113h30m43.919496114s) [noise=true , address=145.79.227.21:4473 , piece_id=DIGA6L2TCAOGHAGUY5K4T3R7H66CF3JLXBXKMC7TPTXVKBCLDJBA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: read tcp 188.166.225.170:49576->145.79.227.21:4473: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:34:45.368068755 (113h30m43.919539206s) [seg_pos={0 0}, upload=99885]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 14 01:34:45.634359530 (113h30m44.185829991s) [upload=99885]: Done uploading segments... Nov 14 01:34:45.634431096 (113h30m44.185901557s) [upload=99885]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.195823448s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.922929691s trace: https://jaeger.dev.storj.io/trace/5bc86ccfe33cf550 FAIL (2025-11-14 01:44:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ce_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288397607 (113h30m2.839868078s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288507285 (113h30m2.839977746s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.289077772 (113h30m2.840548233s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290074169 (113h30m2.84154464s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290622854 (113h30m2.842093345s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292062385 (113h30m2.843532866s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292958642 (113h30m2.844429123s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.294563758 (113h30m2.846034229s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.296755000 (113h30m2.848225462s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.298618379 (113h30m2.85008884s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.300188338 (113h30m2.85165881s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.301125483 (113h30m2.852595944s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302545266 (113h30m2.854015727s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302898339 (113h30m2.8543688s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.303825163 (113h30m2.855295624s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.316046324 (113h30m2.867516785s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.318180529 (113h30m2.86965099s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:05.581380111 (113h30m4.132850592s) [upload=99881]: Next returned nil segment Nov 14 01:34:05.581407382 (113h30m4.132877844s) [upload=99881]: Waiting for error group managing segments... Nov 14 01:34:05.921914348 (113h30m4.473384809s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.005586503 (113h30m4.557056964s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.026850744 (113h30m4.578321205s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.121926626 (113h30m4.673397087s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.235634986 (113h30m4.787105437s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.277897963 (113h30m4.829368424s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.283628915 (113h30m4.835099376s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.287808744 (113h30m4.839279205s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.300041498 (113h30m4.851511949s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.304400718 (113h30m4.855871179s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.325040680 (113h30m4.876511141s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.331007762 (113h30m4.882478223s) [noise=true , address=77.90.31.12:4611 , piece_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.481058726 (113h30m5.032529207s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:35:02.598238012 (113h31m1.149708493s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection Nov 14 01:35:02.599889276 (113h31m1.151359737s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:145.79.227.52:4455): protocol: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection Nov 14 01:35:02.600041686 (113h31m1.151512147s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:127WWD5hr1ZBkRLCHz3jb7kpqqnxPS6VuZ1MyQDmitFbhTSbJNK, address:145.79.227.49:4344): protocol: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection Nov 14 01:35:02.600244733 (113h31m1.151715204s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:145.79.227.35:4488): protocol: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection Nov 14 01:35:02.600569551 (113h31m1.152040022s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12h6MND3rBadG4EjUP18S7izyj23cVBLaB5drFsuNGhcxwAYXSA, address:145.79.227.17:4983): protocol: read tcp 188.166.225.170:33042->145.79.227.17:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:35:02.600603536 (113h31m1.152073997s) [seg_pos={0 0}, upload=99881]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 01:35:02.847367167 (113h31m1.398837648s) [upload=99881]: Done uploading segments... Nov 14 01:35:02.847427322 (113h31m1.398897783s) [upload=99881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.922953235s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X.............X...................X.............] duration: 6.23532713s trace: https://jaeger.dev.storj.io/trace/4377cd58781b9764 FAIL (2025-11-14 01:44:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== DXCQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351147808 (113h30m14.902618269s) [noise=true , address=77.90.31.13:4628 , piece_id=WHP562KP7DABPK6XQT5F43QGNNCDFXYYYP775VGZLXIG2UEWKK4A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351381713 (113h30m14.902852174s) [noise=true , address=145.79.227.30:4763 , piece_id=IW3HMSEGIX2TFUY277OSEF4BULPDWA3AO3MNJ7OCUULORPLHNWOQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351673048 (113h30m14.903143509s) [noise=true , address=145.79.227.26:4653 , piece_id=EEKXCCTYRNCRJFVM2QBQVQADQJ2YVDCDOXFIFUP4BYUEVYDGWZQA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351818645 (113h30m14.903289106s) [noise=true , address=145.79.227.14:4840 , piece_id=3DEHNGEWEG4GA5YTLKGDV4BYUJ7Z6KCVLM343ZB5MYZWEMOASS7A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352062729 (113h30m14.903533191s) [noise=true , address=45.91.116.13:4154 , piece_id=DSC3N7KGSUSEVCEII65IQMYLHEUC5U52NNSREC32Y3Q3MDNKWREQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352224859 (113h30m14.90369532s) [noise=true , address=77.90.31.18:4807 , piece_id=TD3XBJUZ6MM5CB7NMYEIZWQC63U2PZZJPYHIM3OOU7PE3Y7KGCIQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352479703 (113h30m14.903950165s) [noise=true , address=77.90.31.17:4771 , piece_id=ARX2MBQVJSPWKWS2AHE2QBBWNREGMUORBQPNFQJHJDWBT2FCDPAA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352614791 (113h30m14.904085342s) [noise=true , address=145.79.227.21:4473 , piece_id=DIGA6L2TCAOGHAGUY5K4T3R7H66CF3JLXBXKMC7TPTXVKBCLDJBA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352904944 (113h30m14.904375405s) [noise=true , address=45.91.116.6:4973 , piece_id=NEGS6AAGBP4WRJJ6TXEBNHCV2FXW2OBMRWAWL4RNY4YZZZSI3ZLQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.353329152 (113h30m14.904799613s) [noise=true , address=77.90.31.12:4592 , piece_id=ET6LSZ6LXAAAPVJLBLL4UOIF6SKOKDCKXEXKIQ4OS3RD5J5QB2UQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.353685219 (113h30m14.90515568s) [noise=true , address=145.79.227.16:4912 , piece_id=SKK6OIUPSWXXD6QS5ZCVVORVSFXXOE5UJH7POHTKYY6PU4LX2JGA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354114446 (113h30m14.905584907s) [noise=true , address=45.91.116.12:4225 , piece_id=RMDFCX4JGDRPQ46PPTYAWJE7NIYVDE3J2LCKREBFXHR7I6U2G52A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354334907 (113h30m14.905805368s) [noise=true , address=145.79.227.13:4804 , piece_id=NZL7G7WFABHCYGW63IPFVWP45AMPZRDYLDF4JNKJHK5PLSHYHLSQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354804771 (113h30m14.906275222s) [noise=true , address=145.79.227.20:4439 , piece_id=FEDAGRFBHNL6ES3XTCRWPEI4NN2CMMDJIVNW5WOG3KYD7VXFXA7Q, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:18.090813414 (113h30m16.642283865s) [upload=99885]: Next returned nil segment Nov 14 01:34:18.090832882 (113h30m16.642303333s) [upload=99885]: Waiting for error group managing segments... Nov 14 01:34:18.376699271 (113h30m16.928169732s) [noise=true , address=45.91.116.12:4225 , piece_id=RMDFCX4JGDRPQ46PPTYAWJE7NIYVDE3J2LCKREBFXHR7I6U2G52A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.449067032 (113h30m17.000537493s) [noise=true , address=77.90.31.18:4807 , piece_id=TD3XBJUZ6MM5CB7NMYEIZWQC63U2PZZJPYHIM3OOU7PE3Y7KGCIQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.476540927 (113h30m17.028011378s) [noise=true , address=77.90.31.13:4628 , piece_id=WHP562KP7DABPK6XQT5F43QGNNCDFXYYYP775VGZLXIG2UEWKK4A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.502738892 (113h30m17.054209343s) [noise=true , address=77.90.31.12:4592 , piece_id=ET6LSZ6LXAAAPVJLBLL4UOIF6SKOKDCKXEXKIQ4OS3RD5J5QB2UQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.504520516 (113h30m17.055990977s) [noise=true , address=45.91.116.6:4973 , piece_id=NEGS6AAGBP4WRJJ6TXEBNHCV2FXW2OBMRWAWL4RNY4YZZZSI3ZLQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.504778116 (113h30m17.056248577s) [noise=true , address=45.91.116.13:4154 , piece_id=DSC3N7KGSUSEVCEII65IQMYLHEUC5U52NNSREC32Y3Q3MDNKWREQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.511639541 (113h30m17.063110002s) [noise=true , address=77.90.31.10:4517 , piece_id=GDOCT6DE4NL56DWKOS3JNSQVIVE342SDIZ5YR7XPNRXYPOQYDXCQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.529697226 (113h30m17.081167687s) [noise=true , address=77.90.31.17:4771 , piece_id=ARX2MBQVJSPWKWS2AHE2QBBWNREGMUORBQPNFQJHJDWBT2FCDPAA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:45.367038063 (113h30m43.918508514s) [noise=true , address=145.79.227.26:4653 , piece_id=EEKXCCTYRNCRJFVM2QBQVQADQJ2YVDCDOXFIFUP4BYUEVYDGWZQA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection Nov 14 01:34:45.367162339 (113h30m43.91863279s) [noise=true , address=145.79.227.14:4840 , piece_id=3DEHNGEWEG4GA5YTLKGDV4BYUJ7Z6KCVLM343ZB5MYZWEMOASS7A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection Nov 14 01:34:45.367201353 (113h30m43.918671804s) [noise=true , address=145.79.227.9:4366 , piece_id=GXQSDBFPFZPZ352A5KBNQJS5A32C7ZO522AHCHO3A27P4LAMUFZQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection Nov 14 01:34:45.367247291 (113h30m43.918717742s) [noise=true , address=145.79.227.20:4439 , piece_id=FEDAGRFBHNL6ES3XTCRWPEI4NN2CMMDJIVNW5WOG3KYD7VXFXA7Q, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection; write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection Nov 14 01:34:45.367425070 (113h30m43.918895531s) [noise=true , address=145.79.227.13:4804 , piece_id=NZL7G7WFABHCYGW63IPFVWP45AMPZRDYLDF4JNKJHK5PLSHYHLSQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection Nov 14 01:34:45.367779144 (113h30m43.919249605s) [noise=true , address=145.79.227.30:4763 , piece_id=IW3HMSEGIX2TFUY277OSEF4BULPDWA3AO3MNJ7OCUULORPLHNWOQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:50566->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:34:45.367825933 (113h30m43.919296394s) [noise=true , address=145.79.227.16:4912 , piece_id=SKK6OIUPSWXXD6QS5ZCVVORVSFXXOE5UJH7POHTKYY6PU4LX2JGA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection Nov 14 01:34:45.368025623 (113h30m43.919496114s) [noise=true , address=145.79.227.21:4473 , piece_id=DIGA6L2TCAOGHAGUY5K4T3R7H66CF3JLXBXKMC7TPTXVKBCLDJBA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: read tcp 188.166.225.170:49576->145.79.227.21:4473: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:34:45.368068755 (113h30m43.919539206s) [seg_pos={0 0}, upload=99885]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 14 01:34:45.634359530 (113h30m44.185829991s) [upload=99885]: Done uploading segments... Nov 14 01:34:45.634431096 (113h30m44.185901557s) [upload=99885]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.235355425s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X.............X...................X............] duration: 5.98402701s trace: https://jaeger.dev.storj.io/trace/4377cd58781b9764 FAIL (2025-11-14 01:43:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== DXCQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351147808 (113h30m14.902618269s) [noise=true , address=77.90.31.13:4628 , piece_id=WHP562KP7DABPK6XQT5F43QGNNCDFXYYYP775VGZLXIG2UEWKK4A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351381713 (113h30m14.902852174s) [noise=true , address=145.79.227.30:4763 , piece_id=IW3HMSEGIX2TFUY277OSEF4BULPDWA3AO3MNJ7OCUULORPLHNWOQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351673048 (113h30m14.903143509s) [noise=true , address=145.79.227.26:4653 , piece_id=EEKXCCTYRNCRJFVM2QBQVQADQJ2YVDCDOXFIFUP4BYUEVYDGWZQA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351818645 (113h30m14.903289106s) [noise=true , address=145.79.227.14:4840 , piece_id=3DEHNGEWEG4GA5YTLKGDV4BYUJ7Z6KCVLM343ZB5MYZWEMOASS7A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352062729 (113h30m14.903533191s) [noise=true , address=45.91.116.13:4154 , piece_id=DSC3N7KGSUSEVCEII65IQMYLHEUC5U52NNSREC32Y3Q3MDNKWREQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352224859 (113h30m14.90369532s) [noise=true , address=77.90.31.18:4807 , piece_id=TD3XBJUZ6MM5CB7NMYEIZWQC63U2PZZJPYHIM3OOU7PE3Y7KGCIQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352479703 (113h30m14.903950165s) [noise=true , address=77.90.31.17:4771 , piece_id=ARX2MBQVJSPWKWS2AHE2QBBWNREGMUORBQPNFQJHJDWBT2FCDPAA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352614791 (113h30m14.904085342s) [noise=true , address=145.79.227.21:4473 , piece_id=DIGA6L2TCAOGHAGUY5K4T3R7H66CF3JLXBXKMC7TPTXVKBCLDJBA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352904944 (113h30m14.904375405s) [noise=true , address=45.91.116.6:4973 , piece_id=NEGS6AAGBP4WRJJ6TXEBNHCV2FXW2OBMRWAWL4RNY4YZZZSI3ZLQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.353329152 (113h30m14.904799613s) [noise=true , address=77.90.31.12:4592 , piece_id=ET6LSZ6LXAAAPVJLBLL4UOIF6SKOKDCKXEXKIQ4OS3RD5J5QB2UQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.353685219 (113h30m14.90515568s) [noise=true , address=145.79.227.16:4912 , piece_id=SKK6OIUPSWXXD6QS5ZCVVORVSFXXOE5UJH7POHTKYY6PU4LX2JGA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354114446 (113h30m14.905584907s) [noise=true , address=45.91.116.12:4225 , piece_id=RMDFCX4JGDRPQ46PPTYAWJE7NIYVDE3J2LCKREBFXHR7I6U2G52A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354334907 (113h30m14.905805368s) [noise=true , address=145.79.227.13:4804 , piece_id=NZL7G7WFABHCYGW63IPFVWP45AMPZRDYLDF4JNKJHK5PLSHYHLSQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354804771 (113h30m14.906275222s) [noise=true , address=145.79.227.20:4439 , piece_id=FEDAGRFBHNL6ES3XTCRWPEI4NN2CMMDJIVNW5WOG3KYD7VXFXA7Q, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:18.090813414 (113h30m16.642283865s) [upload=99885]: Next returned nil segment Nov 14 01:34:18.090832882 (113h30m16.642303333s) [upload=99885]: Waiting for error group managing segments... Nov 14 01:34:18.376699271 (113h30m16.928169732s) [noise=true , address=45.91.116.12:4225 , piece_id=RMDFCX4JGDRPQ46PPTYAWJE7NIYVDE3J2LCKREBFXHR7I6U2G52A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.449067032 (113h30m17.000537493s) [noise=true , address=77.90.31.18:4807 , piece_id=TD3XBJUZ6MM5CB7NMYEIZWQC63U2PZZJPYHIM3OOU7PE3Y7KGCIQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.476540927 (113h30m17.028011378s) [noise=true , address=77.90.31.13:4628 , piece_id=WHP562KP7DABPK6XQT5F43QGNNCDFXYYYP775VGZLXIG2UEWKK4A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.502738892 (113h30m17.054209343s) [noise=true , address=77.90.31.12:4592 , piece_id=ET6LSZ6LXAAAPVJLBLL4UOIF6SKOKDCKXEXKIQ4OS3RD5J5QB2UQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.504520516 (113h30m17.055990977s) [noise=true , address=45.91.116.6:4973 , piece_id=NEGS6AAGBP4WRJJ6TXEBNHCV2FXW2OBMRWAWL4RNY4YZZZSI3ZLQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.504778116 (113h30m17.056248577s) [noise=true , address=45.91.116.13:4154 , piece_id=DSC3N7KGSUSEVCEII65IQMYLHEUC5U52NNSREC32Y3Q3MDNKWREQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.511639541 (113h30m17.063110002s) [noise=true , address=77.90.31.10:4517 , piece_id=GDOCT6DE4NL56DWKOS3JNSQVIVE342SDIZ5YR7XPNRXYPOQYDXCQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.529697226 (113h30m17.081167687s) [noise=true , address=77.90.31.17:4771 , piece_id=ARX2MBQVJSPWKWS2AHE2QBBWNREGMUORBQPNFQJHJDWBT2FCDPAA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:45.367038063 (113h30m43.918508514s) [noise=true , address=145.79.227.26:4653 , piece_id=EEKXCCTYRNCRJFVM2QBQVQADQJ2YVDCDOXFIFUP4BYUEVYDGWZQA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection Nov 14 01:34:45.367162339 (113h30m43.91863279s) [noise=true , address=145.79.227.14:4840 , piece_id=3DEHNGEWEG4GA5YTLKGDV4BYUJ7Z6KCVLM343ZB5MYZWEMOASS7A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection Nov 14 01:34:45.367201353 (113h30m43.918671804s) [noise=true , address=145.79.227.9:4366 , piece_id=GXQSDBFPFZPZ352A5KBNQJS5A32C7ZO522AHCHO3A27P4LAMUFZQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection Nov 14 01:34:45.367247291 (113h30m43.918717742s) [noise=true , address=145.79.227.20:4439 , piece_id=FEDAGRFBHNL6ES3XTCRWPEI4NN2CMMDJIVNW5WOG3KYD7VXFXA7Q, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection; write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection Nov 14 01:34:45.367425070 (113h30m43.918895531s) [noise=true , address=145.79.227.13:4804 , piece_id=NZL7G7WFABHCYGW63IPFVWP45AMPZRDYLDF4JNKJHK5PLSHYHLSQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection Nov 14 01:34:45.367779144 (113h30m43.919249605s) [noise=true , address=145.79.227.30:4763 , piece_id=IW3HMSEGIX2TFUY277OSEF4BULPDWA3AO3MNJ7OCUULORPLHNWOQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:50566->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:34:45.367825933 (113h30m43.919296394s) [noise=true , address=145.79.227.16:4912 , piece_id=SKK6OIUPSWXXD6QS5ZCVVORVSFXXOE5UJH7POHTKYY6PU4LX2JGA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection Nov 14 01:34:45.368025623 (113h30m43.919496114s) [noise=true , address=145.79.227.21:4473 , piece_id=DIGA6L2TCAOGHAGUY5K4T3R7H66CF3JLXBXKMC7TPTXVKBCLDJBA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: read tcp 188.166.225.170:49576->145.79.227.21:4473: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:34:45.368068755 (113h30m43.919539206s) [seg_pos={0 0}, upload=99885]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 14 01:34:45.634359530 (113h30m44.185829991s) [upload=99885]: Done uploading segments... Nov 14 01:34:45.634431096 (113h30m44.185901557s) [upload=99885]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.984077015s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.782855355s trace: https://jaeger.dev.storj.io/trace/5bc86ccfe33cf550 FAIL (2025-11-14 01:43:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ce_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288397607 (113h30m2.839868078s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288507285 (113h30m2.839977746s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.289077772 (113h30m2.840548233s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290074169 (113h30m2.84154464s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290622854 (113h30m2.842093345s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292062385 (113h30m2.843532866s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292958642 (113h30m2.844429123s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.294563758 (113h30m2.846034229s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.296755000 (113h30m2.848225462s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.298618379 (113h30m2.85008884s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.300188338 (113h30m2.85165881s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.301125483 (113h30m2.852595944s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302545266 (113h30m2.854015727s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302898339 (113h30m2.8543688s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.303825163 (113h30m2.855295624s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.316046324 (113h30m2.867516785s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.318180529 (113h30m2.86965099s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:05.581380111 (113h30m4.132850592s) [upload=99881]: Next returned nil segment Nov 14 01:34:05.581407382 (113h30m4.132877844s) [upload=99881]: Waiting for error group managing segments... Nov 14 01:34:05.921914348 (113h30m4.473384809s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.005586503 (113h30m4.557056964s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.026850744 (113h30m4.578321205s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.121926626 (113h30m4.673397087s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.235634986 (113h30m4.787105437s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.277897963 (113h30m4.829368424s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.283628915 (113h30m4.835099376s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.287808744 (113h30m4.839279205s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.300041498 (113h30m4.851511949s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.304400718 (113h30m4.855871179s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.325040680 (113h30m4.876511141s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.331007762 (113h30m4.882478223s) [noise=true , address=77.90.31.12:4611 , piece_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.481058726 (113h30m5.032529207s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:35:02.598238012 (113h31m1.149708493s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection Nov 14 01:35:02.599889276 (113h31m1.151359737s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:145.79.227.52:4455): protocol: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection Nov 14 01:35:02.600041686 (113h31m1.151512147s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:127WWD5hr1ZBkRLCHz3jb7kpqqnxPS6VuZ1MyQDmitFbhTSbJNK, address:145.79.227.49:4344): protocol: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection Nov 14 01:35:02.600244733 (113h31m1.151715204s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:145.79.227.35:4488): protocol: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection Nov 14 01:35:02.600569551 (113h31m1.152040022s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12h6MND3rBadG4EjUP18S7izyj23cVBLaB5drFsuNGhcxwAYXSA, address:145.79.227.17:4983): protocol: read tcp 188.166.225.170:33042->145.79.227.17:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:35:02.600603536 (113h31m1.152073997s) [seg_pos={0 0}, upload=99881]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 01:35:02.847367167 (113h31m1.398837648s) [upload=99881]: Done uploading segments... Nov 14 01:35:02.847427322 (113h31m1.398897783s) [upload=99881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.782888707s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X.............X...................X...........] duration: 5.924037079s trace: https://jaeger.dev.storj.io/trace/4377cd58781b9764 FAIL (2025-11-14 01:42:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== DXCQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351147808 (113h30m14.902618269s) [noise=true , address=77.90.31.13:4628 , piece_id=WHP562KP7DABPK6XQT5F43QGNNCDFXYYYP775VGZLXIG2UEWKK4A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351381713 (113h30m14.902852174s) [noise=true , address=145.79.227.30:4763 , piece_id=IW3HMSEGIX2TFUY277OSEF4BULPDWA3AO3MNJ7OCUULORPLHNWOQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351673048 (113h30m14.903143509s) [noise=true , address=145.79.227.26:4653 , piece_id=EEKXCCTYRNCRJFVM2QBQVQADQJ2YVDCDOXFIFUP4BYUEVYDGWZQA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351818645 (113h30m14.903289106s) [noise=true , address=145.79.227.14:4840 , piece_id=3DEHNGEWEG4GA5YTLKGDV4BYUJ7Z6KCVLM343ZB5MYZWEMOASS7A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352062729 (113h30m14.903533191s) [noise=true , address=45.91.116.13:4154 , piece_id=DSC3N7KGSUSEVCEII65IQMYLHEUC5U52NNSREC32Y3Q3MDNKWREQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352224859 (113h30m14.90369532s) [noise=true , address=77.90.31.18:4807 , piece_id=TD3XBJUZ6MM5CB7NMYEIZWQC63U2PZZJPYHIM3OOU7PE3Y7KGCIQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352479703 (113h30m14.903950165s) [noise=true , address=77.90.31.17:4771 , piece_id=ARX2MBQVJSPWKWS2AHE2QBBWNREGMUORBQPNFQJHJDWBT2FCDPAA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352614791 (113h30m14.904085342s) [noise=true , address=145.79.227.21:4473 , piece_id=DIGA6L2TCAOGHAGUY5K4T3R7H66CF3JLXBXKMC7TPTXVKBCLDJBA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352904944 (113h30m14.904375405s) [noise=true , address=45.91.116.6:4973 , piece_id=NEGS6AAGBP4WRJJ6TXEBNHCV2FXW2OBMRWAWL4RNY4YZZZSI3ZLQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.353329152 (113h30m14.904799613s) [noise=true , address=77.90.31.12:4592 , piece_id=ET6LSZ6LXAAAPVJLBLL4UOIF6SKOKDCKXEXKIQ4OS3RD5J5QB2UQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.353685219 (113h30m14.90515568s) [noise=true , address=145.79.227.16:4912 , piece_id=SKK6OIUPSWXXD6QS5ZCVVORVSFXXOE5UJH7POHTKYY6PU4LX2JGA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354114446 (113h30m14.905584907s) [noise=true , address=45.91.116.12:4225 , piece_id=RMDFCX4JGDRPQ46PPTYAWJE7NIYVDE3J2LCKREBFXHR7I6U2G52A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354334907 (113h30m14.905805368s) [noise=true , address=145.79.227.13:4804 , piece_id=NZL7G7WFABHCYGW63IPFVWP45AMPZRDYLDF4JNKJHK5PLSHYHLSQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354804771 (113h30m14.906275222s) [noise=true , address=145.79.227.20:4439 , piece_id=FEDAGRFBHNL6ES3XTCRWPEI4NN2CMMDJIVNW5WOG3KYD7VXFXA7Q, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:18.090813414 (113h30m16.642283865s) [upload=99885]: Next returned nil segment Nov 14 01:34:18.090832882 (113h30m16.642303333s) [upload=99885]: Waiting for error group managing segments... Nov 14 01:34:18.376699271 (113h30m16.928169732s) [noise=true , address=45.91.116.12:4225 , piece_id=RMDFCX4JGDRPQ46PPTYAWJE7NIYVDE3J2LCKREBFXHR7I6U2G52A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.449067032 (113h30m17.000537493s) [noise=true , address=77.90.31.18:4807 , piece_id=TD3XBJUZ6MM5CB7NMYEIZWQC63U2PZZJPYHIM3OOU7PE3Y7KGCIQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.476540927 (113h30m17.028011378s) [noise=true , address=77.90.31.13:4628 , piece_id=WHP562KP7DABPK6XQT5F43QGNNCDFXYYYP775VGZLXIG2UEWKK4A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.502738892 (113h30m17.054209343s) [noise=true , address=77.90.31.12:4592 , piece_id=ET6LSZ6LXAAAPVJLBLL4UOIF6SKOKDCKXEXKIQ4OS3RD5J5QB2UQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.504520516 (113h30m17.055990977s) [noise=true , address=45.91.116.6:4973 , piece_id=NEGS6AAGBP4WRJJ6TXEBNHCV2FXW2OBMRWAWL4RNY4YZZZSI3ZLQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.504778116 (113h30m17.056248577s) [noise=true , address=45.91.116.13:4154 , piece_id=DSC3N7KGSUSEVCEII65IQMYLHEUC5U52NNSREC32Y3Q3MDNKWREQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.511639541 (113h30m17.063110002s) [noise=true , address=77.90.31.10:4517 , piece_id=GDOCT6DE4NL56DWKOS3JNSQVIVE342SDIZ5YR7XPNRXYPOQYDXCQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.529697226 (113h30m17.081167687s) [noise=true , address=77.90.31.17:4771 , piece_id=ARX2MBQVJSPWKWS2AHE2QBBWNREGMUORBQPNFQJHJDWBT2FCDPAA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:45.367038063 (113h30m43.918508514s) [noise=true , address=145.79.227.26:4653 , piece_id=EEKXCCTYRNCRJFVM2QBQVQADQJ2YVDCDOXFIFUP4BYUEVYDGWZQA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection Nov 14 01:34:45.367162339 (113h30m43.91863279s) [noise=true , address=145.79.227.14:4840 , piece_id=3DEHNGEWEG4GA5YTLKGDV4BYUJ7Z6KCVLM343ZB5MYZWEMOASS7A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection Nov 14 01:34:45.367201353 (113h30m43.918671804s) [noise=true , address=145.79.227.9:4366 , piece_id=GXQSDBFPFZPZ352A5KBNQJS5A32C7ZO522AHCHO3A27P4LAMUFZQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection Nov 14 01:34:45.367247291 (113h30m43.918717742s) [noise=true , address=145.79.227.20:4439 , piece_id=FEDAGRFBHNL6ES3XTCRWPEI4NN2CMMDJIVNW5WOG3KYD7VXFXA7Q, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection; write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection Nov 14 01:34:45.367425070 (113h30m43.918895531s) [noise=true , address=145.79.227.13:4804 , piece_id=NZL7G7WFABHCYGW63IPFVWP45AMPZRDYLDF4JNKJHK5PLSHYHLSQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection Nov 14 01:34:45.367779144 (113h30m43.919249605s) [noise=true , address=145.79.227.30:4763 , piece_id=IW3HMSEGIX2TFUY277OSEF4BULPDWA3AO3MNJ7OCUULORPLHNWOQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:50566->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:34:45.367825933 (113h30m43.919296394s) [noise=true , address=145.79.227.16:4912 , piece_id=SKK6OIUPSWXXD6QS5ZCVVORVSFXXOE5UJH7POHTKYY6PU4LX2JGA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection Nov 14 01:34:45.368025623 (113h30m43.919496114s) [noise=true , address=145.79.227.21:4473 , piece_id=DIGA6L2TCAOGHAGUY5K4T3R7H66CF3JLXBXKMC7TPTXVKBCLDJBA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: read tcp 188.166.225.170:49576->145.79.227.21:4473: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:34:45.368068755 (113h30m43.919539206s) [seg_pos={0 0}, upload=99885]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 14 01:34:45.634359530 (113h30m44.185829991s) [upload=99885]: Done uploading segments... Nov 14 01:34:45.634431096 (113h30m44.185901557s) [upload=99885]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.924071975s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.699471488s trace: https://jaeger.dev.storj.io/trace/5bc86ccfe33cf550 FAIL (2025-11-14 01:42:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ce_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288397607 (113h30m2.839868078s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288507285 (113h30m2.839977746s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.289077772 (113h30m2.840548233s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290074169 (113h30m2.84154464s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290622854 (113h30m2.842093345s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292062385 (113h30m2.843532866s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292958642 (113h30m2.844429123s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.294563758 (113h30m2.846034229s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.296755000 (113h30m2.848225462s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.298618379 (113h30m2.85008884s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.300188338 (113h30m2.85165881s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.301125483 (113h30m2.852595944s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302545266 (113h30m2.854015727s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302898339 (113h30m2.8543688s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.303825163 (113h30m2.855295624s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.316046324 (113h30m2.867516785s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.318180529 (113h30m2.86965099s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:05.581380111 (113h30m4.132850592s) [upload=99881]: Next returned nil segment Nov 14 01:34:05.581407382 (113h30m4.132877844s) [upload=99881]: Waiting for error group managing segments... Nov 14 01:34:05.921914348 (113h30m4.473384809s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.005586503 (113h30m4.557056964s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.026850744 (113h30m4.578321205s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.121926626 (113h30m4.673397087s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.235634986 (113h30m4.787105437s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.277897963 (113h30m4.829368424s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.283628915 (113h30m4.835099376s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.287808744 (113h30m4.839279205s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.300041498 (113h30m4.851511949s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.304400718 (113h30m4.855871179s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.325040680 (113h30m4.876511141s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.331007762 (113h30m4.882478223s) [noise=true , address=77.90.31.12:4611 , piece_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.481058726 (113h30m5.032529207s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:35:02.598238012 (113h31m1.149708493s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection Nov 14 01:35:02.599889276 (113h31m1.151359737s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:145.79.227.52:4455): protocol: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection Nov 14 01:35:02.600041686 (113h31m1.151512147s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:127WWD5hr1ZBkRLCHz3jb7kpqqnxPS6VuZ1MyQDmitFbhTSbJNK, address:145.79.227.49:4344): protocol: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection Nov 14 01:35:02.600244733 (113h31m1.151715204s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:145.79.227.35:4488): protocol: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection Nov 14 01:35:02.600569551 (113h31m1.152040022s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12h6MND3rBadG4EjUP18S7izyj23cVBLaB5drFsuNGhcxwAYXSA, address:145.79.227.17:4983): protocol: read tcp 188.166.225.170:33042->145.79.227.17:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:35:02.600603536 (113h31m1.152073997s) [seg_pos={0 0}, upload=99881]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 01:35:02.847367167 (113h31m1.398837648s) [upload=99881]: Done uploading segments... Nov 14 01:35:02.847427322 (113h31m1.398897783s) [upload=99881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.699492558s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............X.............X...................X..........] duration: 5.956641044s trace: https://jaeger.dev.storj.io/trace/4377cd58781b9764 FAIL (2025-11-14 01:42:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== DXCQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351147808 (113h30m14.902618269s) [noise=true , address=77.90.31.13:4628 , piece_id=WHP562KP7DABPK6XQT5F43QGNNCDFXYYYP775VGZLXIG2UEWKK4A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351381713 (113h30m14.902852174s) [noise=true , address=145.79.227.30:4763 , piece_id=IW3HMSEGIX2TFUY277OSEF4BULPDWA3AO3MNJ7OCUULORPLHNWOQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351673048 (113h30m14.903143509s) [noise=true , address=145.79.227.26:4653 , piece_id=EEKXCCTYRNCRJFVM2QBQVQADQJ2YVDCDOXFIFUP4BYUEVYDGWZQA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351818645 (113h30m14.903289106s) [noise=true , address=145.79.227.14:4840 , piece_id=3DEHNGEWEG4GA5YTLKGDV4BYUJ7Z6KCVLM343ZB5MYZWEMOASS7A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352062729 (113h30m14.903533191s) [noise=true , address=45.91.116.13:4154 , piece_id=DSC3N7KGSUSEVCEII65IQMYLHEUC5U52NNSREC32Y3Q3MDNKWREQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352224859 (113h30m14.90369532s) [noise=true , address=77.90.31.18:4807 , piece_id=TD3XBJUZ6MM5CB7NMYEIZWQC63U2PZZJPYHIM3OOU7PE3Y7KGCIQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352479703 (113h30m14.903950165s) [noise=true , address=77.90.31.17:4771 , piece_id=ARX2MBQVJSPWKWS2AHE2QBBWNREGMUORBQPNFQJHJDWBT2FCDPAA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352614791 (113h30m14.904085342s) [noise=true , address=145.79.227.21:4473 , piece_id=DIGA6L2TCAOGHAGUY5K4T3R7H66CF3JLXBXKMC7TPTXVKBCLDJBA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352904944 (113h30m14.904375405s) [noise=true , address=45.91.116.6:4973 , piece_id=NEGS6AAGBP4WRJJ6TXEBNHCV2FXW2OBMRWAWL4RNY4YZZZSI3ZLQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.353329152 (113h30m14.904799613s) [noise=true , address=77.90.31.12:4592 , piece_id=ET6LSZ6LXAAAPVJLBLL4UOIF6SKOKDCKXEXKIQ4OS3RD5J5QB2UQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.353685219 (113h30m14.90515568s) [noise=true , address=145.79.227.16:4912 , piece_id=SKK6OIUPSWXXD6QS5ZCVVORVSFXXOE5UJH7POHTKYY6PU4LX2JGA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354114446 (113h30m14.905584907s) [noise=true , address=45.91.116.12:4225 , piece_id=RMDFCX4JGDRPQ46PPTYAWJE7NIYVDE3J2LCKREBFXHR7I6U2G52A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354334907 (113h30m14.905805368s) [noise=true , address=145.79.227.13:4804 , piece_id=NZL7G7WFABHCYGW63IPFVWP45AMPZRDYLDF4JNKJHK5PLSHYHLSQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354804771 (113h30m14.906275222s) [noise=true , address=145.79.227.20:4439 , piece_id=FEDAGRFBHNL6ES3XTCRWPEI4NN2CMMDJIVNW5WOG3KYD7VXFXA7Q, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:18.090813414 (113h30m16.642283865s) [upload=99885]: Next returned nil segment Nov 14 01:34:18.090832882 (113h30m16.642303333s) [upload=99885]: Waiting for error group managing segments... Nov 14 01:34:18.376699271 (113h30m16.928169732s) [noise=true , address=45.91.116.12:4225 , piece_id=RMDFCX4JGDRPQ46PPTYAWJE7NIYVDE3J2LCKREBFXHR7I6U2G52A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.449067032 (113h30m17.000537493s) [noise=true , address=77.90.31.18:4807 , piece_id=TD3XBJUZ6MM5CB7NMYEIZWQC63U2PZZJPYHIM3OOU7PE3Y7KGCIQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.476540927 (113h30m17.028011378s) [noise=true , address=77.90.31.13:4628 , piece_id=WHP562KP7DABPK6XQT5F43QGNNCDFXYYYP775VGZLXIG2UEWKK4A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.502738892 (113h30m17.054209343s) [noise=true , address=77.90.31.12:4592 , piece_id=ET6LSZ6LXAAAPVJLBLL4UOIF6SKOKDCKXEXKIQ4OS3RD5J5QB2UQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.504520516 (113h30m17.055990977s) [noise=true , address=45.91.116.6:4973 , piece_id=NEGS6AAGBP4WRJJ6TXEBNHCV2FXW2OBMRWAWL4RNY4YZZZSI3ZLQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.504778116 (113h30m17.056248577s) [noise=true , address=45.91.116.13:4154 , piece_id=DSC3N7KGSUSEVCEII65IQMYLHEUC5U52NNSREC32Y3Q3MDNKWREQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.511639541 (113h30m17.063110002s) [noise=true , address=77.90.31.10:4517 , piece_id=GDOCT6DE4NL56DWKOS3JNSQVIVE342SDIZ5YR7XPNRXYPOQYDXCQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.529697226 (113h30m17.081167687s) [noise=true , address=77.90.31.17:4771 , piece_id=ARX2MBQVJSPWKWS2AHE2QBBWNREGMUORBQPNFQJHJDWBT2FCDPAA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:45.367038063 (113h30m43.918508514s) [noise=true , address=145.79.227.26:4653 , piece_id=EEKXCCTYRNCRJFVM2QBQVQADQJ2YVDCDOXFIFUP4BYUEVYDGWZQA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection Nov 14 01:34:45.367162339 (113h30m43.91863279s) [noise=true , address=145.79.227.14:4840 , piece_id=3DEHNGEWEG4GA5YTLKGDV4BYUJ7Z6KCVLM343ZB5MYZWEMOASS7A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection Nov 14 01:34:45.367201353 (113h30m43.918671804s) [noise=true , address=145.79.227.9:4366 , piece_id=GXQSDBFPFZPZ352A5KBNQJS5A32C7ZO522AHCHO3A27P4LAMUFZQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection Nov 14 01:34:45.367247291 (113h30m43.918717742s) [noise=true , address=145.79.227.20:4439 , piece_id=FEDAGRFBHNL6ES3XTCRWPEI4NN2CMMDJIVNW5WOG3KYD7VXFXA7Q, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection; write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection Nov 14 01:34:45.367425070 (113h30m43.918895531s) [noise=true , address=145.79.227.13:4804 , piece_id=NZL7G7WFABHCYGW63IPFVWP45AMPZRDYLDF4JNKJHK5PLSHYHLSQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection Nov 14 01:34:45.367779144 (113h30m43.919249605s) [noise=true , address=145.79.227.30:4763 , piece_id=IW3HMSEGIX2TFUY277OSEF4BULPDWA3AO3MNJ7OCUULORPLHNWOQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:50566->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:34:45.367825933 (113h30m43.919296394s) [noise=true , address=145.79.227.16:4912 , piece_id=SKK6OIUPSWXXD6QS5ZCVVORVSFXXOE5UJH7POHTKYY6PU4LX2JGA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection Nov 14 01:34:45.368025623 (113h30m43.919496114s) [noise=true , address=145.79.227.21:4473 , piece_id=DIGA6L2TCAOGHAGUY5K4T3R7H66CF3JLXBXKMC7TPTXVKBCLDJBA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: read tcp 188.166.225.170:49576->145.79.227.21:4473: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:34:45.368068755 (113h30m43.919539206s) [seg_pos={0 0}, upload=99885]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 14 01:34:45.634359530 (113h30m44.185829991s) [upload=99885]: Done uploading segments... Nov 14 01:34:45.634431096 (113h30m44.185901557s) [upload=99885]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.956694805s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................X.............X...................X.........] duration: 5.834805507s trace: https://jaeger.dev.storj.io/trace/4377cd58781b9764 FAIL (2025-11-14 01:41:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== DXCQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351147808 (113h30m14.902618269s) [noise=true , address=77.90.31.13:4628 , piece_id=WHP562KP7DABPK6XQT5F43QGNNCDFXYYYP775VGZLXIG2UEWKK4A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351381713 (113h30m14.902852174s) [noise=true , address=145.79.227.30:4763 , piece_id=IW3HMSEGIX2TFUY277OSEF4BULPDWA3AO3MNJ7OCUULORPLHNWOQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351673048 (113h30m14.903143509s) [noise=true , address=145.79.227.26:4653 , piece_id=EEKXCCTYRNCRJFVM2QBQVQADQJ2YVDCDOXFIFUP4BYUEVYDGWZQA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351818645 (113h30m14.903289106s) [noise=true , address=145.79.227.14:4840 , piece_id=3DEHNGEWEG4GA5YTLKGDV4BYUJ7Z6KCVLM343ZB5MYZWEMOASS7A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352062729 (113h30m14.903533191s) [noise=true , address=45.91.116.13:4154 , piece_id=DSC3N7KGSUSEVCEII65IQMYLHEUC5U52NNSREC32Y3Q3MDNKWREQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352224859 (113h30m14.90369532s) [noise=true , address=77.90.31.18:4807 , piece_id=TD3XBJUZ6MM5CB7NMYEIZWQC63U2PZZJPYHIM3OOU7PE3Y7KGCIQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352479703 (113h30m14.903950165s) [noise=true , address=77.90.31.17:4771 , piece_id=ARX2MBQVJSPWKWS2AHE2QBBWNREGMUORBQPNFQJHJDWBT2FCDPAA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352614791 (113h30m14.904085342s) [noise=true , address=145.79.227.21:4473 , piece_id=DIGA6L2TCAOGHAGUY5K4T3R7H66CF3JLXBXKMC7TPTXVKBCLDJBA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352904944 (113h30m14.904375405s) [noise=true , address=45.91.116.6:4973 , piece_id=NEGS6AAGBP4WRJJ6TXEBNHCV2FXW2OBMRWAWL4RNY4YZZZSI3ZLQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.353329152 (113h30m14.904799613s) [noise=true , address=77.90.31.12:4592 , piece_id=ET6LSZ6LXAAAPVJLBLL4UOIF6SKOKDCKXEXKIQ4OS3RD5J5QB2UQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.353685219 (113h30m14.90515568s) [noise=true , address=145.79.227.16:4912 , piece_id=SKK6OIUPSWXXD6QS5ZCVVORVSFXXOE5UJH7POHTKYY6PU4LX2JGA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354114446 (113h30m14.905584907s) [noise=true , address=45.91.116.12:4225 , piece_id=RMDFCX4JGDRPQ46PPTYAWJE7NIYVDE3J2LCKREBFXHR7I6U2G52A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354334907 (113h30m14.905805368s) [noise=true , address=145.79.227.13:4804 , piece_id=NZL7G7WFABHCYGW63IPFVWP45AMPZRDYLDF4JNKJHK5PLSHYHLSQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354804771 (113h30m14.906275222s) [noise=true , address=145.79.227.20:4439 , piece_id=FEDAGRFBHNL6ES3XTCRWPEI4NN2CMMDJIVNW5WOG3KYD7VXFXA7Q, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:18.090813414 (113h30m16.642283865s) [upload=99885]: Next returned nil segment Nov 14 01:34:18.090832882 (113h30m16.642303333s) [upload=99885]: Waiting for error group managing segments... Nov 14 01:34:18.376699271 (113h30m16.928169732s) [noise=true , address=45.91.116.12:4225 , piece_id=RMDFCX4JGDRPQ46PPTYAWJE7NIYVDE3J2LCKREBFXHR7I6U2G52A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.449067032 (113h30m17.000537493s) [noise=true , address=77.90.31.18:4807 , piece_id=TD3XBJUZ6MM5CB7NMYEIZWQC63U2PZZJPYHIM3OOU7PE3Y7KGCIQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.476540927 (113h30m17.028011378s) [noise=true , address=77.90.31.13:4628 , piece_id=WHP562KP7DABPK6XQT5F43QGNNCDFXYYYP775VGZLXIG2UEWKK4A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.502738892 (113h30m17.054209343s) [noise=true , address=77.90.31.12:4592 , piece_id=ET6LSZ6LXAAAPVJLBLL4UOIF6SKOKDCKXEXKIQ4OS3RD5J5QB2UQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.504520516 (113h30m17.055990977s) [noise=true , address=45.91.116.6:4973 , piece_id=NEGS6AAGBP4WRJJ6TXEBNHCV2FXW2OBMRWAWL4RNY4YZZZSI3ZLQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.504778116 (113h30m17.056248577s) [noise=true , address=45.91.116.13:4154 , piece_id=DSC3N7KGSUSEVCEII65IQMYLHEUC5U52NNSREC32Y3Q3MDNKWREQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.511639541 (113h30m17.063110002s) [noise=true , address=77.90.31.10:4517 , piece_id=GDOCT6DE4NL56DWKOS3JNSQVIVE342SDIZ5YR7XPNRXYPOQYDXCQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.529697226 (113h30m17.081167687s) [noise=true , address=77.90.31.17:4771 , piece_id=ARX2MBQVJSPWKWS2AHE2QBBWNREGMUORBQPNFQJHJDWBT2FCDPAA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:45.367038063 (113h30m43.918508514s) [noise=true , address=145.79.227.26:4653 , piece_id=EEKXCCTYRNCRJFVM2QBQVQADQJ2YVDCDOXFIFUP4BYUEVYDGWZQA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection Nov 14 01:34:45.367162339 (113h30m43.91863279s) [noise=true , address=145.79.227.14:4840 , piece_id=3DEHNGEWEG4GA5YTLKGDV4BYUJ7Z6KCVLM343ZB5MYZWEMOASS7A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection Nov 14 01:34:45.367201353 (113h30m43.918671804s) [noise=true , address=145.79.227.9:4366 , piece_id=GXQSDBFPFZPZ352A5KBNQJS5A32C7ZO522AHCHO3A27P4LAMUFZQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection Nov 14 01:34:45.367247291 (113h30m43.918717742s) [noise=true , address=145.79.227.20:4439 , piece_id=FEDAGRFBHNL6ES3XTCRWPEI4NN2CMMDJIVNW5WOG3KYD7VXFXA7Q, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection; write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection Nov 14 01:34:45.367425070 (113h30m43.918895531s) [noise=true , address=145.79.227.13:4804 , piece_id=NZL7G7WFABHCYGW63IPFVWP45AMPZRDYLDF4JNKJHK5PLSHYHLSQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection Nov 14 01:34:45.367779144 (113h30m43.919249605s) [noise=true , address=145.79.227.30:4763 , piece_id=IW3HMSEGIX2TFUY277OSEF4BULPDWA3AO3MNJ7OCUULORPLHNWOQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:50566->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:34:45.367825933 (113h30m43.919296394s) [noise=true , address=145.79.227.16:4912 , piece_id=SKK6OIUPSWXXD6QS5ZCVVORVSFXXOE5UJH7POHTKYY6PU4LX2JGA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection Nov 14 01:34:45.368025623 (113h30m43.919496114s) [noise=true , address=145.79.227.21:4473 , piece_id=DIGA6L2TCAOGHAGUY5K4T3R7H66CF3JLXBXKMC7TPTXVKBCLDJBA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: read tcp 188.166.225.170:49576->145.79.227.21:4473: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:34:45.368068755 (113h30m43.919539206s) [seg_pos={0 0}, upload=99885]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 14 01:34:45.634359530 (113h30m44.185829991s) [upload=99885]: Done uploading segments... Nov 14 01:34:45.634431096 (113h30m44.185901557s) [upload=99885]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.834831658s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.712637498s trace: https://jaeger.dev.storj.io/trace/5bc86ccfe33cf550 FAIL (2025-11-14 01:41:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ce_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288397607 (113h30m2.839868078s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288507285 (113h30m2.839977746s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.289077772 (113h30m2.840548233s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290074169 (113h30m2.84154464s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290622854 (113h30m2.842093345s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292062385 (113h30m2.843532866s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292958642 (113h30m2.844429123s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.294563758 (113h30m2.846034229s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.296755000 (113h30m2.848225462s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.298618379 (113h30m2.85008884s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.300188338 (113h30m2.85165881s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.301125483 (113h30m2.852595944s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302545266 (113h30m2.854015727s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302898339 (113h30m2.8543688s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.303825163 (113h30m2.855295624s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.316046324 (113h30m2.867516785s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.318180529 (113h30m2.86965099s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:05.581380111 (113h30m4.132850592s) [upload=99881]: Next returned nil segment Nov 14 01:34:05.581407382 (113h30m4.132877844s) [upload=99881]: Waiting for error group managing segments... Nov 14 01:34:05.921914348 (113h30m4.473384809s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.005586503 (113h30m4.557056964s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.026850744 (113h30m4.578321205s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.121926626 (113h30m4.673397087s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.235634986 (113h30m4.787105437s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.277897963 (113h30m4.829368424s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.283628915 (113h30m4.835099376s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.287808744 (113h30m4.839279205s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.300041498 (113h30m4.851511949s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.304400718 (113h30m4.855871179s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.325040680 (113h30m4.876511141s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.331007762 (113h30m4.882478223s) [noise=true , address=77.90.31.12:4611 , piece_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.481058726 (113h30m5.032529207s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:35:02.598238012 (113h31m1.149708493s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection Nov 14 01:35:02.599889276 (113h31m1.151359737s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:145.79.227.52:4455): protocol: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection Nov 14 01:35:02.600041686 (113h31m1.151512147s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:127WWD5hr1ZBkRLCHz3jb7kpqqnxPS6VuZ1MyQDmitFbhTSbJNK, address:145.79.227.49:4344): protocol: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection Nov 14 01:35:02.600244733 (113h31m1.151715204s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:145.79.227.35:4488): protocol: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection Nov 14 01:35:02.600569551 (113h31m1.152040022s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12h6MND3rBadG4EjUP18S7izyj23cVBLaB5drFsuNGhcxwAYXSA, address:145.79.227.17:4983): protocol: read tcp 188.166.225.170:33042->145.79.227.17:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:35:02.600603536 (113h31m1.152073997s) [seg_pos={0 0}, upload=99881]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 01:35:02.847367167 (113h31m1.398837648s) [upload=99881]: Done uploading segments... Nov 14 01:35:02.847427322 (113h31m1.398897783s) [upload=99881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.712817762s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................X.............X...................X........] duration: 7.265415719s trace: https://jaeger.dev.storj.io/trace/4377cd58781b9764 FAIL (2025-11-14 01:40:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== DXCQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351147808 (113h30m14.902618269s) [noise=true , address=77.90.31.13:4628 , piece_id=WHP562KP7DABPK6XQT5F43QGNNCDFXYYYP775VGZLXIG2UEWKK4A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351381713 (113h30m14.902852174s) [noise=true , address=145.79.227.30:4763 , piece_id=IW3HMSEGIX2TFUY277OSEF4BULPDWA3AO3MNJ7OCUULORPLHNWOQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351673048 (113h30m14.903143509s) [noise=true , address=145.79.227.26:4653 , piece_id=EEKXCCTYRNCRJFVM2QBQVQADQJ2YVDCDOXFIFUP4BYUEVYDGWZQA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351818645 (113h30m14.903289106s) [noise=true , address=145.79.227.14:4840 , piece_id=3DEHNGEWEG4GA5YTLKGDV4BYUJ7Z6KCVLM343ZB5MYZWEMOASS7A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352062729 (113h30m14.903533191s) [noise=true , address=45.91.116.13:4154 , piece_id=DSC3N7KGSUSEVCEII65IQMYLHEUC5U52NNSREC32Y3Q3MDNKWREQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352224859 (113h30m14.90369532s) [noise=true , address=77.90.31.18:4807 , piece_id=TD3XBJUZ6MM5CB7NMYEIZWQC63U2PZZJPYHIM3OOU7PE3Y7KGCIQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352479703 (113h30m14.903950165s) [noise=true , address=77.90.31.17:4771 , piece_id=ARX2MBQVJSPWKWS2AHE2QBBWNREGMUORBQPNFQJHJDWBT2FCDPAA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352614791 (113h30m14.904085342s) [noise=true , address=145.79.227.21:4473 , piece_id=DIGA6L2TCAOGHAGUY5K4T3R7H66CF3JLXBXKMC7TPTXVKBCLDJBA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352904944 (113h30m14.904375405s) [noise=true , address=45.91.116.6:4973 , piece_id=NEGS6AAGBP4WRJJ6TXEBNHCV2FXW2OBMRWAWL4RNY4YZZZSI3ZLQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.353329152 (113h30m14.904799613s) [noise=true , address=77.90.31.12:4592 , piece_id=ET6LSZ6LXAAAPVJLBLL4UOIF6SKOKDCKXEXKIQ4OS3RD5J5QB2UQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.353685219 (113h30m14.90515568s) [noise=true , address=145.79.227.16:4912 , piece_id=SKK6OIUPSWXXD6QS5ZCVVORVSFXXOE5UJH7POHTKYY6PU4LX2JGA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354114446 (113h30m14.905584907s) [noise=true , address=45.91.116.12:4225 , piece_id=RMDFCX4JGDRPQ46PPTYAWJE7NIYVDE3J2LCKREBFXHR7I6U2G52A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354334907 (113h30m14.905805368s) [noise=true , address=145.79.227.13:4804 , piece_id=NZL7G7WFABHCYGW63IPFVWP45AMPZRDYLDF4JNKJHK5PLSHYHLSQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354804771 (113h30m14.906275222s) [noise=true , address=145.79.227.20:4439 , piece_id=FEDAGRFBHNL6ES3XTCRWPEI4NN2CMMDJIVNW5WOG3KYD7VXFXA7Q, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:18.090813414 (113h30m16.642283865s) [upload=99885]: Next returned nil segment Nov 14 01:34:18.090832882 (113h30m16.642303333s) [upload=99885]: Waiting for error group managing segments... Nov 14 01:34:18.376699271 (113h30m16.928169732s) [noise=true , address=45.91.116.12:4225 , piece_id=RMDFCX4JGDRPQ46PPTYAWJE7NIYVDE3J2LCKREBFXHR7I6U2G52A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.449067032 (113h30m17.000537493s) [noise=true , address=77.90.31.18:4807 , piece_id=TD3XBJUZ6MM5CB7NMYEIZWQC63U2PZZJPYHIM3OOU7PE3Y7KGCIQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.476540927 (113h30m17.028011378s) [noise=true , address=77.90.31.13:4628 , piece_id=WHP562KP7DABPK6XQT5F43QGNNCDFXYYYP775VGZLXIG2UEWKK4A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.502738892 (113h30m17.054209343s) [noise=true , address=77.90.31.12:4592 , piece_id=ET6LSZ6LXAAAPVJLBLL4UOIF6SKOKDCKXEXKIQ4OS3RD5J5QB2UQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.504520516 (113h30m17.055990977s) [noise=true , address=45.91.116.6:4973 , piece_id=NEGS6AAGBP4WRJJ6TXEBNHCV2FXW2OBMRWAWL4RNY4YZZZSI3ZLQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.504778116 (113h30m17.056248577s) [noise=true , address=45.91.116.13:4154 , piece_id=DSC3N7KGSUSEVCEII65IQMYLHEUC5U52NNSREC32Y3Q3MDNKWREQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.511639541 (113h30m17.063110002s) [noise=true , address=77.90.31.10:4517 , piece_id=GDOCT6DE4NL56DWKOS3JNSQVIVE342SDIZ5YR7XPNRXYPOQYDXCQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.529697226 (113h30m17.081167687s) [noise=true , address=77.90.31.17:4771 , piece_id=ARX2MBQVJSPWKWS2AHE2QBBWNREGMUORBQPNFQJHJDWBT2FCDPAA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:45.367038063 (113h30m43.918508514s) [noise=true , address=145.79.227.26:4653 , piece_id=EEKXCCTYRNCRJFVM2QBQVQADQJ2YVDCDOXFIFUP4BYUEVYDGWZQA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection Nov 14 01:34:45.367162339 (113h30m43.91863279s) [noise=true , address=145.79.227.14:4840 , piece_id=3DEHNGEWEG4GA5YTLKGDV4BYUJ7Z6KCVLM343ZB5MYZWEMOASS7A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection Nov 14 01:34:45.367201353 (113h30m43.918671804s) [noise=true , address=145.79.227.9:4366 , piece_id=GXQSDBFPFZPZ352A5KBNQJS5A32C7ZO522AHCHO3A27P4LAMUFZQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection Nov 14 01:34:45.367247291 (113h30m43.918717742s) [noise=true , address=145.79.227.20:4439 , piece_id=FEDAGRFBHNL6ES3XTCRWPEI4NN2CMMDJIVNW5WOG3KYD7VXFXA7Q, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection; write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection Nov 14 01:34:45.367425070 (113h30m43.918895531s) [noise=true , address=145.79.227.13:4804 , piece_id=NZL7G7WFABHCYGW63IPFVWP45AMPZRDYLDF4JNKJHK5PLSHYHLSQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection Nov 14 01:34:45.367779144 (113h30m43.919249605s) [noise=true , address=145.79.227.30:4763 , piece_id=IW3HMSEGIX2TFUY277OSEF4BULPDWA3AO3MNJ7OCUULORPLHNWOQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:50566->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:34:45.367825933 (113h30m43.919296394s) [noise=true , address=145.79.227.16:4912 , piece_id=SKK6OIUPSWXXD6QS5ZCVVORVSFXXOE5UJH7POHTKYY6PU4LX2JGA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection Nov 14 01:34:45.368025623 (113h30m43.919496114s) [noise=true , address=145.79.227.21:4473 , piece_id=DIGA6L2TCAOGHAGUY5K4T3R7H66CF3JLXBXKMC7TPTXVKBCLDJBA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: read tcp 188.166.225.170:49576->145.79.227.21:4473: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:34:45.368068755 (113h30m43.919539206s) [seg_pos={0 0}, upload=99885]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 14 01:34:45.634359530 (113h30m44.185829991s) [upload=99885]: Done uploading segments... Nov 14 01:34:45.634431096 (113h30m44.185901557s) [upload=99885]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 7.265445306s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 6.681513966s trace: https://jaeger.dev.storj.io/trace/5bc86ccfe33cf550 FAIL (2025-11-14 01:40:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ce_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288397607 (113h30m2.839868078s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288507285 (113h30m2.839977746s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.289077772 (113h30m2.840548233s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290074169 (113h30m2.84154464s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290622854 (113h30m2.842093345s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292062385 (113h30m2.843532866s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292958642 (113h30m2.844429123s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.294563758 (113h30m2.846034229s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.296755000 (113h30m2.848225462s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.298618379 (113h30m2.85008884s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.300188338 (113h30m2.85165881s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.301125483 (113h30m2.852595944s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302545266 (113h30m2.854015727s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302898339 (113h30m2.8543688s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.303825163 (113h30m2.855295624s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.316046324 (113h30m2.867516785s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.318180529 (113h30m2.86965099s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:05.581380111 (113h30m4.132850592s) [upload=99881]: Next returned nil segment Nov 14 01:34:05.581407382 (113h30m4.132877844s) [upload=99881]: Waiting for error group managing segments... Nov 14 01:34:05.921914348 (113h30m4.473384809s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.005586503 (113h30m4.557056964s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.026850744 (113h30m4.578321205s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.121926626 (113h30m4.673397087s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.235634986 (113h30m4.787105437s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.277897963 (113h30m4.829368424s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.283628915 (113h30m4.835099376s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.287808744 (113h30m4.839279205s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.300041498 (113h30m4.851511949s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.304400718 (113h30m4.855871179s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.325040680 (113h30m4.876511141s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.331007762 (113h30m4.882478223s) [noise=true , address=77.90.31.12:4611 , piece_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.481058726 (113h30m5.032529207s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:35:02.598238012 (113h31m1.149708493s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection Nov 14 01:35:02.599889276 (113h31m1.151359737s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:145.79.227.52:4455): protocol: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection Nov 14 01:35:02.600041686 (113h31m1.151512147s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:127WWD5hr1ZBkRLCHz3jb7kpqqnxPS6VuZ1MyQDmitFbhTSbJNK, address:145.79.227.49:4344): protocol: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection Nov 14 01:35:02.600244733 (113h31m1.151715204s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:145.79.227.35:4488): protocol: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection Nov 14 01:35:02.600569551 (113h31m1.152040022s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12h6MND3rBadG4EjUP18S7izyj23cVBLaB5drFsuNGhcxwAYXSA, address:145.79.227.17:4983): protocol: read tcp 188.166.225.170:33042->145.79.227.17:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:35:02.600603536 (113h31m1.152073997s) [seg_pos={0 0}, upload=99881]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 01:35:02.847367167 (113h31m1.398837648s) [upload=99881]: Done uploading segments... Nov 14 01:35:02.847427322 (113h31m1.398897783s) [upload=99881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.681542552s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................X.............X...................X.......] duration: 5.94317414s trace: https://jaeger.dev.storj.io/trace/4377cd58781b9764 FAIL (2025-11-14 01:40:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== DXCQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351147808 (113h30m14.902618269s) [noise=true , address=77.90.31.13:4628 , piece_id=WHP562KP7DABPK6XQT5F43QGNNCDFXYYYP775VGZLXIG2UEWKK4A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351381713 (113h30m14.902852174s) [noise=true , address=145.79.227.30:4763 , piece_id=IW3HMSEGIX2TFUY277OSEF4BULPDWA3AO3MNJ7OCUULORPLHNWOQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351673048 (113h30m14.903143509s) [noise=true , address=145.79.227.26:4653 , piece_id=EEKXCCTYRNCRJFVM2QBQVQADQJ2YVDCDOXFIFUP4BYUEVYDGWZQA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351818645 (113h30m14.903289106s) [noise=true , address=145.79.227.14:4840 , piece_id=3DEHNGEWEG4GA5YTLKGDV4BYUJ7Z6KCVLM343ZB5MYZWEMOASS7A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352062729 (113h30m14.903533191s) [noise=true , address=45.91.116.13:4154 , piece_id=DSC3N7KGSUSEVCEII65IQMYLHEUC5U52NNSREC32Y3Q3MDNKWREQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352224859 (113h30m14.90369532s) [noise=true , address=77.90.31.18:4807 , piece_id=TD3XBJUZ6MM5CB7NMYEIZWQC63U2PZZJPYHIM3OOU7PE3Y7KGCIQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352479703 (113h30m14.903950165s) [noise=true , address=77.90.31.17:4771 , piece_id=ARX2MBQVJSPWKWS2AHE2QBBWNREGMUORBQPNFQJHJDWBT2FCDPAA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352614791 (113h30m14.904085342s) [noise=true , address=145.79.227.21:4473 , piece_id=DIGA6L2TCAOGHAGUY5K4T3R7H66CF3JLXBXKMC7TPTXVKBCLDJBA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352904944 (113h30m14.904375405s) [noise=true , address=45.91.116.6:4973 , piece_id=NEGS6AAGBP4WRJJ6TXEBNHCV2FXW2OBMRWAWL4RNY4YZZZSI3ZLQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.353329152 (113h30m14.904799613s) [noise=true , address=77.90.31.12:4592 , piece_id=ET6LSZ6LXAAAPVJLBLL4UOIF6SKOKDCKXEXKIQ4OS3RD5J5QB2UQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.353685219 (113h30m14.90515568s) [noise=true , address=145.79.227.16:4912 , piece_id=SKK6OIUPSWXXD6QS5ZCVVORVSFXXOE5UJH7POHTKYY6PU4LX2JGA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354114446 (113h30m14.905584907s) [noise=true , address=45.91.116.12:4225 , piece_id=RMDFCX4JGDRPQ46PPTYAWJE7NIYVDE3J2LCKREBFXHR7I6U2G52A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354334907 (113h30m14.905805368s) [noise=true , address=145.79.227.13:4804 , piece_id=NZL7G7WFABHCYGW63IPFVWP45AMPZRDYLDF4JNKJHK5PLSHYHLSQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354804771 (113h30m14.906275222s) [noise=true , address=145.79.227.20:4439 , piece_id=FEDAGRFBHNL6ES3XTCRWPEI4NN2CMMDJIVNW5WOG3KYD7VXFXA7Q, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:18.090813414 (113h30m16.642283865s) [upload=99885]: Next returned nil segment Nov 14 01:34:18.090832882 (113h30m16.642303333s) [upload=99885]: Waiting for error group managing segments... Nov 14 01:34:18.376699271 (113h30m16.928169732s) [noise=true , address=45.91.116.12:4225 , piece_id=RMDFCX4JGDRPQ46PPTYAWJE7NIYVDE3J2LCKREBFXHR7I6U2G52A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.449067032 (113h30m17.000537493s) [noise=true , address=77.90.31.18:4807 , piece_id=TD3XBJUZ6MM5CB7NMYEIZWQC63U2PZZJPYHIM3OOU7PE3Y7KGCIQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.476540927 (113h30m17.028011378s) [noise=true , address=77.90.31.13:4628 , piece_id=WHP562KP7DABPK6XQT5F43QGNNCDFXYYYP775VGZLXIG2UEWKK4A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.502738892 (113h30m17.054209343s) [noise=true , address=77.90.31.12:4592 , piece_id=ET6LSZ6LXAAAPVJLBLL4UOIF6SKOKDCKXEXKIQ4OS3RD5J5QB2UQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.504520516 (113h30m17.055990977s) [noise=true , address=45.91.116.6:4973 , piece_id=NEGS6AAGBP4WRJJ6TXEBNHCV2FXW2OBMRWAWL4RNY4YZZZSI3ZLQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.504778116 (113h30m17.056248577s) [noise=true , address=45.91.116.13:4154 , piece_id=DSC3N7KGSUSEVCEII65IQMYLHEUC5U52NNSREC32Y3Q3MDNKWREQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.511639541 (113h30m17.063110002s) [noise=true , address=77.90.31.10:4517 , piece_id=GDOCT6DE4NL56DWKOS3JNSQVIVE342SDIZ5YR7XPNRXYPOQYDXCQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.529697226 (113h30m17.081167687s) [noise=true , address=77.90.31.17:4771 , piece_id=ARX2MBQVJSPWKWS2AHE2QBBWNREGMUORBQPNFQJHJDWBT2FCDPAA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:45.367038063 (113h30m43.918508514s) [noise=true , address=145.79.227.26:4653 , piece_id=EEKXCCTYRNCRJFVM2QBQVQADQJ2YVDCDOXFIFUP4BYUEVYDGWZQA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection Nov 14 01:34:45.367162339 (113h30m43.91863279s) [noise=true , address=145.79.227.14:4840 , piece_id=3DEHNGEWEG4GA5YTLKGDV4BYUJ7Z6KCVLM343ZB5MYZWEMOASS7A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection Nov 14 01:34:45.367201353 (113h30m43.918671804s) [noise=true , address=145.79.227.9:4366 , piece_id=GXQSDBFPFZPZ352A5KBNQJS5A32C7ZO522AHCHO3A27P4LAMUFZQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection Nov 14 01:34:45.367247291 (113h30m43.918717742s) [noise=true , address=145.79.227.20:4439 , piece_id=FEDAGRFBHNL6ES3XTCRWPEI4NN2CMMDJIVNW5WOG3KYD7VXFXA7Q, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection; write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection Nov 14 01:34:45.367425070 (113h30m43.918895531s) [noise=true , address=145.79.227.13:4804 , piece_id=NZL7G7WFABHCYGW63IPFVWP45AMPZRDYLDF4JNKJHK5PLSHYHLSQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection Nov 14 01:34:45.367779144 (113h30m43.919249605s) [noise=true , address=145.79.227.30:4763 , piece_id=IW3HMSEGIX2TFUY277OSEF4BULPDWA3AO3MNJ7OCUULORPLHNWOQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:50566->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:34:45.367825933 (113h30m43.919296394s) [noise=true , address=145.79.227.16:4912 , piece_id=SKK6OIUPSWXXD6QS5ZCVVORVSFXXOE5UJH7POHTKYY6PU4LX2JGA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection Nov 14 01:34:45.368025623 (113h30m43.919496114s) [noise=true , address=145.79.227.21:4473 , piece_id=DIGA6L2TCAOGHAGUY5K4T3R7H66CF3JLXBXKMC7TPTXVKBCLDJBA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: read tcp 188.166.225.170:49576->145.79.227.21:4473: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:34:45.368068755 (113h30m43.919539206s) [seg_pos={0 0}, upload=99885]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 14 01:34:45.634359530 (113h30m44.185829991s) [upload=99885]: Done uploading segments... Nov 14 01:34:45.634431096 (113h30m44.185901557s) [upload=99885]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.943206622s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.820513282s trace: https://jaeger.dev.storj.io/trace/5bc86ccfe33cf550 FAIL (2025-11-14 01:39:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ce_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288397607 (113h30m2.839868078s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288507285 (113h30m2.839977746s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.289077772 (113h30m2.840548233s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290074169 (113h30m2.84154464s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290622854 (113h30m2.842093345s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292062385 (113h30m2.843532866s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292958642 (113h30m2.844429123s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.294563758 (113h30m2.846034229s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.296755000 (113h30m2.848225462s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.298618379 (113h30m2.85008884s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.300188338 (113h30m2.85165881s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.301125483 (113h30m2.852595944s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302545266 (113h30m2.854015727s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302898339 (113h30m2.8543688s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.303825163 (113h30m2.855295624s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.316046324 (113h30m2.867516785s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.318180529 (113h30m2.86965099s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:05.581380111 (113h30m4.132850592s) [upload=99881]: Next returned nil segment Nov 14 01:34:05.581407382 (113h30m4.132877844s) [upload=99881]: Waiting for error group managing segments... Nov 14 01:34:05.921914348 (113h30m4.473384809s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.005586503 (113h30m4.557056964s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.026850744 (113h30m4.578321205s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.121926626 (113h30m4.673397087s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.235634986 (113h30m4.787105437s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.277897963 (113h30m4.829368424s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.283628915 (113h30m4.835099376s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.287808744 (113h30m4.839279205s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.300041498 (113h30m4.851511949s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.304400718 (113h30m4.855871179s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.325040680 (113h30m4.876511141s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.331007762 (113h30m4.882478223s) [noise=true , address=77.90.31.12:4611 , piece_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.481058726 (113h30m5.032529207s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:35:02.598238012 (113h31m1.149708493s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection Nov 14 01:35:02.599889276 (113h31m1.151359737s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:145.79.227.52:4455): protocol: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection Nov 14 01:35:02.600041686 (113h31m1.151512147s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:127WWD5hr1ZBkRLCHz3jb7kpqqnxPS6VuZ1MyQDmitFbhTSbJNK, address:145.79.227.49:4344): protocol: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection Nov 14 01:35:02.600244733 (113h31m1.151715204s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:145.79.227.35:4488): protocol: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection Nov 14 01:35:02.600569551 (113h31m1.152040022s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12h6MND3rBadG4EjUP18S7izyj23cVBLaB5drFsuNGhcxwAYXSA, address:145.79.227.17:4983): protocol: read tcp 188.166.225.170:33042->145.79.227.17:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:35:02.600603536 (113h31m1.152073997s) [seg_pos={0 0}, upload=99881]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 01:35:02.847367167 (113h31m1.398837648s) [upload=99881]: Done uploading segments... Nov 14 01:35:02.847427322 (113h31m1.398897783s) [upload=99881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.820542457s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................X.............X...................X......] duration: 6.000749354s trace: https://jaeger.dev.storj.io/trace/4377cd58781b9764 FAIL (2025-11-14 01:39:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== DXCQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351147808 (113h30m14.902618269s) [noise=true , address=77.90.31.13:4628 , piece_id=WHP562KP7DABPK6XQT5F43QGNNCDFXYYYP775VGZLXIG2UEWKK4A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351381713 (113h30m14.902852174s) [noise=true , address=145.79.227.30:4763 , piece_id=IW3HMSEGIX2TFUY277OSEF4BULPDWA3AO3MNJ7OCUULORPLHNWOQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351673048 (113h30m14.903143509s) [noise=true , address=145.79.227.26:4653 , piece_id=EEKXCCTYRNCRJFVM2QBQVQADQJ2YVDCDOXFIFUP4BYUEVYDGWZQA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351818645 (113h30m14.903289106s) [noise=true , address=145.79.227.14:4840 , piece_id=3DEHNGEWEG4GA5YTLKGDV4BYUJ7Z6KCVLM343ZB5MYZWEMOASS7A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352062729 (113h30m14.903533191s) [noise=true , address=45.91.116.13:4154 , piece_id=DSC3N7KGSUSEVCEII65IQMYLHEUC5U52NNSREC32Y3Q3MDNKWREQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352224859 (113h30m14.90369532s) [noise=true , address=77.90.31.18:4807 , piece_id=TD3XBJUZ6MM5CB7NMYEIZWQC63U2PZZJPYHIM3OOU7PE3Y7KGCIQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352479703 (113h30m14.903950165s) [noise=true , address=77.90.31.17:4771 , piece_id=ARX2MBQVJSPWKWS2AHE2QBBWNREGMUORBQPNFQJHJDWBT2FCDPAA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352614791 (113h30m14.904085342s) [noise=true , address=145.79.227.21:4473 , piece_id=DIGA6L2TCAOGHAGUY5K4T3R7H66CF3JLXBXKMC7TPTXVKBCLDJBA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352904944 (113h30m14.904375405s) [noise=true , address=45.91.116.6:4973 , piece_id=NEGS6AAGBP4WRJJ6TXEBNHCV2FXW2OBMRWAWL4RNY4YZZZSI3ZLQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.353329152 (113h30m14.904799613s) [noise=true , address=77.90.31.12:4592 , piece_id=ET6LSZ6LXAAAPVJLBLL4UOIF6SKOKDCKXEXKIQ4OS3RD5J5QB2UQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.353685219 (113h30m14.90515568s) [noise=true , address=145.79.227.16:4912 , piece_id=SKK6OIUPSWXXD6QS5ZCVVORVSFXXOE5UJH7POHTKYY6PU4LX2JGA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354114446 (113h30m14.905584907s) [noise=true , address=45.91.116.12:4225 , piece_id=RMDFCX4JGDRPQ46PPTYAWJE7NIYVDE3J2LCKREBFXHR7I6U2G52A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354334907 (113h30m14.905805368s) [noise=true , address=145.79.227.13:4804 , piece_id=NZL7G7WFABHCYGW63IPFVWP45AMPZRDYLDF4JNKJHK5PLSHYHLSQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354804771 (113h30m14.906275222s) [noise=true , address=145.79.227.20:4439 , piece_id=FEDAGRFBHNL6ES3XTCRWPEI4NN2CMMDJIVNW5WOG3KYD7VXFXA7Q, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:18.090813414 (113h30m16.642283865s) [upload=99885]: Next returned nil segment Nov 14 01:34:18.090832882 (113h30m16.642303333s) [upload=99885]: Waiting for error group managing segments... Nov 14 01:34:18.376699271 (113h30m16.928169732s) [noise=true , address=45.91.116.12:4225 , piece_id=RMDFCX4JGDRPQ46PPTYAWJE7NIYVDE3J2LCKREBFXHR7I6U2G52A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.449067032 (113h30m17.000537493s) [noise=true , address=77.90.31.18:4807 , piece_id=TD3XBJUZ6MM5CB7NMYEIZWQC63U2PZZJPYHIM3OOU7PE3Y7KGCIQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.476540927 (113h30m17.028011378s) [noise=true , address=77.90.31.13:4628 , piece_id=WHP562KP7DABPK6XQT5F43QGNNCDFXYYYP775VGZLXIG2UEWKK4A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.502738892 (113h30m17.054209343s) [noise=true , address=77.90.31.12:4592 , piece_id=ET6LSZ6LXAAAPVJLBLL4UOIF6SKOKDCKXEXKIQ4OS3RD5J5QB2UQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.504520516 (113h30m17.055990977s) [noise=true , address=45.91.116.6:4973 , piece_id=NEGS6AAGBP4WRJJ6TXEBNHCV2FXW2OBMRWAWL4RNY4YZZZSI3ZLQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.504778116 (113h30m17.056248577s) [noise=true , address=45.91.116.13:4154 , piece_id=DSC3N7KGSUSEVCEII65IQMYLHEUC5U52NNSREC32Y3Q3MDNKWREQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.511639541 (113h30m17.063110002s) [noise=true , address=77.90.31.10:4517 , piece_id=GDOCT6DE4NL56DWKOS3JNSQVIVE342SDIZ5YR7XPNRXYPOQYDXCQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.529697226 (113h30m17.081167687s) [noise=true , address=77.90.31.17:4771 , piece_id=ARX2MBQVJSPWKWS2AHE2QBBWNREGMUORBQPNFQJHJDWBT2FCDPAA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:45.367038063 (113h30m43.918508514s) [noise=true , address=145.79.227.26:4653 , piece_id=EEKXCCTYRNCRJFVM2QBQVQADQJ2YVDCDOXFIFUP4BYUEVYDGWZQA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection Nov 14 01:34:45.367162339 (113h30m43.91863279s) [noise=true , address=145.79.227.14:4840 , piece_id=3DEHNGEWEG4GA5YTLKGDV4BYUJ7Z6KCVLM343ZB5MYZWEMOASS7A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection Nov 14 01:34:45.367201353 (113h30m43.918671804s) [noise=true , address=145.79.227.9:4366 , piece_id=GXQSDBFPFZPZ352A5KBNQJS5A32C7ZO522AHCHO3A27P4LAMUFZQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection Nov 14 01:34:45.367247291 (113h30m43.918717742s) [noise=true , address=145.79.227.20:4439 , piece_id=FEDAGRFBHNL6ES3XTCRWPEI4NN2CMMDJIVNW5WOG3KYD7VXFXA7Q, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection; write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection Nov 14 01:34:45.367425070 (113h30m43.918895531s) [noise=true , address=145.79.227.13:4804 , piece_id=NZL7G7WFABHCYGW63IPFVWP45AMPZRDYLDF4JNKJHK5PLSHYHLSQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection Nov 14 01:34:45.367779144 (113h30m43.919249605s) [noise=true , address=145.79.227.30:4763 , piece_id=IW3HMSEGIX2TFUY277OSEF4BULPDWA3AO3MNJ7OCUULORPLHNWOQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:50566->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:34:45.367825933 (113h30m43.919296394s) [noise=true , address=145.79.227.16:4912 , piece_id=SKK6OIUPSWXXD6QS5ZCVVORVSFXXOE5UJH7POHTKYY6PU4LX2JGA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection Nov 14 01:34:45.368025623 (113h30m43.919496114s) [noise=true , address=145.79.227.21:4473 , piece_id=DIGA6L2TCAOGHAGUY5K4T3R7H66CF3JLXBXKMC7TPTXVKBCLDJBA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: read tcp 188.166.225.170:49576->145.79.227.21:4473: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:34:45.368068755 (113h30m43.919539206s) [seg_pos={0 0}, upload=99885]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 14 01:34:45.634359530 (113h30m44.185829991s) [upload=99885]: Done uploading segments... Nov 14 01:34:45.634431096 (113h30m44.185901557s) [upload=99885]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.000801242s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.666852144s trace: https://jaeger.dev.storj.io/trace/5bc86ccfe33cf550 FAIL (2025-11-14 01:39:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ce_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288397607 (113h30m2.839868078s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288507285 (113h30m2.839977746s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.289077772 (113h30m2.840548233s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290074169 (113h30m2.84154464s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290622854 (113h30m2.842093345s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292062385 (113h30m2.843532866s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292958642 (113h30m2.844429123s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.294563758 (113h30m2.846034229s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.296755000 (113h30m2.848225462s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.298618379 (113h30m2.85008884s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.300188338 (113h30m2.85165881s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.301125483 (113h30m2.852595944s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302545266 (113h30m2.854015727s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302898339 (113h30m2.8543688s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.303825163 (113h30m2.855295624s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.316046324 (113h30m2.867516785s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.318180529 (113h30m2.86965099s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:05.581380111 (113h30m4.132850592s) [upload=99881]: Next returned nil segment Nov 14 01:34:05.581407382 (113h30m4.132877844s) [upload=99881]: Waiting for error group managing segments... Nov 14 01:34:05.921914348 (113h30m4.473384809s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.005586503 (113h30m4.557056964s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.026850744 (113h30m4.578321205s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.121926626 (113h30m4.673397087s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.235634986 (113h30m4.787105437s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.277897963 (113h30m4.829368424s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.283628915 (113h30m4.835099376s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.287808744 (113h30m4.839279205s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.300041498 (113h30m4.851511949s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.304400718 (113h30m4.855871179s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.325040680 (113h30m4.876511141s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.331007762 (113h30m4.882478223s) [noise=true , address=77.90.31.12:4611 , piece_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.481058726 (113h30m5.032529207s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:35:02.598238012 (113h31m1.149708493s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection Nov 14 01:35:02.599889276 (113h31m1.151359737s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:145.79.227.52:4455): protocol: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection Nov 14 01:35:02.600041686 (113h31m1.151512147s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:127WWD5hr1ZBkRLCHz3jb7kpqqnxPS6VuZ1MyQDmitFbhTSbJNK, address:145.79.227.49:4344): protocol: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection Nov 14 01:35:02.600244733 (113h31m1.151715204s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:145.79.227.35:4488): protocol: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection Nov 14 01:35:02.600569551 (113h31m1.152040022s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12h6MND3rBadG4EjUP18S7izyj23cVBLaB5drFsuNGhcxwAYXSA, address:145.79.227.17:4983): protocol: read tcp 188.166.225.170:33042->145.79.227.17:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:35:02.600603536 (113h31m1.152073997s) [seg_pos={0 0}, upload=99881]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 01:35:02.847367167 (113h31m1.398837648s) [upload=99881]: Done uploading segments... Nov 14 01:35:02.847427322 (113h31m1.398897783s) [upload=99881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.666880831s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.762740794s trace: https://jaeger.dev.storj.io/trace/5bc86ccfe33cf550 FAIL (2025-11-14 01:39:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ce_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288397607 (113h30m2.839868078s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288507285 (113h30m2.839977746s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.289077772 (113h30m2.840548233s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290074169 (113h30m2.84154464s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290622854 (113h30m2.842093345s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292062385 (113h30m2.843532866s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292958642 (113h30m2.844429123s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.294563758 (113h30m2.846034229s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.296755000 (113h30m2.848225462s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.298618379 (113h30m2.85008884s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.300188338 (113h30m2.85165881s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.301125483 (113h30m2.852595944s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302545266 (113h30m2.854015727s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302898339 (113h30m2.8543688s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.303825163 (113h30m2.855295624s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.316046324 (113h30m2.867516785s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.318180529 (113h30m2.86965099s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:05.581380111 (113h30m4.132850592s) [upload=99881]: Next returned nil segment Nov 14 01:34:05.581407382 (113h30m4.132877844s) [upload=99881]: Waiting for error group managing segments... Nov 14 01:34:05.921914348 (113h30m4.473384809s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.005586503 (113h30m4.557056964s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.026850744 (113h30m4.578321205s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.121926626 (113h30m4.673397087s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.235634986 (113h30m4.787105437s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.277897963 (113h30m4.829368424s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.283628915 (113h30m4.835099376s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.287808744 (113h30m4.839279205s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.300041498 (113h30m4.851511949s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.304400718 (113h30m4.855871179s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.325040680 (113h30m4.876511141s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.331007762 (113h30m4.882478223s) [noise=true , address=77.90.31.12:4611 , piece_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.481058726 (113h30m5.032529207s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:35:02.598238012 (113h31m1.149708493s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection Nov 14 01:35:02.599889276 (113h31m1.151359737s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:145.79.227.52:4455): protocol: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection Nov 14 01:35:02.600041686 (113h31m1.151512147s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:127WWD5hr1ZBkRLCHz3jb7kpqqnxPS6VuZ1MyQDmitFbhTSbJNK, address:145.79.227.49:4344): protocol: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection Nov 14 01:35:02.600244733 (113h31m1.151715204s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:145.79.227.35:4488): protocol: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection Nov 14 01:35:02.600569551 (113h31m1.152040022s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12h6MND3rBadG4EjUP18S7izyj23cVBLaB5drFsuNGhcxwAYXSA, address:145.79.227.17:4983): protocol: read tcp 188.166.225.170:33042->145.79.227.17:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:35:02.600603536 (113h31m1.152073997s) [seg_pos={0 0}, upload=99881]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 01:35:02.847367167 (113h31m1.398837648s) [upload=99881]: Done uploading segments... Nov 14 01:35:02.847427322 (113h31m1.398897783s) [upload=99881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.762760191s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................X.............X...................X.....] duration: 6.704046535s trace: https://jaeger.dev.storj.io/trace/4377cd58781b9764 FAIL (2025-11-14 01:38:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== DXCQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351147808 (113h30m14.902618269s) [noise=true , address=77.90.31.13:4628 , piece_id=WHP562KP7DABPK6XQT5F43QGNNCDFXYYYP775VGZLXIG2UEWKK4A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351381713 (113h30m14.902852174s) [noise=true , address=145.79.227.30:4763 , piece_id=IW3HMSEGIX2TFUY277OSEF4BULPDWA3AO3MNJ7OCUULORPLHNWOQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351673048 (113h30m14.903143509s) [noise=true , address=145.79.227.26:4653 , piece_id=EEKXCCTYRNCRJFVM2QBQVQADQJ2YVDCDOXFIFUP4BYUEVYDGWZQA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351818645 (113h30m14.903289106s) [noise=true , address=145.79.227.14:4840 , piece_id=3DEHNGEWEG4GA5YTLKGDV4BYUJ7Z6KCVLM343ZB5MYZWEMOASS7A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352062729 (113h30m14.903533191s) [noise=true , address=45.91.116.13:4154 , piece_id=DSC3N7KGSUSEVCEII65IQMYLHEUC5U52NNSREC32Y3Q3MDNKWREQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352224859 (113h30m14.90369532s) [noise=true , address=77.90.31.18:4807 , piece_id=TD3XBJUZ6MM5CB7NMYEIZWQC63U2PZZJPYHIM3OOU7PE3Y7KGCIQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352479703 (113h30m14.903950165s) [noise=true , address=77.90.31.17:4771 , piece_id=ARX2MBQVJSPWKWS2AHE2QBBWNREGMUORBQPNFQJHJDWBT2FCDPAA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352614791 (113h30m14.904085342s) [noise=true , address=145.79.227.21:4473 , piece_id=DIGA6L2TCAOGHAGUY5K4T3R7H66CF3JLXBXKMC7TPTXVKBCLDJBA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352904944 (113h30m14.904375405s) [noise=true , address=45.91.116.6:4973 , piece_id=NEGS6AAGBP4WRJJ6TXEBNHCV2FXW2OBMRWAWL4RNY4YZZZSI3ZLQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.353329152 (113h30m14.904799613s) [noise=true , address=77.90.31.12:4592 , piece_id=ET6LSZ6LXAAAPVJLBLL4UOIF6SKOKDCKXEXKIQ4OS3RD5J5QB2UQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.353685219 (113h30m14.90515568s) [noise=true , address=145.79.227.16:4912 , piece_id=SKK6OIUPSWXXD6QS5ZCVVORVSFXXOE5UJH7POHTKYY6PU4LX2JGA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354114446 (113h30m14.905584907s) [noise=true , address=45.91.116.12:4225 , piece_id=RMDFCX4JGDRPQ46PPTYAWJE7NIYVDE3J2LCKREBFXHR7I6U2G52A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354334907 (113h30m14.905805368s) [noise=true , address=145.79.227.13:4804 , piece_id=NZL7G7WFABHCYGW63IPFVWP45AMPZRDYLDF4JNKJHK5PLSHYHLSQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354804771 (113h30m14.906275222s) [noise=true , address=145.79.227.20:4439 , piece_id=FEDAGRFBHNL6ES3XTCRWPEI4NN2CMMDJIVNW5WOG3KYD7VXFXA7Q, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:18.090813414 (113h30m16.642283865s) [upload=99885]: Next returned nil segment Nov 14 01:34:18.090832882 (113h30m16.642303333s) [upload=99885]: Waiting for error group managing segments... Nov 14 01:34:18.376699271 (113h30m16.928169732s) [noise=true , address=45.91.116.12:4225 , piece_id=RMDFCX4JGDRPQ46PPTYAWJE7NIYVDE3J2LCKREBFXHR7I6U2G52A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.449067032 (113h30m17.000537493s) [noise=true , address=77.90.31.18:4807 , piece_id=TD3XBJUZ6MM5CB7NMYEIZWQC63U2PZZJPYHIM3OOU7PE3Y7KGCIQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.476540927 (113h30m17.028011378s) [noise=true , address=77.90.31.13:4628 , piece_id=WHP562KP7DABPK6XQT5F43QGNNCDFXYYYP775VGZLXIG2UEWKK4A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.502738892 (113h30m17.054209343s) [noise=true , address=77.90.31.12:4592 , piece_id=ET6LSZ6LXAAAPVJLBLL4UOIF6SKOKDCKXEXKIQ4OS3RD5J5QB2UQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.504520516 (113h30m17.055990977s) [noise=true , address=45.91.116.6:4973 , piece_id=NEGS6AAGBP4WRJJ6TXEBNHCV2FXW2OBMRWAWL4RNY4YZZZSI3ZLQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.504778116 (113h30m17.056248577s) [noise=true , address=45.91.116.13:4154 , piece_id=DSC3N7KGSUSEVCEII65IQMYLHEUC5U52NNSREC32Y3Q3MDNKWREQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.511639541 (113h30m17.063110002s) [noise=true , address=77.90.31.10:4517 , piece_id=GDOCT6DE4NL56DWKOS3JNSQVIVE342SDIZ5YR7XPNRXYPOQYDXCQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.529697226 (113h30m17.081167687s) [noise=true , address=77.90.31.17:4771 , piece_id=ARX2MBQVJSPWKWS2AHE2QBBWNREGMUORBQPNFQJHJDWBT2FCDPAA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:45.367038063 (113h30m43.918508514s) [noise=true , address=145.79.227.26:4653 , piece_id=EEKXCCTYRNCRJFVM2QBQVQADQJ2YVDCDOXFIFUP4BYUEVYDGWZQA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection Nov 14 01:34:45.367162339 (113h30m43.91863279s) [noise=true , address=145.79.227.14:4840 , piece_id=3DEHNGEWEG4GA5YTLKGDV4BYUJ7Z6KCVLM343ZB5MYZWEMOASS7A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection Nov 14 01:34:45.367201353 (113h30m43.918671804s) [noise=true , address=145.79.227.9:4366 , piece_id=GXQSDBFPFZPZ352A5KBNQJS5A32C7ZO522AHCHO3A27P4LAMUFZQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection Nov 14 01:34:45.367247291 (113h30m43.918717742s) [noise=true , address=145.79.227.20:4439 , piece_id=FEDAGRFBHNL6ES3XTCRWPEI4NN2CMMDJIVNW5WOG3KYD7VXFXA7Q, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection; write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection Nov 14 01:34:45.367425070 (113h30m43.918895531s) [noise=true , address=145.79.227.13:4804 , piece_id=NZL7G7WFABHCYGW63IPFVWP45AMPZRDYLDF4JNKJHK5PLSHYHLSQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection Nov 14 01:34:45.367779144 (113h30m43.919249605s) [noise=true , address=145.79.227.30:4763 , piece_id=IW3HMSEGIX2TFUY277OSEF4BULPDWA3AO3MNJ7OCUULORPLHNWOQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:50566->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:34:45.367825933 (113h30m43.919296394s) [noise=true , address=145.79.227.16:4912 , piece_id=SKK6OIUPSWXXD6QS5ZCVVORVSFXXOE5UJH7POHTKYY6PU4LX2JGA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection Nov 14 01:34:45.368025623 (113h30m43.919496114s) [noise=true , address=145.79.227.21:4473 , piece_id=DIGA6L2TCAOGHAGUY5K4T3R7H66CF3JLXBXKMC7TPTXVKBCLDJBA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: read tcp 188.166.225.170:49576->145.79.227.21:4473: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:34:45.368068755 (113h30m43.919539206s) [seg_pos={0 0}, upload=99885]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 14 01:34:45.634359530 (113h30m44.185829991s) [upload=99885]: Done uploading segments... Nov 14 01:34:45.634431096 (113h30m44.185901557s) [upload=99885]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.704075789s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.99002344s trace: https://jaeger.dev.storj.io/trace/5bc86ccfe33cf550 FAIL (2025-11-14 01:38:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ce_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288397607 (113h30m2.839868078s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288507285 (113h30m2.839977746s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.289077772 (113h30m2.840548233s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290074169 (113h30m2.84154464s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290622854 (113h30m2.842093345s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292062385 (113h30m2.843532866s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292958642 (113h30m2.844429123s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.294563758 (113h30m2.846034229s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.296755000 (113h30m2.848225462s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.298618379 (113h30m2.85008884s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.300188338 (113h30m2.85165881s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.301125483 (113h30m2.852595944s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302545266 (113h30m2.854015727s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302898339 (113h30m2.8543688s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.303825163 (113h30m2.855295624s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.316046324 (113h30m2.867516785s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.318180529 (113h30m2.86965099s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:05.581380111 (113h30m4.132850592s) [upload=99881]: Next returned nil segment Nov 14 01:34:05.581407382 (113h30m4.132877844s) [upload=99881]: Waiting for error group managing segments... Nov 14 01:34:05.921914348 (113h30m4.473384809s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.005586503 (113h30m4.557056964s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.026850744 (113h30m4.578321205s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.121926626 (113h30m4.673397087s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.235634986 (113h30m4.787105437s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.277897963 (113h30m4.829368424s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.283628915 (113h30m4.835099376s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.287808744 (113h30m4.839279205s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.300041498 (113h30m4.851511949s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.304400718 (113h30m4.855871179s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.325040680 (113h30m4.876511141s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.331007762 (113h30m4.882478223s) [noise=true , address=77.90.31.12:4611 , piece_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.481058726 (113h30m5.032529207s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:35:02.598238012 (113h31m1.149708493s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection Nov 14 01:35:02.599889276 (113h31m1.151359737s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:145.79.227.52:4455): protocol: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection Nov 14 01:35:02.600041686 (113h31m1.151512147s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:127WWD5hr1ZBkRLCHz3jb7kpqqnxPS6VuZ1MyQDmitFbhTSbJNK, address:145.79.227.49:4344): protocol: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection Nov 14 01:35:02.600244733 (113h31m1.151715204s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:145.79.227.35:4488): protocol: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection Nov 14 01:35:02.600569551 (113h31m1.152040022s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12h6MND3rBadG4EjUP18S7izyj23cVBLaB5drFsuNGhcxwAYXSA, address:145.79.227.17:4983): protocol: read tcp 188.166.225.170:33042->145.79.227.17:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:35:02.600603536 (113h31m1.152073997s) [seg_pos={0 0}, upload=99881]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 01:35:02.847367167 (113h31m1.398837648s) [upload=99881]: Done uploading segments... Nov 14 01:35:02.847427322 (113h31m1.398897783s) [upload=99881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.990052975s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.8589699s trace: https://jaeger.dev.storj.io/trace/5bc86ccfe33cf550 FAIL (2025-11-14 01:37:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ce_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288397607 (113h30m2.839868078s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288507285 (113h30m2.839977746s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.289077772 (113h30m2.840548233s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290074169 (113h30m2.84154464s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290622854 (113h30m2.842093345s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292062385 (113h30m2.843532866s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292958642 (113h30m2.844429123s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.294563758 (113h30m2.846034229s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.296755000 (113h30m2.848225462s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.298618379 (113h30m2.85008884s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.300188338 (113h30m2.85165881s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.301125483 (113h30m2.852595944s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302545266 (113h30m2.854015727s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302898339 (113h30m2.8543688s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.303825163 (113h30m2.855295624s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.316046324 (113h30m2.867516785s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.318180529 (113h30m2.86965099s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:05.581380111 (113h30m4.132850592s) [upload=99881]: Next returned nil segment Nov 14 01:34:05.581407382 (113h30m4.132877844s) [upload=99881]: Waiting for error group managing segments... Nov 14 01:34:05.921914348 (113h30m4.473384809s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.005586503 (113h30m4.557056964s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.026850744 (113h30m4.578321205s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.121926626 (113h30m4.673397087s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.235634986 (113h30m4.787105437s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.277897963 (113h30m4.829368424s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.283628915 (113h30m4.835099376s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.287808744 (113h30m4.839279205s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.300041498 (113h30m4.851511949s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.304400718 (113h30m4.855871179s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.325040680 (113h30m4.876511141s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.331007762 (113h30m4.882478223s) [noise=true , address=77.90.31.12:4611 , piece_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.481058726 (113h30m5.032529207s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:35:02.598238012 (113h31m1.149708493s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection Nov 14 01:35:02.599889276 (113h31m1.151359737s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:145.79.227.52:4455): protocol: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection Nov 14 01:35:02.600041686 (113h31m1.151512147s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:127WWD5hr1ZBkRLCHz3jb7kpqqnxPS6VuZ1MyQDmitFbhTSbJNK, address:145.79.227.49:4344): protocol: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection Nov 14 01:35:02.600244733 (113h31m1.151715204s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:145.79.227.35:4488): protocol: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection Nov 14 01:35:02.600569551 (113h31m1.152040022s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12h6MND3rBadG4EjUP18S7izyj23cVBLaB5drFsuNGhcxwAYXSA, address:145.79.227.17:4983): protocol: read tcp 188.166.225.170:33042->145.79.227.17:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:35:02.600603536 (113h31m1.152073997s) [seg_pos={0 0}, upload=99881]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 01:35:02.847367167 (113h31m1.398837648s) [upload=99881]: Done uploading segments... Nov 14 01:35:02.847427322 (113h31m1.398897783s) [upload=99881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.859002753s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................X.............X...................X....] duration: 6.049169865s trace: https://jaeger.dev.storj.io/trace/4377cd58781b9764 FAIL (2025-11-14 01:37:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== DXCQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351147808 (113h30m14.902618269s) [noise=true , address=77.90.31.13:4628 , piece_id=WHP562KP7DABPK6XQT5F43QGNNCDFXYYYP775VGZLXIG2UEWKK4A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351381713 (113h30m14.902852174s) [noise=true , address=145.79.227.30:4763 , piece_id=IW3HMSEGIX2TFUY277OSEF4BULPDWA3AO3MNJ7OCUULORPLHNWOQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351673048 (113h30m14.903143509s) [noise=true , address=145.79.227.26:4653 , piece_id=EEKXCCTYRNCRJFVM2QBQVQADQJ2YVDCDOXFIFUP4BYUEVYDGWZQA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351818645 (113h30m14.903289106s) [noise=true , address=145.79.227.14:4840 , piece_id=3DEHNGEWEG4GA5YTLKGDV4BYUJ7Z6KCVLM343ZB5MYZWEMOASS7A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352062729 (113h30m14.903533191s) [noise=true , address=45.91.116.13:4154 , piece_id=DSC3N7KGSUSEVCEII65IQMYLHEUC5U52NNSREC32Y3Q3MDNKWREQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352224859 (113h30m14.90369532s) [noise=true , address=77.90.31.18:4807 , piece_id=TD3XBJUZ6MM5CB7NMYEIZWQC63U2PZZJPYHIM3OOU7PE3Y7KGCIQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352479703 (113h30m14.903950165s) [noise=true , address=77.90.31.17:4771 , piece_id=ARX2MBQVJSPWKWS2AHE2QBBWNREGMUORBQPNFQJHJDWBT2FCDPAA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352614791 (113h30m14.904085342s) [noise=true , address=145.79.227.21:4473 , piece_id=DIGA6L2TCAOGHAGUY5K4T3R7H66CF3JLXBXKMC7TPTXVKBCLDJBA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352904944 (113h30m14.904375405s) [noise=true , address=45.91.116.6:4973 , piece_id=NEGS6AAGBP4WRJJ6TXEBNHCV2FXW2OBMRWAWL4RNY4YZZZSI3ZLQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.353329152 (113h30m14.904799613s) [noise=true , address=77.90.31.12:4592 , piece_id=ET6LSZ6LXAAAPVJLBLL4UOIF6SKOKDCKXEXKIQ4OS3RD5J5QB2UQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.353685219 (113h30m14.90515568s) [noise=true , address=145.79.227.16:4912 , piece_id=SKK6OIUPSWXXD6QS5ZCVVORVSFXXOE5UJH7POHTKYY6PU4LX2JGA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354114446 (113h30m14.905584907s) [noise=true , address=45.91.116.12:4225 , piece_id=RMDFCX4JGDRPQ46PPTYAWJE7NIYVDE3J2LCKREBFXHR7I6U2G52A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354334907 (113h30m14.905805368s) [noise=true , address=145.79.227.13:4804 , piece_id=NZL7G7WFABHCYGW63IPFVWP45AMPZRDYLDF4JNKJHK5PLSHYHLSQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354804771 (113h30m14.906275222s) [noise=true , address=145.79.227.20:4439 , piece_id=FEDAGRFBHNL6ES3XTCRWPEI4NN2CMMDJIVNW5WOG3KYD7VXFXA7Q, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:18.090813414 (113h30m16.642283865s) [upload=99885]: Next returned nil segment Nov 14 01:34:18.090832882 (113h30m16.642303333s) [upload=99885]: Waiting for error group managing segments... Nov 14 01:34:18.376699271 (113h30m16.928169732s) [noise=true , address=45.91.116.12:4225 , piece_id=RMDFCX4JGDRPQ46PPTYAWJE7NIYVDE3J2LCKREBFXHR7I6U2G52A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.449067032 (113h30m17.000537493s) [noise=true , address=77.90.31.18:4807 , piece_id=TD3XBJUZ6MM5CB7NMYEIZWQC63U2PZZJPYHIM3OOU7PE3Y7KGCIQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.476540927 (113h30m17.028011378s) [noise=true , address=77.90.31.13:4628 , piece_id=WHP562KP7DABPK6XQT5F43QGNNCDFXYYYP775VGZLXIG2UEWKK4A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.502738892 (113h30m17.054209343s) [noise=true , address=77.90.31.12:4592 , piece_id=ET6LSZ6LXAAAPVJLBLL4UOIF6SKOKDCKXEXKIQ4OS3RD5J5QB2UQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.504520516 (113h30m17.055990977s) [noise=true , address=45.91.116.6:4973 , piece_id=NEGS6AAGBP4WRJJ6TXEBNHCV2FXW2OBMRWAWL4RNY4YZZZSI3ZLQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.504778116 (113h30m17.056248577s) [noise=true , address=45.91.116.13:4154 , piece_id=DSC3N7KGSUSEVCEII65IQMYLHEUC5U52NNSREC32Y3Q3MDNKWREQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.511639541 (113h30m17.063110002s) [noise=true , address=77.90.31.10:4517 , piece_id=GDOCT6DE4NL56DWKOS3JNSQVIVE342SDIZ5YR7XPNRXYPOQYDXCQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.529697226 (113h30m17.081167687s) [noise=true , address=77.90.31.17:4771 , piece_id=ARX2MBQVJSPWKWS2AHE2QBBWNREGMUORBQPNFQJHJDWBT2FCDPAA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:45.367038063 (113h30m43.918508514s) [noise=true , address=145.79.227.26:4653 , piece_id=EEKXCCTYRNCRJFVM2QBQVQADQJ2YVDCDOXFIFUP4BYUEVYDGWZQA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection Nov 14 01:34:45.367162339 (113h30m43.91863279s) [noise=true , address=145.79.227.14:4840 , piece_id=3DEHNGEWEG4GA5YTLKGDV4BYUJ7Z6KCVLM343ZB5MYZWEMOASS7A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection Nov 14 01:34:45.367201353 (113h30m43.918671804s) [noise=true , address=145.79.227.9:4366 , piece_id=GXQSDBFPFZPZ352A5KBNQJS5A32C7ZO522AHCHO3A27P4LAMUFZQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection Nov 14 01:34:45.367247291 (113h30m43.918717742s) [noise=true , address=145.79.227.20:4439 , piece_id=FEDAGRFBHNL6ES3XTCRWPEI4NN2CMMDJIVNW5WOG3KYD7VXFXA7Q, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection; write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection Nov 14 01:34:45.367425070 (113h30m43.918895531s) [noise=true , address=145.79.227.13:4804 , piece_id=NZL7G7WFABHCYGW63IPFVWP45AMPZRDYLDF4JNKJHK5PLSHYHLSQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection Nov 14 01:34:45.367779144 (113h30m43.919249605s) [noise=true , address=145.79.227.30:4763 , piece_id=IW3HMSEGIX2TFUY277OSEF4BULPDWA3AO3MNJ7OCUULORPLHNWOQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:50566->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:34:45.367825933 (113h30m43.919296394s) [noise=true , address=145.79.227.16:4912 , piece_id=SKK6OIUPSWXXD6QS5ZCVVORVSFXXOE5UJH7POHTKYY6PU4LX2JGA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection Nov 14 01:34:45.368025623 (113h30m43.919496114s) [noise=true , address=145.79.227.21:4473 , piece_id=DIGA6L2TCAOGHAGUY5K4T3R7H66CF3JLXBXKMC7TPTXVKBCLDJBA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: read tcp 188.166.225.170:49576->145.79.227.21:4473: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:34:45.368068755 (113h30m43.919539206s) [seg_pos={0 0}, upload=99885]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 14 01:34:45.634359530 (113h30m44.185829991s) [upload=99885]: Done uploading segments... Nov 14 01:34:45.634431096 (113h30m44.185901557s) [upload=99885]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.049197857s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................X.............X...................X...] duration: 6.080063238s trace: https://jaeger.dev.storj.io/trace/4377cd58781b9764 FAIL (2025-11-14 01:37:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== DXCQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351147808 (113h30m14.902618269s) [noise=true , address=77.90.31.13:4628 , piece_id=WHP562KP7DABPK6XQT5F43QGNNCDFXYYYP775VGZLXIG2UEWKK4A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351381713 (113h30m14.902852174s) [noise=true , address=145.79.227.30:4763 , piece_id=IW3HMSEGIX2TFUY277OSEF4BULPDWA3AO3MNJ7OCUULORPLHNWOQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351673048 (113h30m14.903143509s) [noise=true , address=145.79.227.26:4653 , piece_id=EEKXCCTYRNCRJFVM2QBQVQADQJ2YVDCDOXFIFUP4BYUEVYDGWZQA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351818645 (113h30m14.903289106s) [noise=true , address=145.79.227.14:4840 , piece_id=3DEHNGEWEG4GA5YTLKGDV4BYUJ7Z6KCVLM343ZB5MYZWEMOASS7A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352062729 (113h30m14.903533191s) [noise=true , address=45.91.116.13:4154 , piece_id=DSC3N7KGSUSEVCEII65IQMYLHEUC5U52NNSREC32Y3Q3MDNKWREQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352224859 (113h30m14.90369532s) [noise=true , address=77.90.31.18:4807 , piece_id=TD3XBJUZ6MM5CB7NMYEIZWQC63U2PZZJPYHIM3OOU7PE3Y7KGCIQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352479703 (113h30m14.903950165s) [noise=true , address=77.90.31.17:4771 , piece_id=ARX2MBQVJSPWKWS2AHE2QBBWNREGMUORBQPNFQJHJDWBT2FCDPAA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352614791 (113h30m14.904085342s) [noise=true , address=145.79.227.21:4473 , piece_id=DIGA6L2TCAOGHAGUY5K4T3R7H66CF3JLXBXKMC7TPTXVKBCLDJBA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352904944 (113h30m14.904375405s) [noise=true , address=45.91.116.6:4973 , piece_id=NEGS6AAGBP4WRJJ6TXEBNHCV2FXW2OBMRWAWL4RNY4YZZZSI3ZLQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.353329152 (113h30m14.904799613s) [noise=true , address=77.90.31.12:4592 , piece_id=ET6LSZ6LXAAAPVJLBLL4UOIF6SKOKDCKXEXKIQ4OS3RD5J5QB2UQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.353685219 (113h30m14.90515568s) [noise=true , address=145.79.227.16:4912 , piece_id=SKK6OIUPSWXXD6QS5ZCVVORVSFXXOE5UJH7POHTKYY6PU4LX2JGA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354114446 (113h30m14.905584907s) [noise=true , address=45.91.116.12:4225 , piece_id=RMDFCX4JGDRPQ46PPTYAWJE7NIYVDE3J2LCKREBFXHR7I6U2G52A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354334907 (113h30m14.905805368s) [noise=true , address=145.79.227.13:4804 , piece_id=NZL7G7WFABHCYGW63IPFVWP45AMPZRDYLDF4JNKJHK5PLSHYHLSQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354804771 (113h30m14.906275222s) [noise=true , address=145.79.227.20:4439 , piece_id=FEDAGRFBHNL6ES3XTCRWPEI4NN2CMMDJIVNW5WOG3KYD7VXFXA7Q, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:18.090813414 (113h30m16.642283865s) [upload=99885]: Next returned nil segment Nov 14 01:34:18.090832882 (113h30m16.642303333s) [upload=99885]: Waiting for error group managing segments... Nov 14 01:34:18.376699271 (113h30m16.928169732s) [noise=true , address=45.91.116.12:4225 , piece_id=RMDFCX4JGDRPQ46PPTYAWJE7NIYVDE3J2LCKREBFXHR7I6U2G52A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.449067032 (113h30m17.000537493s) [noise=true , address=77.90.31.18:4807 , piece_id=TD3XBJUZ6MM5CB7NMYEIZWQC63U2PZZJPYHIM3OOU7PE3Y7KGCIQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.476540927 (113h30m17.028011378s) [noise=true , address=77.90.31.13:4628 , piece_id=WHP562KP7DABPK6XQT5F43QGNNCDFXYYYP775VGZLXIG2UEWKK4A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.502738892 (113h30m17.054209343s) [noise=true , address=77.90.31.12:4592 , piece_id=ET6LSZ6LXAAAPVJLBLL4UOIF6SKOKDCKXEXKIQ4OS3RD5J5QB2UQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.504520516 (113h30m17.055990977s) [noise=true , address=45.91.116.6:4973 , piece_id=NEGS6AAGBP4WRJJ6TXEBNHCV2FXW2OBMRWAWL4RNY4YZZZSI3ZLQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.504778116 (113h30m17.056248577s) [noise=true , address=45.91.116.13:4154 , piece_id=DSC3N7KGSUSEVCEII65IQMYLHEUC5U52NNSREC32Y3Q3MDNKWREQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.511639541 (113h30m17.063110002s) [noise=true , address=77.90.31.10:4517 , piece_id=GDOCT6DE4NL56DWKOS3JNSQVIVE342SDIZ5YR7XPNRXYPOQYDXCQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.529697226 (113h30m17.081167687s) [noise=true , address=77.90.31.17:4771 , piece_id=ARX2MBQVJSPWKWS2AHE2QBBWNREGMUORBQPNFQJHJDWBT2FCDPAA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:45.367038063 (113h30m43.918508514s) [noise=true , address=145.79.227.26:4653 , piece_id=EEKXCCTYRNCRJFVM2QBQVQADQJ2YVDCDOXFIFUP4BYUEVYDGWZQA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection Nov 14 01:34:45.367162339 (113h30m43.91863279s) [noise=true , address=145.79.227.14:4840 , piece_id=3DEHNGEWEG4GA5YTLKGDV4BYUJ7Z6KCVLM343ZB5MYZWEMOASS7A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection Nov 14 01:34:45.367201353 (113h30m43.918671804s) [noise=true , address=145.79.227.9:4366 , piece_id=GXQSDBFPFZPZ352A5KBNQJS5A32C7ZO522AHCHO3A27P4LAMUFZQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection Nov 14 01:34:45.367247291 (113h30m43.918717742s) [noise=true , address=145.79.227.20:4439 , piece_id=FEDAGRFBHNL6ES3XTCRWPEI4NN2CMMDJIVNW5WOG3KYD7VXFXA7Q, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection; write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection Nov 14 01:34:45.367425070 (113h30m43.918895531s) [noise=true , address=145.79.227.13:4804 , piece_id=NZL7G7WFABHCYGW63IPFVWP45AMPZRDYLDF4JNKJHK5PLSHYHLSQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection Nov 14 01:34:45.367779144 (113h30m43.919249605s) [noise=true , address=145.79.227.30:4763 , piece_id=IW3HMSEGIX2TFUY277OSEF4BULPDWA3AO3MNJ7OCUULORPLHNWOQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:50566->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:34:45.367825933 (113h30m43.919296394s) [noise=true , address=145.79.227.16:4912 , piece_id=SKK6OIUPSWXXD6QS5ZCVVORVSFXXOE5UJH7POHTKYY6PU4LX2JGA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection Nov 14 01:34:45.368025623 (113h30m43.919496114s) [noise=true , address=145.79.227.21:4473 , piece_id=DIGA6L2TCAOGHAGUY5K4T3R7H66CF3JLXBXKMC7TPTXVKBCLDJBA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: read tcp 188.166.225.170:49576->145.79.227.21:4473: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:34:45.368068755 (113h30m43.919539206s) [seg_pos={0 0}, upload=99885]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 14 01:34:45.634359530 (113h30m44.185829991s) [upload=99885]: Done uploading segments... Nov 14 01:34:45.634431096 (113h30m44.185901557s) [upload=99885]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.080093206s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.698533582s trace: https://jaeger.dev.storj.io/trace/5bc86ccfe33cf550 FAIL (2025-11-14 01:36:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ce_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288397607 (113h30m2.839868078s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.288507285 (113h30m2.839977746s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.289077772 (113h30m2.840548233s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290074169 (113h30m2.84154464s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.290622854 (113h30m2.842093345s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292062385 (113h30m2.843532866s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.292958642 (113h30m2.844429123s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.294563758 (113h30m2.846034229s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.296755000 (113h30m2.848225462s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.298618379 (113h30m2.85008884s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.300188338 (113h30m2.85165881s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.301125483 (113h30m2.852595944s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302545266 (113h30m2.854015727s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.302898339 (113h30m2.8543688s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.303825163 (113h30m2.855295624s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.316046324 (113h30m2.867516785s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:04.318180529 (113h30m2.86965099s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Uploading piece... Nov 14 01:34:05.581380111 (113h30m4.132850592s) [upload=99881]: Next returned nil segment Nov 14 01:34:05.581407382 (113h30m4.132877844s) [upload=99881]: Waiting for error group managing segments... Nov 14 01:34:05.921914348 (113h30m4.473384809s) [noise=true , address=64.49.13.27:4967 , piece_id=J6FSO7TBSGCJ5PHCLSGL5C3OY4ODFDRXUK5CJRB4MIMRNMHDJHTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.005586503 (113h30m4.557056964s) [noise=true , address=64.49.13.26:4953 , piece_id=JPGEEO5LGI6LZNFWFNUFC4IZQGFNQJ753RROLFBY5PENKRMSTO6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.026850744 (113h30m4.578321205s) [noise=true , address=64.49.13.137:4498 , piece_id=QH7TY6RISGVZ37FE2ZKPXMV5C73ODMZ6RVWOKRCNIRYPPVBPJCUQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.121926626 (113h30m4.673397087s) [noise=true , address=5.231.73.5:4015 , piece_id=NAL7ZGLDEZIXW2TE5QPX3AMTFGMOTPRGN6T7AYT2643DEBEFFYTQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.235634986 (113h30m4.787105437s) [noise=true , address=77.90.31.19:4852 , piece_id=E6FWZFQO24X5D3LJCJPPO565IJ4N2HK3RAJSG7HRNQIU257EWJSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.277897963 (113h30m4.829368424s) [noise=true , address=45.91.116.5:4328 , piece_id=4EPKGXDIGH6HBCWJZ3T67QABXXYWVESSZJX43UDLMAICNYQBYIRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.283628915 (113h30m4.835099376s) [noise=true , address=45.91.116.11:4126 , piece_id=OLVFJTDS5YDASB4LQJA2T753A6BKQC5B5HSEWXBMNQBXRX7DDG4A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.287808744 (113h30m4.839279205s) [noise=true , address=77.90.31.14:4690 , piece_id=UO5YCVZ26S6LGSXNR65OA6RMHPQ3UHMVHTKHRBJYO4OSKQVQWBFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.300041498 (113h30m4.851511949s) [noise=true , address=77.90.31.13:4648 , piece_id=IAHA4J3CIDSDCABBUMTPTSLODNV5AL6E3CIMUBEMB6MKQ7IFVH6A, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.304400718 (113h30m4.855871179s) [noise=true , address=45.91.116.9:4274 , piece_id=NPJJK37U7AUVJP4GAYRTDC2BB4WVOKFMW5PZ6XA6YVLJTCBGAKSQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.325040680 (113h30m4.876511141s) [noise=true , address=77.90.31.18:4828 , piece_id=IVI2PAA2KURB3AMI2NLNELBU6CJ5YTARKVLNZRWHHCHVYEMP25HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.331007762 (113h30m4.882478223s) [noise=true , address=77.90.31.12:4611 , piece_id=WB6YW6FPAXDX2DQP4LG2TGTZTI4RGJFNQYUEYIN5V3Q3PVG3BLGQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:34:06.481058726 (113h30m5.032529207s) [noise=true , address=45.91.116.4:4965 , piece_id=MHKSWZELOB6BWNFAGPL53C2VPDCXGVM3VS5MEETCOFGT2G2EB2HA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: <nil> Nov 14 01:35:02.598238012 (113h31m1.149708493s) [noise=true , address=145.79.227.37:4573 , piece_id=ZWYGFDSZHTKKRRD2MQT7Z4BMVKOQMBWP6K6JH6QGPVSOHTNLDKCA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50834->145.79.227.37:4573: use of closed network connection Nov 14 01:35:02.599889276 (113h31m1.151359737s) [noise=true , address=145.79.227.52:4455 , piece_id=PSNO4NPPGP3YQZT2LNOUHPULS7MHIQCMOWOVSW6HU5RQEBZ63CFQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1wsyjF5X3iPrJsezUjakUArxc8K88T36yT4rE3yu6q9BMKfycm, address:145.79.227.52:4455): protocol: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55144->145.79.227.52:4455: use of closed network connection Nov 14 01:35:02.600041686 (113h31m1.151512147s) [noise=true , address=145.79.227.49:4344 , piece_id=2XDWE5SZTAV6DTKIVN6MIUBINDAZSGFKEGF3ZLRWXUNNPA43VEQQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:127WWD5hr1ZBkRLCHz3jb7kpqqnxPS6VuZ1MyQDmitFbhTSbJNK, address:145.79.227.49:4344): protocol: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50642->145.79.227.49:4344: use of closed network connection Nov 14 01:35:02.600244733 (113h31m1.151715204s) [noise=true , address=145.79.227.35:4488 , piece_id=6M5F4JPR2PY4WF4MOBJZTE6BNNYMH44J24Z36SOBTWGTQYF67MRA, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:145.79.227.35:4488): protocol: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34396->145.79.227.35:4488: use of closed network connection Nov 14 01:35:02.600569551 (113h31m1.152040022s) [noise=true , address=145.79.227.17:4983 , piece_id=EOSPXJ3SBL5TRUJNDYIAQC3RYIT463WNILSKCPRWRUWCYQTGOBVQ, seg_pos={0 0}, upload=99881]: Done uploading piece. err: ecclient: upload failed (node:12h6MND3rBadG4EjUP18S7izyj23cVBLaB5drFsuNGhcxwAYXSA, address:145.79.227.17:4983): protocol: read tcp 188.166.225.170:33042->145.79.227.17:4983: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:35:02.600603536 (113h31m1.152073997s) [seg_pos={0 0}, upload=99881]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 14 01:35:02.847367167 (113h31m1.398837648s) [upload=99881]: Done uploading segments... Nov 14 01:35:02.847427322 (113h31m1.398897783s) [upload=99881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.698570472s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................X.............X...................X..] duration: 6.015869323s trace: https://jaeger.dev.storj.io/trace/4377cd58781b9764 FAIL (2025-11-14 01:36:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== DXCQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351147808 (113h30m14.902618269s) [noise=true , address=77.90.31.13:4628 , piece_id=WHP562KP7DABPK6XQT5F43QGNNCDFXYYYP775VGZLXIG2UEWKK4A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351381713 (113h30m14.902852174s) [noise=true , address=145.79.227.30:4763 , piece_id=IW3HMSEGIX2TFUY277OSEF4BULPDWA3AO3MNJ7OCUULORPLHNWOQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351673048 (113h30m14.903143509s) [noise=true , address=145.79.227.26:4653 , piece_id=EEKXCCTYRNCRJFVM2QBQVQADQJ2YVDCDOXFIFUP4BYUEVYDGWZQA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.351818645 (113h30m14.903289106s) [noise=true , address=145.79.227.14:4840 , piece_id=3DEHNGEWEG4GA5YTLKGDV4BYUJ7Z6KCVLM343ZB5MYZWEMOASS7A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352062729 (113h30m14.903533191s) [noise=true , address=45.91.116.13:4154 , piece_id=DSC3N7KGSUSEVCEII65IQMYLHEUC5U52NNSREC32Y3Q3MDNKWREQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352224859 (113h30m14.90369532s) [noise=true , address=77.90.31.18:4807 , piece_id=TD3XBJUZ6MM5CB7NMYEIZWQC63U2PZZJPYHIM3OOU7PE3Y7KGCIQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352479703 (113h30m14.903950165s) [noise=true , address=77.90.31.17:4771 , piece_id=ARX2MBQVJSPWKWS2AHE2QBBWNREGMUORBQPNFQJHJDWBT2FCDPAA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352614791 (113h30m14.904085342s) [noise=true , address=145.79.227.21:4473 , piece_id=DIGA6L2TCAOGHAGUY5K4T3R7H66CF3JLXBXKMC7TPTXVKBCLDJBA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.352904944 (113h30m14.904375405s) [noise=true , address=45.91.116.6:4973 , piece_id=NEGS6AAGBP4WRJJ6TXEBNHCV2FXW2OBMRWAWL4RNY4YZZZSI3ZLQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.353329152 (113h30m14.904799613s) [noise=true , address=77.90.31.12:4592 , piece_id=ET6LSZ6LXAAAPVJLBLL4UOIF6SKOKDCKXEXKIQ4OS3RD5J5QB2UQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.353685219 (113h30m14.90515568s) [noise=true , address=145.79.227.16:4912 , piece_id=SKK6OIUPSWXXD6QS5ZCVVORVSFXXOE5UJH7POHTKYY6PU4LX2JGA, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354114446 (113h30m14.905584907s) [noise=true , address=45.91.116.12:4225 , piece_id=RMDFCX4JGDRPQ46PPTYAWJE7NIYVDE3J2LCKREBFXHR7I6U2G52A, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354334907 (113h30m14.905805368s) [noise=true , address=145.79.227.13:4804 , piece_id=NZL7G7WFABHCYGW63IPFVWP45AMPZRDYLDF4JNKJHK5PLSHYHLSQ, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:16.354804771 (113h30m14.906275222s) [noise=true , address=145.79.227.20:4439 , piece_id=FEDAGRFBHNL6ES3XTCRWPEI4NN2CMMDJIVNW5WOG3KYD7VXFXA7Q, seg_pos={0 0}, upload=99885]: Uploading piece... Nov 14 01:34:18.090813414 (113h30m16.642283865s) [upload=99885]: Next returned nil segment Nov 14 01:34:18.090832882 (113h30m16.642303333s) [upload=99885]: Waiting for error group managing segments... Nov 14 01:34:18.376699271 (113h30m16.928169732s) [noise=true , address=45.91.116.12:4225 , piece_id=RMDFCX4JGDRPQ46PPTYAWJE7NIYVDE3J2LCKREBFXHR7I6U2G52A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.449067032 (113h30m17.000537493s) [noise=true , address=77.90.31.18:4807 , piece_id=TD3XBJUZ6MM5CB7NMYEIZWQC63U2PZZJPYHIM3OOU7PE3Y7KGCIQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.476540927 (113h30m17.028011378s) [noise=true , address=77.90.31.13:4628 , piece_id=WHP562KP7DABPK6XQT5F43QGNNCDFXYYYP775VGZLXIG2UEWKK4A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.502738892 (113h30m17.054209343s) [noise=true , address=77.90.31.12:4592 , piece_id=ET6LSZ6LXAAAPVJLBLL4UOIF6SKOKDCKXEXKIQ4OS3RD5J5QB2UQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.504520516 (113h30m17.055990977s) [noise=true , address=45.91.116.6:4973 , piece_id=NEGS6AAGBP4WRJJ6TXEBNHCV2FXW2OBMRWAWL4RNY4YZZZSI3ZLQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.504778116 (113h30m17.056248577s) [noise=true , address=45.91.116.13:4154 , piece_id=DSC3N7KGSUSEVCEII65IQMYLHEUC5U52NNSREC32Y3Q3MDNKWREQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.511639541 (113h30m17.063110002s) [noise=true , address=77.90.31.10:4517 , piece_id=GDOCT6DE4NL56DWKOS3JNSQVIVE342SDIZ5YR7XPNRXYPOQYDXCQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:18.529697226 (113h30m17.081167687s) [noise=true , address=77.90.31.17:4771 , piece_id=ARX2MBQVJSPWKWS2AHE2QBBWNREGMUORBQPNFQJHJDWBT2FCDPAA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: <nil> Nov 14 01:34:45.367038063 (113h30m43.918508514s) [noise=true , address=145.79.227.26:4653 , piece_id=EEKXCCTYRNCRJFVM2QBQVQADQJ2YVDCDOXFIFUP4BYUEVYDGWZQA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57142->145.79.227.26:4653: use of closed network connection Nov 14 01:34:45.367162339 (113h30m43.91863279s) [noise=true , address=145.79.227.14:4840 , piece_id=3DEHNGEWEG4GA5YTLKGDV4BYUJ7Z6KCVLM343ZB5MYZWEMOASS7A, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49430->145.79.227.14:4840: use of closed network connection Nov 14 01:34:45.367201353 (113h30m43.918671804s) [noise=true , address=145.79.227.9:4366 , piece_id=GXQSDBFPFZPZ352A5KBNQJS5A32C7ZO522AHCHO3A27P4LAMUFZQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55662->145.79.227.9:4366: use of closed network connection Nov 14 01:34:45.367247291 (113h30m43.918717742s) [noise=true , address=145.79.227.20:4439 , piece_id=FEDAGRFBHNL6ES3XTCRWPEI4NN2CMMDJIVNW5WOG3KYD7VXFXA7Q, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection; write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35070->145.79.227.20:4439: use of closed network connection Nov 14 01:34:45.367425070 (113h30m43.918895531s) [noise=true , address=145.79.227.13:4804 , piece_id=NZL7G7WFABHCYGW63IPFVWP45AMPZRDYLDF4JNKJHK5PLSHYHLSQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50818->145.79.227.13:4804: use of closed network connection Nov 14 01:34:45.367779144 (113h30m43.919249605s) [noise=true , address=145.79.227.30:4763 , piece_id=IW3HMSEGIX2TFUY277OSEF4BULPDWA3AO3MNJ7OCUULORPLHNWOQ, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:50566->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:34:45.367825933 (113h30m43.919296394s) [noise=true , address=145.79.227.16:4912 , piece_id=SKK6OIUPSWXXD6QS5ZCVVORVSFXXOE5UJH7POHTKYY6PU4LX2JGA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51772->145.79.227.16:4912: use of closed network connection Nov 14 01:34:45.368025623 (113h30m43.919496114s) [noise=true , address=145.79.227.21:4473 , piece_id=DIGA6L2TCAOGHAGUY5K4T3R7H66CF3JLXBXKMC7TPTXVKBCLDJBA, seg_pos={0 0}, upload=99885]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: read tcp 188.166.225.170:49576->145.79.227.21:4473: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 14 01:34:45.368068755 (113h30m43.919539206s) [seg_pos={0 0}, upload=99885]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 14 01:34:45.634359530 (113h30m44.185829991s) [upload=99885]: Done uploading segments... Nov 14 01:34:45.634431096 (113h30m44.185901557s) [upload=99885]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.015898079s core uplink us-select-1

Orbiter Check History

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)