Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.804466687s trace: https://jaeger.dev.storj.io/trace/3c605efa50984721 FAIL (2025-11-28 07:52:28 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 5.80449516s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.933168511s trace: https://jaeger.dev.storj.io/trace/3c605efa50984721 FAIL (2025-11-28 07:51:37 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 5.933197027s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.779879943s trace: https://jaeger.dev.storj.io/trace/3c605efa50984721 FAIL (2025-11-28 07:50:57 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 5.77989432s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 6.02389727s trace: https://jaeger.dev.storj.io/trace/3c605efa50984721 FAIL (2025-11-28 07:50:03 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 6.024166463s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.949396173s trace: https://jaeger.dev.storj.io/trace/3c605efa50984721 FAIL (2025-11-28 07:48:55 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 5.949481916s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 6.574725535s trace: https://jaeger.dev.storj.io/trace/3c605efa50984721 FAIL (2025-11-28 07:48:24 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 6.574756965s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.882705632s trace: https://jaeger.dev.storj.io/trace/3c605efa50984721 FAIL (2025-11-28 07:47:47 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 5.882723856s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 6.644258993s trace: https://jaeger.dev.storj.io/trace/3c605efa50984721 FAIL (2025-11-28 07:47:11 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 6.644294261s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 6.055640679s trace: https://jaeger.dev.storj.io/trace/3c605efa50984721 FAIL (2025-11-28 07:46:18 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 6.055668803s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.980118142s trace: https://jaeger.dev.storj.io/trace/3c605efa50984721 FAIL (2025-11-28 07:46:03 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 5.980148581s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 6.579350384s trace: https://jaeger.dev.storj.io/trace/3c605efa50984721 FAIL (2025-11-28 07:45:27 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 6.579380241s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.694161865s trace: https://jaeger.dev.storj.io/trace/3c605efa50984721 FAIL (2025-11-28 07:44:58 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 5.694177354s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.904978813s trace: https://jaeger.dev.storj.io/trace/3c605efa50984721 FAIL (2025-11-28 07:43:46 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 5.905005204s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.889253429s trace: https://jaeger.dev.storj.io/trace/3c605efa50984721 FAIL (2025-11-28 07:43:25 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 5.889284197s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.948664037s trace: https://jaeger.dev.storj.io/trace/3c605efa50984721 FAIL (2025-11-28 07:42:55 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 5.948690478s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.452836308s trace: https://jaeger.dev.storj.io/trace/3c605efa50984721 FAIL (2025-11-28 07:42:39 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 6.452864422s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.835170203s trace: https://jaeger.dev.storj.io/trace/3c605efa50984721 FAIL (2025-11-28 07:42:19 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 5.835192325s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 6.028405929s trace: https://jaeger.dev.storj.io/trace/3c605efa50984721 FAIL (2025-11-28 07:41:38 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 6.028439882s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001113571s trace: https://jaeger.dev.storj.io/trace/3c605efa50984721 FAIL (2025-11-28 07:40:54 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 30.001138647s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.909468296s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 22:18:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.909507721s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 6.016451467s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 22:17:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.016546939s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.939139432s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 22:17:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.939170843s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X..X........................................................] duration: 5.828313628s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 22:16:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.82835212s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X..X.......................................................] duration: 6.519851435s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 22:15:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.51986962s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X..X......................................................] duration: 5.78433412s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 22:14:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.784362223s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X..X.....................................................] duration: 5.681337444s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 22:13:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.681441673s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X..X....................................................] duration: 5.886300034s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 22:12:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.886324432s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X..X...................................................] duration: 5.892567251s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 22:12:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.89261881s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X..X..................................................] duration: 5.880079492s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 22:11:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.880112395s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X..X.................................................] duration: 5.764569531s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 22:11:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.764602173s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X..X................................................] duration: 5.856799631s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 22:10:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.856824968s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X..X...............................................] duration: 5.686455445s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 22:09:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.686472266s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X..X..............................................] duration: 5.829957452s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 22:09:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.82998867s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X..X.............................................] duration: 6.013269313s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 22:08:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.013296485s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X..X............................................] duration: 5.793148488s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 22:08:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.793177794s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X............X..X...........................................] duration: 5.683723974s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 22:07:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.683763148s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X............X..X..........................................] duration: 5.729923061s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 22:06:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.729965672s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X............X..X.........................................] duration: 5.801676219s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 22:05:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.801693512s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X............X..X........................................] duration: 5.701997209s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 22:04:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.702019241s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X............X..X.......................................] duration: 5.754840795s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 22:04:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.754868949s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X............X..X......................................] duration: 5.848744624s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 22:03:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.848775824s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X............X..X.....................................] duration: 5.93033344s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 22:02:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.930367844s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X............X..X....................................] duration: 5.73670085s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 22:01:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.736749201s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X.......X............X..X...................................] duration: 5.95980143s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 22:01:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.959835524s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X.......X............X..X..................................] duration: 5.790520266s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 22:00:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.790549694s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X.......X............X..X.................................] duration: 6.028783165s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 22:00:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.028820225s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X.......X............X..X................................] duration: 5.77107242s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 21:59:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.77109919s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X.......X............X..X...............................] duration: 5.751424946s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 21:58:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.751446468s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X.......X............X..X..............................] duration: 5.725139144s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 21:58:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.725172298s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......X.......X............X..X.............................] duration: 5.879892593s trace: https://jaeger.dev.storj.io/trace/3aa215075bb22d6f FAIL (2025-11-27 21:57:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 27 21:35:14.055059723 (445h31m12.606530184s) [upload=392315]: Starting upload Nov 27 21:35:14.055190261 (445h31m12.606660722s) [upload=392315]: Done starting upload Nov 27 21:35:14.055401814 (445h31m12.606872315s) [upload=392315]: Uploading segments... Nov 27 21:35:14.055565195 (445h31m12.607035686s) [upload=392315]: Got next segment. Inline: false Nov 27 21:35:14.055621453 (445h31m12.607091924s) [upload=392315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 21:35:14.055630099 (445h31m12.60710058s) [upload=392315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 21:35:14.299822401 (445h31m12.851292862s) [seg_pos={0 0}, upload=392315]: Begin upload segment... Nov 27 21:35:14.299881303 (445h31m12.851351764s) [seg_pos={0 0}, upload=392315]: Done begin upload segment. Nov 27 21:35:14.299916139 (445h31m12.8513866s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.301362764 (445h31m12.852833225s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.302421910 (445h31m12.853892371s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303044917 (445h31m12.854515368s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.303689244 (445h31m12.855159705s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.305554125 (445h31m12.857024586s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.308535833 (445h31m12.860006294s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310242392 (445h31m12.861712853s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.310394952 (445h31m12.861865403s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312151979 (445h31m12.86362244s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312616994 (445h31m12.864087455s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.312644086 (445h31m12.864114547s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.314687747 (445h31m12.866158208s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.316482976 (445h31m12.867953437s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.320653797 (445h31m12.872124248s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:14.325068454 (445h31m12.876538915s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Uploading piece... Nov 27 21:35:16.063892479 (445h31m14.61536293s) [upload=392315]: Next returned nil segment Nov 27 21:35:16.063909361 (445h31m14.615379822s) [upload=392315]: Waiting for error group managing segments... Nov 27 21:35:16.402942749 (445h31m14.95441321s) [noise=true , address=77.90.31.12:4592 , piece_id=MQBRQKGIIUMGF32HUHSPUSXRYTXCFBP5EC4PV6OAWYEM5DY2FLMA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:16.407751456 (445h31m14.959221917s) [noise=true , address=45.91.116.4:4940 , piece_id=BXETT2H3RUXVN2DAPTCF4PBQCJPBZGSBGFKBIEHJGAGABTBEVUSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.647881947 (445h31m16.199352408s) [noise=true , address=145.79.227.13:4804 , piece_id=2HEAZ7G2GZWMGBCEOXGEFX4DFUB52RYAGRFAUG2XLIABWT5JPNSA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.766691765 (445h31m16.318162216s) [noise=true , address=145.79.227.9:4365 , piece_id=LEZ2FXCPPM4QP7OK2WBPJG7AZTS6WZ36VTGE55RSIOON2AGGVGRQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.854669572 (445h31m16.406140033s) [noise=true , address=145.79.227.32:4836 , piece_id=Q7ZUMV5DQME4W52OS3OUAEKZAMP5VLTHWYPRBCVWPE7WJBGUYXAA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:17.908619224 (445h31m16.460089685s) [noise=true , address=145.79.227.9:4366 , piece_id=2BRKANBUVNVBXUDZWWAD7K75CX7YYQKICORSSZVYS37J34TEFFHA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.433711877 (445h31m16.985182338s) [noise=true , address=145.79.227.44:4194 , piece_id=LJB7YP6CXFXDX7ZOBAZ73OBOYLZRFMHI6LHOKGGJMP4WVXOHYTWA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.667200106 (445h31m17.218670567s) [noise=true , address=145.79.227.21:4474 , piece_id=NCQXZGU5AHOUK4T4BRLYLXKHWKUS5YVAVYU4B5DWIJ2JFK4HHOUQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.668977781 (445h31m17.220448242s) [noise=true , address=145.79.227.43:4158 , piece_id=TXVMTYKAMV6S2SKZ7KTAH2QBIUR27MWKM3JYDDFYAZYCVJBU3JXQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.853987367 (445h31m17.405457818s) [noise=true , address=145.79.227.28:4727 , piece_id=L4A3WS7FRFMG47VN4FU2XM3GQPPZJCW7IOMFEJMAWLTE4YMZ7HKA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:18.962431827 (445h31m17.513902289s) [noise=true , address=145.79.227.34:4907 , piece_id=LF32SQFVL2VMCTMJXZRCA4FMJQWWVHJNMMUHC73ST5KCEMA75JFA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:19.919583248 (445h31m18.471053709s) [noise=true , address=145.79.227.10:4401 , piece_id=BR65KAL4NOMLAGO3GBJZQR7QJ6XMY5TNOVYGLAL53YJA7SXDHCFQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: <nil> Nov 27 21:35:43.364821475 (445h31m41.916291936s) [noise=true , address=145.79.227.20:4437 , piece_id=KFRMYAEHJ7PAK4FMKYMDAJI3TA26SX44NEXOACILG3W2LT7XNCLA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, address:145.79.227.20:4437): protocol: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57462->145.79.227.20:4437: use of closed network connection Nov 27 21:35:43.364932427 (445h31m41.916402888s) [noise=true , address=145.79.227.11:4730 , piece_id=MYXMFBVDCWBTEJFT63HINHNQUBEEGOGZAT32DBC5BG4KG5FYHA3A, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47596->145.79.227.11:4730: use of closed network connection Nov 27 21:35:43.365332358 (445h31m41.916802809s) [noise=true , address=145.79.227.22:4509 , piece_id=PV2JUM4IIQGZMQ4YQMRKSJSMMNJU7B4JQIPFOQMXRO2XJI7UO5EQ, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: read tcp 188.166.225.170:41194->145.79.227.22:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 21:35:43.365522461 (445h31m41.916992932s) [noise=true , address=145.79.227.30:4761 , piece_id=M6XRDJ3NY4NY4B65DKJKZWONSI6TNAT2NIV44CTBRFN4NA5PE2ZA, seg_pos={0 0}, upload=392315]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:44412->145.79.227.30:4761: use of closed network connection; context deadline exceeded Nov 27 21:35:43.365558229 (445h31m41.91702868s) [seg_pos={0 0}, upload=392315]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 27 21:35:43.604089540 (445h31m42.155560011s) [upload=392315]: Done uploading segments... Nov 27 21:35:43.604126991 (445h31m42.155597443s) [upload=392315]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.879919613s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)