Orbiter History

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 737.492274ms trace: https://jaeger.dev.storj.io/trace/786d2ef7c9a6f29c FAIL (2025-12-05 00:10:08 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:34402->45.38.24.2:443: read: connection reset by peer Duration: 737.5173ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 716.129947ms trace: https://jaeger.dev.storj.io/trace/786d2ef7c9a6f29c FAIL (2025-12-05 00:09:31 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:34402->45.38.24.2:443: read: connection reset by peer Duration: 716.161988ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 737.639715ms trace: https://jaeger.dev.storj.io/trace/786d2ef7c9a6f29c FAIL (2025-12-05 00:09:00 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:34402->45.38.24.2:443: read: connection reset by peer Duration: 737.669882ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 738.021962ms trace: https://jaeger.dev.storj.io/trace/786d2ef7c9a6f29c FAIL (2025-12-05 00:07:59 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:34402->45.38.24.2:443: read: connection reset by peer Duration: 738.05242ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 739.395177ms trace: https://jaeger.dev.storj.io/trace/786d2ef7c9a6f29c FAIL (2025-12-05 00:07:05 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:34402->45.38.24.2:443: read: connection reset by peer Duration: 739.421878ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 752.502215ms trace: https://jaeger.dev.storj.io/trace/786d2ef7c9a6f29c FAIL (2025-12-05 00:06:23 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:34402->45.38.24.2:443: read: connection reset by peer Duration: 752.52653ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 765.927907ms trace: https://jaeger.dev.storj.io/trace/786d2ef7c9a6f29c FAIL (2025-12-05 00:05:14 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:34402->45.38.24.2:443: read: connection reset by peer Duration: 765.959498ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 742.817624ms trace: https://jaeger.dev.storj.io/trace/786d2ef7c9a6f29c FAIL (2025-12-05 00:04:53 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:34402->45.38.24.2:443: read: connection reset by peer Duration: 742.84671ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 724.047874ms trace: https://jaeger.dev.storj.io/trace/786d2ef7c9a6f29c FAIL (2025-12-05 00:04:08 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:34402->45.38.24.2:443: read: connection reset by peer Duration: 724.074173ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 696.150462ms trace: https://jaeger.dev.storj.io/trace/786d2ef7c9a6f29c FAIL (2025-12-05 00:03:22 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:34402->45.38.24.2:443: read: connection reset by peer Duration: 696.168407ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 725.179038ms trace: https://jaeger.dev.storj.io/trace/786d2ef7c9a6f29c FAIL (2025-12-05 00:02:18 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:34402->45.38.24.2:443: read: connection reset by peer Duration: 725.208483ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 827.382237ms trace: https://jaeger.dev.storj.io/trace/786d2ef7c9a6f29c FAIL (2025-12-05 00:01:11 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:34402->45.38.24.2:443: read: connection reset by peer Duration: 827.412204ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 753.166373ms trace: https://jaeger.dev.storj.io/trace/786d2ef7c9a6f29c FAIL (2025-12-05 00:00:42 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:34402->45.38.24.2:443: read: connection reset by peer Duration: 753.196461ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 714.711002ms trace: https://jaeger.dev.storj.io/trace/786d2ef7c9a6f29c FAIL (2025-12-05 00:00:04 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:34402->45.38.24.2:443: read: connection reset by peer Duration: 714.902177ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 749.088924ms trace: https://jaeger.dev.storj.io/trace/786d2ef7c9a6f29c FAIL (2025-12-04 23:59:07 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:34402->45.38.24.2:443: read: connection reset by peer Duration: 749.129321ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 718.417771ms trace: https://jaeger.dev.storj.io/trace/786d2ef7c9a6f29c FAIL (2025-12-04 23:58:51 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:34402->45.38.24.2:443: read: connection reset by peer Duration: 718.448439ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 746.087579ms trace: https://jaeger.dev.storj.io/trace/786d2ef7c9a6f29c FAIL (2025-12-04 23:58:15 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:34402->45.38.24.2:443: read: connection reset by peer Duration: 746.114631ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 750.089668ms trace: https://jaeger.dev.storj.io/trace/786d2ef7c9a6f29c FAIL (2025-12-04 23:57:53 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:34402->45.38.24.2:443: read: connection reset by peer Duration: 750.119366ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 684.646896ms trace: https://jaeger.dev.storj.io/trace/786d2ef7c9a6f29c FAIL (2025-12-04 23:57:07 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:34402->45.38.24.2:443: read: connection reset by peer Duration: 684.671882ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 711.507953ms trace: https://jaeger.dev.storj.io/trace/786d2ef7c9a6f29c FAIL (2025-12-04 23:56:01 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:34402->45.38.24.2:443: read: connection reset by peer Duration: 711.539495ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 674.872373ms trace: https://jaeger.dev.storj.io/trace/786d2ef7c9a6f29c FAIL (2025-12-04 23:55:34 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:34402->45.38.24.2:443: read: connection reset by peer Duration: 674.895917ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.878566258s trace: https://jaeger.dev.storj.io/trace/53db3ad570a94624 FAIL (2025-12-04 23:40:27 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 4 22:55:49.159528058 (30h0m25.630738787s) [upload=26406]: Starting upload Dec 4 22:55:49.159656774 (30h0m25.630867513s) [upload=26406]: Done starting upload Dec 4 22:55:49.159698914 (30h0m25.630909653s) [upload=26406]: Uploading segments... Dec 4 22:55:49.159881322 (30h0m25.631092061s) [upload=26406]: Got next segment. Inline: false Dec 4 22:55:49.159921538 (30h0m25.631132267s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 22:55:49.159927079 (30h0m25.631137808s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 22:55:49.407632244 (30h0m25.878842983s) [seg_pos={0 0}, upload=26406]: Begin upload segment... Dec 4 22:55:49.407685676 (30h0m25.878896415s) [seg_pos={0 0}, upload=26406]: Done begin upload segment. Dec 4 22:55:49.407698249 (30h0m25.878908988s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.407776349 (30h0m25.878987078s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.408596580 (30h0m25.879807319s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409349504 (30h0m25.880560243s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409963062 (30h0m25.881173801s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410210894 (30h0m25.881421623s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410541033 (30h0m25.881751772s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411049931 (30h0m25.88226068s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411451476 (30h0m25.882662215s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411905290 (30h0m25.883116039s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412457602 (30h0m25.883668331s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412740861 (30h0m25.8839516s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413363186 (30h0m25.884573926s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413762196 (30h0m25.884972935s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414199659 (30h0m25.885410398s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414579071 (30h0m25.88578981s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:50.491991030 (30h0m26.963201759s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 22:55:51.092442279 (30h0m27.563653008s) [upload=26406]: Next returned nil segment Dec 4 22:55:51.092455273 (30h0m27.563666012s) [upload=26406]: Waiting for error group managing segments... Dec 4 22:55:51.416315726 (30h0m27.887526465s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.416474488 (30h0m27.887685217s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.418547936 (30h0m27.889758665s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.431075520 (30h0m27.902286249s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.438237447 (30h0m27.909448186s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.444558713 (30h0m27.915769452s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.454863965 (30h0m27.926074704s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.500395572 (30h0m27.971606301s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.002667307 (30h0m28.473878056s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.056889902 (30h0m28.528100641s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.357024719 (30h0m28.828235448s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.423527005 (30h0m28.894737734s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759657335 (30h0m54.230868064s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759677794 (30h0m54.230888523s) [seg_pos={0 0}, upload=26406]: Segment reached optimal threshold of 13 pieces. Dec 4 22:56:17.759902682 (30h0m54.231113661s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection Dec 4 22:56:17.759916668 (30h0m54.231127407s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760005618 (30h0m54.231216347s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection Dec 4 22:56:17.760014334 (30h0m54.231225063s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760032339 (30h0m54.231243068s) [seg_pos={0 0}, upload=26406]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 22:56:17.760113693 (30h0m54.231324423s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 22:56:17.760119615 (30h0m54.231330344s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 22:56:18.057470049 (30h0m54.528680788s) [upload=26406]: Done uploading segments... Dec 4 22:56:18.057547447 (30h0m54.528758187s) [upload=26406]: Upload finished. err: <nil> Duration: 5.878727505s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.872639086s trace: https://jaeger.dev.storj.io/trace/53db3ad570a94624 FAIL (2025-12-04 23:40:06 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 4 22:55:49.159528058 (30h0m25.630738787s) [upload=26406]: Starting upload Dec 4 22:55:49.159656774 (30h0m25.630867513s) [upload=26406]: Done starting upload Dec 4 22:55:49.159698914 (30h0m25.630909653s) [upload=26406]: Uploading segments... Dec 4 22:55:49.159881322 (30h0m25.631092061s) [upload=26406]: Got next segment. Inline: false Dec 4 22:55:49.159921538 (30h0m25.631132267s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 22:55:49.159927079 (30h0m25.631137808s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 22:55:49.407632244 (30h0m25.878842983s) [seg_pos={0 0}, upload=26406]: Begin upload segment... Dec 4 22:55:49.407685676 (30h0m25.878896415s) [seg_pos={0 0}, upload=26406]: Done begin upload segment. Dec 4 22:55:49.407698249 (30h0m25.878908988s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.407776349 (30h0m25.878987078s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.408596580 (30h0m25.879807319s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409349504 (30h0m25.880560243s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409963062 (30h0m25.881173801s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410210894 (30h0m25.881421623s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410541033 (30h0m25.881751772s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411049931 (30h0m25.88226068s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411451476 (30h0m25.882662215s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411905290 (30h0m25.883116039s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412457602 (30h0m25.883668331s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412740861 (30h0m25.8839516s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413363186 (30h0m25.884573926s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413762196 (30h0m25.884972935s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414199659 (30h0m25.885410398s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414579071 (30h0m25.88578981s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:50.491991030 (30h0m26.963201759s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 22:55:51.092442279 (30h0m27.563653008s) [upload=26406]: Next returned nil segment Dec 4 22:55:51.092455273 (30h0m27.563666012s) [upload=26406]: Waiting for error group managing segments... Dec 4 22:55:51.416315726 (30h0m27.887526465s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.416474488 (30h0m27.887685217s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.418547936 (30h0m27.889758665s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.431075520 (30h0m27.902286249s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.438237447 (30h0m27.909448186s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.444558713 (30h0m27.915769452s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.454863965 (30h0m27.926074704s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.500395572 (30h0m27.971606301s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.002667307 (30h0m28.473878056s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.056889902 (30h0m28.528100641s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.357024719 (30h0m28.828235448s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.423527005 (30h0m28.894737734s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759657335 (30h0m54.230868064s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759677794 (30h0m54.230888523s) [seg_pos={0 0}, upload=26406]: Segment reached optimal threshold of 13 pieces. Dec 4 22:56:17.759902682 (30h0m54.231113661s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection Dec 4 22:56:17.759916668 (30h0m54.231127407s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760005618 (30h0m54.231216347s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection Dec 4 22:56:17.760014334 (30h0m54.231225063s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760032339 (30h0m54.231243068s) [seg_pos={0 0}, upload=26406]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 22:56:17.760113693 (30h0m54.231324423s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 22:56:17.760119615 (30h0m54.231330344s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 22:56:18.057470049 (30h0m54.528680788s) [upload=26406]: Done uploading segments... Dec 4 22:56:18.057547447 (30h0m54.528758187s) [upload=26406]: Upload finished. err: <nil> Duration: 5.872675867s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.744468012s trace: https://jaeger.dev.storj.io/trace/53db3ad570a94624 FAIL (2025-12-04 23:39:19 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 4 22:55:49.159528058 (30h0m25.630738787s) [upload=26406]: Starting upload Dec 4 22:55:49.159656774 (30h0m25.630867513s) [upload=26406]: Done starting upload Dec 4 22:55:49.159698914 (30h0m25.630909653s) [upload=26406]: Uploading segments... Dec 4 22:55:49.159881322 (30h0m25.631092061s) [upload=26406]: Got next segment. Inline: false Dec 4 22:55:49.159921538 (30h0m25.631132267s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 22:55:49.159927079 (30h0m25.631137808s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 22:55:49.407632244 (30h0m25.878842983s) [seg_pos={0 0}, upload=26406]: Begin upload segment... Dec 4 22:55:49.407685676 (30h0m25.878896415s) [seg_pos={0 0}, upload=26406]: Done begin upload segment. Dec 4 22:55:49.407698249 (30h0m25.878908988s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.407776349 (30h0m25.878987078s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.408596580 (30h0m25.879807319s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409349504 (30h0m25.880560243s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409963062 (30h0m25.881173801s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410210894 (30h0m25.881421623s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410541033 (30h0m25.881751772s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411049931 (30h0m25.88226068s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411451476 (30h0m25.882662215s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411905290 (30h0m25.883116039s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412457602 (30h0m25.883668331s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412740861 (30h0m25.8839516s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413363186 (30h0m25.884573926s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413762196 (30h0m25.884972935s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414199659 (30h0m25.885410398s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414579071 (30h0m25.88578981s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:50.491991030 (30h0m26.963201759s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 22:55:51.092442279 (30h0m27.563653008s) [upload=26406]: Next returned nil segment Dec 4 22:55:51.092455273 (30h0m27.563666012s) [upload=26406]: Waiting for error group managing segments... Dec 4 22:55:51.416315726 (30h0m27.887526465s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.416474488 (30h0m27.887685217s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.418547936 (30h0m27.889758665s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.431075520 (30h0m27.902286249s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.438237447 (30h0m27.909448186s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.444558713 (30h0m27.915769452s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.454863965 (30h0m27.926074704s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.500395572 (30h0m27.971606301s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.002667307 (30h0m28.473878056s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.056889902 (30h0m28.528100641s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.357024719 (30h0m28.828235448s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.423527005 (30h0m28.894737734s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759657335 (30h0m54.230868064s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759677794 (30h0m54.230888523s) [seg_pos={0 0}, upload=26406]: Segment reached optimal threshold of 13 pieces. Dec 4 22:56:17.759902682 (30h0m54.231113661s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection Dec 4 22:56:17.759916668 (30h0m54.231127407s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760005618 (30h0m54.231216347s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection Dec 4 22:56:17.760014334 (30h0m54.231225063s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760032339 (30h0m54.231243068s) [seg_pos={0 0}, upload=26406]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 22:56:17.760113693 (30h0m54.231324423s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 22:56:17.760119615 (30h0m54.231330344s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 22:56:18.057470049 (30h0m54.528680788s) [upload=26406]: Done uploading segments... Dec 4 22:56:18.057547447 (30h0m54.528758187s) [upload=26406]: Upload finished. err: <nil> Duration: 5.744498731s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.828589382s trace: https://jaeger.dev.storj.io/trace/53db3ad570a94624 FAIL (2025-12-04 23:38:52 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 4 22:55:49.159528058 (30h0m25.630738787s) [upload=26406]: Starting upload Dec 4 22:55:49.159656774 (30h0m25.630867513s) [upload=26406]: Done starting upload Dec 4 22:55:49.159698914 (30h0m25.630909653s) [upload=26406]: Uploading segments... Dec 4 22:55:49.159881322 (30h0m25.631092061s) [upload=26406]: Got next segment. Inline: false Dec 4 22:55:49.159921538 (30h0m25.631132267s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 22:55:49.159927079 (30h0m25.631137808s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 22:55:49.407632244 (30h0m25.878842983s) [seg_pos={0 0}, upload=26406]: Begin upload segment... Dec 4 22:55:49.407685676 (30h0m25.878896415s) [seg_pos={0 0}, upload=26406]: Done begin upload segment. Dec 4 22:55:49.407698249 (30h0m25.878908988s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.407776349 (30h0m25.878987078s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.408596580 (30h0m25.879807319s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409349504 (30h0m25.880560243s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409963062 (30h0m25.881173801s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410210894 (30h0m25.881421623s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410541033 (30h0m25.881751772s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411049931 (30h0m25.88226068s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411451476 (30h0m25.882662215s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411905290 (30h0m25.883116039s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412457602 (30h0m25.883668331s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412740861 (30h0m25.8839516s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413363186 (30h0m25.884573926s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413762196 (30h0m25.884972935s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414199659 (30h0m25.885410398s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414579071 (30h0m25.88578981s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:50.491991030 (30h0m26.963201759s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 22:55:51.092442279 (30h0m27.563653008s) [upload=26406]: Next returned nil segment Dec 4 22:55:51.092455273 (30h0m27.563666012s) [upload=26406]: Waiting for error group managing segments... Dec 4 22:55:51.416315726 (30h0m27.887526465s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.416474488 (30h0m27.887685217s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.418547936 (30h0m27.889758665s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.431075520 (30h0m27.902286249s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.438237447 (30h0m27.909448186s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.444558713 (30h0m27.915769452s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.454863965 (30h0m27.926074704s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.500395572 (30h0m27.971606301s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.002667307 (30h0m28.473878056s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.056889902 (30h0m28.528100641s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.357024719 (30h0m28.828235448s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.423527005 (30h0m28.894737734s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759657335 (30h0m54.230868064s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759677794 (30h0m54.230888523s) [seg_pos={0 0}, upload=26406]: Segment reached optimal threshold of 13 pieces. Dec 4 22:56:17.759902682 (30h0m54.231113661s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection Dec 4 22:56:17.759916668 (30h0m54.231127407s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760005618 (30h0m54.231216347s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection Dec 4 22:56:17.760014334 (30h0m54.231225063s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760032339 (30h0m54.231243068s) [seg_pos={0 0}, upload=26406]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 22:56:17.760113693 (30h0m54.231324423s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 22:56:17.760119615 (30h0m54.231330344s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 22:56:18.057470049 (30h0m54.528680788s) [upload=26406]: Done uploading segments... Dec 4 22:56:18.057547447 (30h0m54.528758187s) [upload=26406]: Upload finished. err: <nil> Duration: 5.828620954s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.797440941s trace: https://jaeger.dev.storj.io/trace/53db3ad570a94624 FAIL (2025-12-04 23:38:03 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 4 22:55:49.159528058 (30h0m25.630738787s) [upload=26406]: Starting upload Dec 4 22:55:49.159656774 (30h0m25.630867513s) [upload=26406]: Done starting upload Dec 4 22:55:49.159698914 (30h0m25.630909653s) [upload=26406]: Uploading segments... Dec 4 22:55:49.159881322 (30h0m25.631092061s) [upload=26406]: Got next segment. Inline: false Dec 4 22:55:49.159921538 (30h0m25.631132267s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 22:55:49.159927079 (30h0m25.631137808s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 22:55:49.407632244 (30h0m25.878842983s) [seg_pos={0 0}, upload=26406]: Begin upload segment... Dec 4 22:55:49.407685676 (30h0m25.878896415s) [seg_pos={0 0}, upload=26406]: Done begin upload segment. Dec 4 22:55:49.407698249 (30h0m25.878908988s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.407776349 (30h0m25.878987078s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.408596580 (30h0m25.879807319s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409349504 (30h0m25.880560243s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409963062 (30h0m25.881173801s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410210894 (30h0m25.881421623s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410541033 (30h0m25.881751772s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411049931 (30h0m25.88226068s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411451476 (30h0m25.882662215s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411905290 (30h0m25.883116039s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412457602 (30h0m25.883668331s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412740861 (30h0m25.8839516s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413363186 (30h0m25.884573926s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413762196 (30h0m25.884972935s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414199659 (30h0m25.885410398s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414579071 (30h0m25.88578981s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:50.491991030 (30h0m26.963201759s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 22:55:51.092442279 (30h0m27.563653008s) [upload=26406]: Next returned nil segment Dec 4 22:55:51.092455273 (30h0m27.563666012s) [upload=26406]: Waiting for error group managing segments... Dec 4 22:55:51.416315726 (30h0m27.887526465s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.416474488 (30h0m27.887685217s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.418547936 (30h0m27.889758665s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.431075520 (30h0m27.902286249s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.438237447 (30h0m27.909448186s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.444558713 (30h0m27.915769452s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.454863965 (30h0m27.926074704s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.500395572 (30h0m27.971606301s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.002667307 (30h0m28.473878056s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.056889902 (30h0m28.528100641s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.357024719 (30h0m28.828235448s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.423527005 (30h0m28.894737734s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759657335 (30h0m54.230868064s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759677794 (30h0m54.230888523s) [seg_pos={0 0}, upload=26406]: Segment reached optimal threshold of 13 pieces. Dec 4 22:56:17.759902682 (30h0m54.231113661s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection Dec 4 22:56:17.759916668 (30h0m54.231127407s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760005618 (30h0m54.231216347s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection Dec 4 22:56:17.760014334 (30h0m54.231225063s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760032339 (30h0m54.231243068s) [seg_pos={0 0}, upload=26406]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 22:56:17.760113693 (30h0m54.231324423s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 22:56:17.760119615 (30h0m54.231330344s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 22:56:18.057470049 (30h0m54.528680788s) [upload=26406]: Done uploading segments... Dec 4 22:56:18.057547447 (30h0m54.528758187s) [upload=26406]: Upload finished. err: <nil> Duration: 5.797457362s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.72355504s trace: https://jaeger.dev.storj.io/trace/53db3ad570a94624 FAIL (2025-12-04 23:37:21 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 4 22:55:49.159528058 (30h0m25.630738787s) [upload=26406]: Starting upload Dec 4 22:55:49.159656774 (30h0m25.630867513s) [upload=26406]: Done starting upload Dec 4 22:55:49.159698914 (30h0m25.630909653s) [upload=26406]: Uploading segments... Dec 4 22:55:49.159881322 (30h0m25.631092061s) [upload=26406]: Got next segment. Inline: false Dec 4 22:55:49.159921538 (30h0m25.631132267s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 22:55:49.159927079 (30h0m25.631137808s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 22:55:49.407632244 (30h0m25.878842983s) [seg_pos={0 0}, upload=26406]: Begin upload segment... Dec 4 22:55:49.407685676 (30h0m25.878896415s) [seg_pos={0 0}, upload=26406]: Done begin upload segment. Dec 4 22:55:49.407698249 (30h0m25.878908988s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.407776349 (30h0m25.878987078s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.408596580 (30h0m25.879807319s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409349504 (30h0m25.880560243s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409963062 (30h0m25.881173801s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410210894 (30h0m25.881421623s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410541033 (30h0m25.881751772s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411049931 (30h0m25.88226068s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411451476 (30h0m25.882662215s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411905290 (30h0m25.883116039s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412457602 (30h0m25.883668331s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412740861 (30h0m25.8839516s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413363186 (30h0m25.884573926s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413762196 (30h0m25.884972935s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414199659 (30h0m25.885410398s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414579071 (30h0m25.88578981s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:50.491991030 (30h0m26.963201759s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 22:55:51.092442279 (30h0m27.563653008s) [upload=26406]: Next returned nil segment Dec 4 22:55:51.092455273 (30h0m27.563666012s) [upload=26406]: Waiting for error group managing segments... Dec 4 22:55:51.416315726 (30h0m27.887526465s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.416474488 (30h0m27.887685217s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.418547936 (30h0m27.889758665s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.431075520 (30h0m27.902286249s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.438237447 (30h0m27.909448186s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.444558713 (30h0m27.915769452s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.454863965 (30h0m27.926074704s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.500395572 (30h0m27.971606301s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.002667307 (30h0m28.473878056s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.056889902 (30h0m28.528100641s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.357024719 (30h0m28.828235448s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.423527005 (30h0m28.894737734s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759657335 (30h0m54.230868064s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759677794 (30h0m54.230888523s) [seg_pos={0 0}, upload=26406]: Segment reached optimal threshold of 13 pieces. Dec 4 22:56:17.759902682 (30h0m54.231113661s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection Dec 4 22:56:17.759916668 (30h0m54.231127407s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760005618 (30h0m54.231216347s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection Dec 4 22:56:17.760014334 (30h0m54.231225063s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760032339 (30h0m54.231243068s) [seg_pos={0 0}, upload=26406]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 22:56:17.760113693 (30h0m54.231324423s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 22:56:17.760119615 (30h0m54.231330344s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 22:56:18.057470049 (30h0m54.528680788s) [upload=26406]: Done uploading segments... Dec 4 22:56:18.057547447 (30h0m54.528758187s) [upload=26406]: Upload finished. err: <nil> Duration: 5.723587062s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.713831261s trace: https://jaeger.dev.storj.io/trace/53db3ad570a94624 FAIL (2025-12-04 23:36:32 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 4 22:55:49.159528058 (30h0m25.630738787s) [upload=26406]: Starting upload Dec 4 22:55:49.159656774 (30h0m25.630867513s) [upload=26406]: Done starting upload Dec 4 22:55:49.159698914 (30h0m25.630909653s) [upload=26406]: Uploading segments... Dec 4 22:55:49.159881322 (30h0m25.631092061s) [upload=26406]: Got next segment. Inline: false Dec 4 22:55:49.159921538 (30h0m25.631132267s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 22:55:49.159927079 (30h0m25.631137808s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 22:55:49.407632244 (30h0m25.878842983s) [seg_pos={0 0}, upload=26406]: Begin upload segment... Dec 4 22:55:49.407685676 (30h0m25.878896415s) [seg_pos={0 0}, upload=26406]: Done begin upload segment. Dec 4 22:55:49.407698249 (30h0m25.878908988s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.407776349 (30h0m25.878987078s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.408596580 (30h0m25.879807319s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409349504 (30h0m25.880560243s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409963062 (30h0m25.881173801s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410210894 (30h0m25.881421623s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410541033 (30h0m25.881751772s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411049931 (30h0m25.88226068s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411451476 (30h0m25.882662215s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411905290 (30h0m25.883116039s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412457602 (30h0m25.883668331s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412740861 (30h0m25.8839516s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413363186 (30h0m25.884573926s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413762196 (30h0m25.884972935s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414199659 (30h0m25.885410398s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414579071 (30h0m25.88578981s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:50.491991030 (30h0m26.963201759s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 22:55:51.092442279 (30h0m27.563653008s) [upload=26406]: Next returned nil segment Dec 4 22:55:51.092455273 (30h0m27.563666012s) [upload=26406]: Waiting for error group managing segments... Dec 4 22:55:51.416315726 (30h0m27.887526465s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.416474488 (30h0m27.887685217s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.418547936 (30h0m27.889758665s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.431075520 (30h0m27.902286249s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.438237447 (30h0m27.909448186s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.444558713 (30h0m27.915769452s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.454863965 (30h0m27.926074704s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.500395572 (30h0m27.971606301s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.002667307 (30h0m28.473878056s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.056889902 (30h0m28.528100641s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.357024719 (30h0m28.828235448s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.423527005 (30h0m28.894737734s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759657335 (30h0m54.230868064s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759677794 (30h0m54.230888523s) [seg_pos={0 0}, upload=26406]: Segment reached optimal threshold of 13 pieces. Dec 4 22:56:17.759902682 (30h0m54.231113661s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection Dec 4 22:56:17.759916668 (30h0m54.231127407s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760005618 (30h0m54.231216347s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection Dec 4 22:56:17.760014334 (30h0m54.231225063s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760032339 (30h0m54.231243068s) [seg_pos={0 0}, upload=26406]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 22:56:17.760113693 (30h0m54.231324423s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 22:56:17.760119615 (30h0m54.231330344s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 22:56:18.057470049 (30h0m54.528680788s) [upload=26406]: Done uploading segments... Dec 4 22:56:18.057547447 (30h0m54.528758187s) [upload=26406]: Upload finished. err: <nil> Duration: 5.713854405s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.690749892s trace: https://jaeger.dev.storj.io/trace/53db3ad570a94624 FAIL (2025-12-04 23:36:12 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 4 22:55:49.159528058 (30h0m25.630738787s) [upload=26406]: Starting upload Dec 4 22:55:49.159656774 (30h0m25.630867513s) [upload=26406]: Done starting upload Dec 4 22:55:49.159698914 (30h0m25.630909653s) [upload=26406]: Uploading segments... Dec 4 22:55:49.159881322 (30h0m25.631092061s) [upload=26406]: Got next segment. Inline: false Dec 4 22:55:49.159921538 (30h0m25.631132267s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 22:55:49.159927079 (30h0m25.631137808s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 22:55:49.407632244 (30h0m25.878842983s) [seg_pos={0 0}, upload=26406]: Begin upload segment... Dec 4 22:55:49.407685676 (30h0m25.878896415s) [seg_pos={0 0}, upload=26406]: Done begin upload segment. Dec 4 22:55:49.407698249 (30h0m25.878908988s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.407776349 (30h0m25.878987078s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.408596580 (30h0m25.879807319s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409349504 (30h0m25.880560243s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409963062 (30h0m25.881173801s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410210894 (30h0m25.881421623s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410541033 (30h0m25.881751772s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411049931 (30h0m25.88226068s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411451476 (30h0m25.882662215s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411905290 (30h0m25.883116039s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412457602 (30h0m25.883668331s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412740861 (30h0m25.8839516s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413363186 (30h0m25.884573926s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413762196 (30h0m25.884972935s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414199659 (30h0m25.885410398s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414579071 (30h0m25.88578981s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:50.491991030 (30h0m26.963201759s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 22:55:51.092442279 (30h0m27.563653008s) [upload=26406]: Next returned nil segment Dec 4 22:55:51.092455273 (30h0m27.563666012s) [upload=26406]: Waiting for error group managing segments... Dec 4 22:55:51.416315726 (30h0m27.887526465s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.416474488 (30h0m27.887685217s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.418547936 (30h0m27.889758665s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.431075520 (30h0m27.902286249s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.438237447 (30h0m27.909448186s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.444558713 (30h0m27.915769452s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.454863965 (30h0m27.926074704s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.500395572 (30h0m27.971606301s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.002667307 (30h0m28.473878056s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.056889902 (30h0m28.528100641s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.357024719 (30h0m28.828235448s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.423527005 (30h0m28.894737734s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759657335 (30h0m54.230868064s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759677794 (30h0m54.230888523s) [seg_pos={0 0}, upload=26406]: Segment reached optimal threshold of 13 pieces. Dec 4 22:56:17.759902682 (30h0m54.231113661s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection Dec 4 22:56:17.759916668 (30h0m54.231127407s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760005618 (30h0m54.231216347s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection Dec 4 22:56:17.760014334 (30h0m54.231225063s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760032339 (30h0m54.231243068s) [seg_pos={0 0}, upload=26406]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 22:56:17.760113693 (30h0m54.231324423s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 22:56:17.760119615 (30h0m54.231330344s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 22:56:18.057470049 (30h0m54.528680788s) [upload=26406]: Done uploading segments... Dec 4 22:56:18.057547447 (30h0m54.528758187s) [upload=26406]: Upload finished. err: <nil> Duration: 5.690779349s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.762029524s trace: https://jaeger.dev.storj.io/trace/53db3ad570a94624 FAIL (2025-12-04 23:35:32 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 4 22:55:49.159528058 (30h0m25.630738787s) [upload=26406]: Starting upload Dec 4 22:55:49.159656774 (30h0m25.630867513s) [upload=26406]: Done starting upload Dec 4 22:55:49.159698914 (30h0m25.630909653s) [upload=26406]: Uploading segments... Dec 4 22:55:49.159881322 (30h0m25.631092061s) [upload=26406]: Got next segment. Inline: false Dec 4 22:55:49.159921538 (30h0m25.631132267s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 22:55:49.159927079 (30h0m25.631137808s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 22:55:49.407632244 (30h0m25.878842983s) [seg_pos={0 0}, upload=26406]: Begin upload segment... Dec 4 22:55:49.407685676 (30h0m25.878896415s) [seg_pos={0 0}, upload=26406]: Done begin upload segment. Dec 4 22:55:49.407698249 (30h0m25.878908988s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.407776349 (30h0m25.878987078s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.408596580 (30h0m25.879807319s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409349504 (30h0m25.880560243s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409963062 (30h0m25.881173801s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410210894 (30h0m25.881421623s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410541033 (30h0m25.881751772s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411049931 (30h0m25.88226068s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411451476 (30h0m25.882662215s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411905290 (30h0m25.883116039s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412457602 (30h0m25.883668331s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412740861 (30h0m25.8839516s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413363186 (30h0m25.884573926s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413762196 (30h0m25.884972935s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414199659 (30h0m25.885410398s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414579071 (30h0m25.88578981s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:50.491991030 (30h0m26.963201759s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 22:55:51.092442279 (30h0m27.563653008s) [upload=26406]: Next returned nil segment Dec 4 22:55:51.092455273 (30h0m27.563666012s) [upload=26406]: Waiting for error group managing segments... Dec 4 22:55:51.416315726 (30h0m27.887526465s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.416474488 (30h0m27.887685217s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.418547936 (30h0m27.889758665s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.431075520 (30h0m27.902286249s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.438237447 (30h0m27.909448186s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.444558713 (30h0m27.915769452s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.454863965 (30h0m27.926074704s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.500395572 (30h0m27.971606301s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.002667307 (30h0m28.473878056s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.056889902 (30h0m28.528100641s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.357024719 (30h0m28.828235448s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.423527005 (30h0m28.894737734s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759657335 (30h0m54.230868064s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759677794 (30h0m54.230888523s) [seg_pos={0 0}, upload=26406]: Segment reached optimal threshold of 13 pieces. Dec 4 22:56:17.759902682 (30h0m54.231113661s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection Dec 4 22:56:17.759916668 (30h0m54.231127407s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760005618 (30h0m54.231216347s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection Dec 4 22:56:17.760014334 (30h0m54.231225063s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760032339 (30h0m54.231243068s) [seg_pos={0 0}, upload=26406]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 22:56:17.760113693 (30h0m54.231324423s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 22:56:17.760119615 (30h0m54.231330344s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 22:56:18.057470049 (30h0m54.528680788s) [upload=26406]: Done uploading segments... Dec 4 22:56:18.057547447 (30h0m54.528758187s) [upload=26406]: Upload finished. err: <nil> Duration: 5.762061394s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.765783899s trace: https://jaeger.dev.storj.io/trace/53db3ad570a94624 FAIL (2025-12-04 23:34:18 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 4 22:55:49.159528058 (30h0m25.630738787s) [upload=26406]: Starting upload Dec 4 22:55:49.159656774 (30h0m25.630867513s) [upload=26406]: Done starting upload Dec 4 22:55:49.159698914 (30h0m25.630909653s) [upload=26406]: Uploading segments... Dec 4 22:55:49.159881322 (30h0m25.631092061s) [upload=26406]: Got next segment. Inline: false Dec 4 22:55:49.159921538 (30h0m25.631132267s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 22:55:49.159927079 (30h0m25.631137808s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 22:55:49.407632244 (30h0m25.878842983s) [seg_pos={0 0}, upload=26406]: Begin upload segment... Dec 4 22:55:49.407685676 (30h0m25.878896415s) [seg_pos={0 0}, upload=26406]: Done begin upload segment. Dec 4 22:55:49.407698249 (30h0m25.878908988s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.407776349 (30h0m25.878987078s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.408596580 (30h0m25.879807319s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409349504 (30h0m25.880560243s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409963062 (30h0m25.881173801s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410210894 (30h0m25.881421623s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410541033 (30h0m25.881751772s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411049931 (30h0m25.88226068s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411451476 (30h0m25.882662215s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411905290 (30h0m25.883116039s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412457602 (30h0m25.883668331s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412740861 (30h0m25.8839516s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413363186 (30h0m25.884573926s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413762196 (30h0m25.884972935s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414199659 (30h0m25.885410398s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414579071 (30h0m25.88578981s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:50.491991030 (30h0m26.963201759s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 22:55:51.092442279 (30h0m27.563653008s) [upload=26406]: Next returned nil segment Dec 4 22:55:51.092455273 (30h0m27.563666012s) [upload=26406]: Waiting for error group managing segments... Dec 4 22:55:51.416315726 (30h0m27.887526465s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.416474488 (30h0m27.887685217s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.418547936 (30h0m27.889758665s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.431075520 (30h0m27.902286249s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.438237447 (30h0m27.909448186s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.444558713 (30h0m27.915769452s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.454863965 (30h0m27.926074704s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.500395572 (30h0m27.971606301s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.002667307 (30h0m28.473878056s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.056889902 (30h0m28.528100641s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.357024719 (30h0m28.828235448s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.423527005 (30h0m28.894737734s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759657335 (30h0m54.230868064s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759677794 (30h0m54.230888523s) [seg_pos={0 0}, upload=26406]: Segment reached optimal threshold of 13 pieces. Dec 4 22:56:17.759902682 (30h0m54.231113661s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection Dec 4 22:56:17.759916668 (30h0m54.231127407s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760005618 (30h0m54.231216347s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection Dec 4 22:56:17.760014334 (30h0m54.231225063s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760032339 (30h0m54.231243068s) [seg_pos={0 0}, upload=26406]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 22:56:17.760113693 (30h0m54.231324423s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 22:56:17.760119615 (30h0m54.231330344s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 22:56:18.057470049 (30h0m54.528680788s) [upload=26406]: Done uploading segments... Dec 4 22:56:18.057547447 (30h0m54.528758187s) [upload=26406]: Upload finished. err: <nil> Duration: 5.765800923s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.718168643s trace: https://jaeger.dev.storj.io/trace/53db3ad570a94624 FAIL (2025-12-04 23:33:27 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 4 22:55:49.159528058 (30h0m25.630738787s) [upload=26406]: Starting upload Dec 4 22:55:49.159656774 (30h0m25.630867513s) [upload=26406]: Done starting upload Dec 4 22:55:49.159698914 (30h0m25.630909653s) [upload=26406]: Uploading segments... Dec 4 22:55:49.159881322 (30h0m25.631092061s) [upload=26406]: Got next segment. Inline: false Dec 4 22:55:49.159921538 (30h0m25.631132267s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 22:55:49.159927079 (30h0m25.631137808s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 22:55:49.407632244 (30h0m25.878842983s) [seg_pos={0 0}, upload=26406]: Begin upload segment... Dec 4 22:55:49.407685676 (30h0m25.878896415s) [seg_pos={0 0}, upload=26406]: Done begin upload segment. Dec 4 22:55:49.407698249 (30h0m25.878908988s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.407776349 (30h0m25.878987078s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.408596580 (30h0m25.879807319s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409349504 (30h0m25.880560243s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409963062 (30h0m25.881173801s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410210894 (30h0m25.881421623s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410541033 (30h0m25.881751772s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411049931 (30h0m25.88226068s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411451476 (30h0m25.882662215s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411905290 (30h0m25.883116039s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412457602 (30h0m25.883668331s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412740861 (30h0m25.8839516s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413363186 (30h0m25.884573926s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413762196 (30h0m25.884972935s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414199659 (30h0m25.885410398s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414579071 (30h0m25.88578981s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:50.491991030 (30h0m26.963201759s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 22:55:51.092442279 (30h0m27.563653008s) [upload=26406]: Next returned nil segment Dec 4 22:55:51.092455273 (30h0m27.563666012s) [upload=26406]: Waiting for error group managing segments... Dec 4 22:55:51.416315726 (30h0m27.887526465s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.416474488 (30h0m27.887685217s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.418547936 (30h0m27.889758665s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.431075520 (30h0m27.902286249s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.438237447 (30h0m27.909448186s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.444558713 (30h0m27.915769452s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.454863965 (30h0m27.926074704s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.500395572 (30h0m27.971606301s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.002667307 (30h0m28.473878056s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.056889902 (30h0m28.528100641s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.357024719 (30h0m28.828235448s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.423527005 (30h0m28.894737734s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759657335 (30h0m54.230868064s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759677794 (30h0m54.230888523s) [seg_pos={0 0}, upload=26406]: Segment reached optimal threshold of 13 pieces. Dec 4 22:56:17.759902682 (30h0m54.231113661s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection Dec 4 22:56:17.759916668 (30h0m54.231127407s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760005618 (30h0m54.231216347s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection Dec 4 22:56:17.760014334 (30h0m54.231225063s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760032339 (30h0m54.231243068s) [seg_pos={0 0}, upload=26406]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 22:56:17.760113693 (30h0m54.231324423s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 22:56:17.760119615 (30h0m54.231330344s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 22:56:18.057470049 (30h0m54.528680788s) [upload=26406]: Done uploading segments... Dec 4 22:56:18.057547447 (30h0m54.528758187s) [upload=26406]: Upload finished. err: <nil> Duration: 5.718196867s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.757903875s trace: https://jaeger.dev.storj.io/trace/53db3ad570a94624 FAIL (2025-12-04 23:32:24 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 4 22:55:49.159528058 (30h0m25.630738787s) [upload=26406]: Starting upload Dec 4 22:55:49.159656774 (30h0m25.630867513s) [upload=26406]: Done starting upload Dec 4 22:55:49.159698914 (30h0m25.630909653s) [upload=26406]: Uploading segments... Dec 4 22:55:49.159881322 (30h0m25.631092061s) [upload=26406]: Got next segment. Inline: false Dec 4 22:55:49.159921538 (30h0m25.631132267s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 22:55:49.159927079 (30h0m25.631137808s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 22:55:49.407632244 (30h0m25.878842983s) [seg_pos={0 0}, upload=26406]: Begin upload segment... Dec 4 22:55:49.407685676 (30h0m25.878896415s) [seg_pos={0 0}, upload=26406]: Done begin upload segment. Dec 4 22:55:49.407698249 (30h0m25.878908988s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.407776349 (30h0m25.878987078s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.408596580 (30h0m25.879807319s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409349504 (30h0m25.880560243s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409963062 (30h0m25.881173801s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410210894 (30h0m25.881421623s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410541033 (30h0m25.881751772s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411049931 (30h0m25.88226068s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411451476 (30h0m25.882662215s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411905290 (30h0m25.883116039s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412457602 (30h0m25.883668331s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412740861 (30h0m25.8839516s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413363186 (30h0m25.884573926s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413762196 (30h0m25.884972935s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414199659 (30h0m25.885410398s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414579071 (30h0m25.88578981s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:50.491991030 (30h0m26.963201759s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 22:55:51.092442279 (30h0m27.563653008s) [upload=26406]: Next returned nil segment Dec 4 22:55:51.092455273 (30h0m27.563666012s) [upload=26406]: Waiting for error group managing segments... Dec 4 22:55:51.416315726 (30h0m27.887526465s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.416474488 (30h0m27.887685217s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.418547936 (30h0m27.889758665s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.431075520 (30h0m27.902286249s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.438237447 (30h0m27.909448186s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.444558713 (30h0m27.915769452s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.454863965 (30h0m27.926074704s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.500395572 (30h0m27.971606301s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.002667307 (30h0m28.473878056s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.056889902 (30h0m28.528100641s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.357024719 (30h0m28.828235448s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.423527005 (30h0m28.894737734s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759657335 (30h0m54.230868064s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759677794 (30h0m54.230888523s) [seg_pos={0 0}, upload=26406]: Segment reached optimal threshold of 13 pieces. Dec 4 22:56:17.759902682 (30h0m54.231113661s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection Dec 4 22:56:17.759916668 (30h0m54.231127407s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760005618 (30h0m54.231216347s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection Dec 4 22:56:17.760014334 (30h0m54.231225063s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760032339 (30h0m54.231243068s) [seg_pos={0 0}, upload=26406]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 22:56:17.760113693 (30h0m54.231324423s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 22:56:17.760119615 (30h0m54.231330344s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 22:56:18.057470049 (30h0m54.528680788s) [upload=26406]: Done uploading segments... Dec 4 22:56:18.057547447 (30h0m54.528758187s) [upload=26406]: Upload finished. err: <nil> Duration: 5.757938543s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.753435567s trace: https://jaeger.dev.storj.io/trace/53db3ad570a94624 FAIL (2025-12-04 23:31:50 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 4 22:55:49.159528058 (30h0m25.630738787s) [upload=26406]: Starting upload Dec 4 22:55:49.159656774 (30h0m25.630867513s) [upload=26406]: Done starting upload Dec 4 22:55:49.159698914 (30h0m25.630909653s) [upload=26406]: Uploading segments... Dec 4 22:55:49.159881322 (30h0m25.631092061s) [upload=26406]: Got next segment. Inline: false Dec 4 22:55:49.159921538 (30h0m25.631132267s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 22:55:49.159927079 (30h0m25.631137808s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 22:55:49.407632244 (30h0m25.878842983s) [seg_pos={0 0}, upload=26406]: Begin upload segment... Dec 4 22:55:49.407685676 (30h0m25.878896415s) [seg_pos={0 0}, upload=26406]: Done begin upload segment. Dec 4 22:55:49.407698249 (30h0m25.878908988s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.407776349 (30h0m25.878987078s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.408596580 (30h0m25.879807319s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409349504 (30h0m25.880560243s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409963062 (30h0m25.881173801s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410210894 (30h0m25.881421623s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410541033 (30h0m25.881751772s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411049931 (30h0m25.88226068s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411451476 (30h0m25.882662215s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411905290 (30h0m25.883116039s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412457602 (30h0m25.883668331s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412740861 (30h0m25.8839516s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413363186 (30h0m25.884573926s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413762196 (30h0m25.884972935s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414199659 (30h0m25.885410398s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414579071 (30h0m25.88578981s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:50.491991030 (30h0m26.963201759s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 22:55:51.092442279 (30h0m27.563653008s) [upload=26406]: Next returned nil segment Dec 4 22:55:51.092455273 (30h0m27.563666012s) [upload=26406]: Waiting for error group managing segments... Dec 4 22:55:51.416315726 (30h0m27.887526465s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.416474488 (30h0m27.887685217s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.418547936 (30h0m27.889758665s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.431075520 (30h0m27.902286249s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.438237447 (30h0m27.909448186s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.444558713 (30h0m27.915769452s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.454863965 (30h0m27.926074704s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.500395572 (30h0m27.971606301s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.002667307 (30h0m28.473878056s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.056889902 (30h0m28.528100641s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.357024719 (30h0m28.828235448s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.423527005 (30h0m28.894737734s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759657335 (30h0m54.230868064s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759677794 (30h0m54.230888523s) [seg_pos={0 0}, upload=26406]: Segment reached optimal threshold of 13 pieces. Dec 4 22:56:17.759902682 (30h0m54.231113661s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection Dec 4 22:56:17.759916668 (30h0m54.231127407s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760005618 (30h0m54.231216347s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection Dec 4 22:56:17.760014334 (30h0m54.231225063s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760032339 (30h0m54.231243068s) [seg_pos={0 0}, upload=26406]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 22:56:17.760113693 (30h0m54.231324423s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 22:56:17.760119615 (30h0m54.231330344s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 22:56:18.057470049 (30h0m54.528680788s) [upload=26406]: Done uploading segments... Dec 4 22:56:18.057547447 (30h0m54.528758187s) [upload=26406]: Upload finished. err: <nil> Duration: 5.753462879s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.795465883s trace: https://jaeger.dev.storj.io/trace/53db3ad570a94624 FAIL (2025-12-04 23:31:22 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 4 22:55:49.159528058 (30h0m25.630738787s) [upload=26406]: Starting upload Dec 4 22:55:49.159656774 (30h0m25.630867513s) [upload=26406]: Done starting upload Dec 4 22:55:49.159698914 (30h0m25.630909653s) [upload=26406]: Uploading segments... Dec 4 22:55:49.159881322 (30h0m25.631092061s) [upload=26406]: Got next segment. Inline: false Dec 4 22:55:49.159921538 (30h0m25.631132267s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 22:55:49.159927079 (30h0m25.631137808s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 22:55:49.407632244 (30h0m25.878842983s) [seg_pos={0 0}, upload=26406]: Begin upload segment... Dec 4 22:55:49.407685676 (30h0m25.878896415s) [seg_pos={0 0}, upload=26406]: Done begin upload segment. Dec 4 22:55:49.407698249 (30h0m25.878908988s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.407776349 (30h0m25.878987078s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.408596580 (30h0m25.879807319s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409349504 (30h0m25.880560243s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409963062 (30h0m25.881173801s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410210894 (30h0m25.881421623s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410541033 (30h0m25.881751772s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411049931 (30h0m25.88226068s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411451476 (30h0m25.882662215s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411905290 (30h0m25.883116039s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412457602 (30h0m25.883668331s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412740861 (30h0m25.8839516s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413363186 (30h0m25.884573926s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413762196 (30h0m25.884972935s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414199659 (30h0m25.885410398s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414579071 (30h0m25.88578981s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:50.491991030 (30h0m26.963201759s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 22:55:51.092442279 (30h0m27.563653008s) [upload=26406]: Next returned nil segment Dec 4 22:55:51.092455273 (30h0m27.563666012s) [upload=26406]: Waiting for error group managing segments... Dec 4 22:55:51.416315726 (30h0m27.887526465s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.416474488 (30h0m27.887685217s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.418547936 (30h0m27.889758665s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.431075520 (30h0m27.902286249s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.438237447 (30h0m27.909448186s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.444558713 (30h0m27.915769452s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.454863965 (30h0m27.926074704s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.500395572 (30h0m27.971606301s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.002667307 (30h0m28.473878056s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.056889902 (30h0m28.528100641s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.357024719 (30h0m28.828235448s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.423527005 (30h0m28.894737734s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759657335 (30h0m54.230868064s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759677794 (30h0m54.230888523s) [seg_pos={0 0}, upload=26406]: Segment reached optimal threshold of 13 pieces. Dec 4 22:56:17.759902682 (30h0m54.231113661s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection Dec 4 22:56:17.759916668 (30h0m54.231127407s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760005618 (30h0m54.231216347s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection Dec 4 22:56:17.760014334 (30h0m54.231225063s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760032339 (30h0m54.231243068s) [seg_pos={0 0}, upload=26406]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 22:56:17.760113693 (30h0m54.231324423s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 22:56:17.760119615 (30h0m54.231330344s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 22:56:18.057470049 (30h0m54.528680788s) [upload=26406]: Done uploading segments... Dec 4 22:56:18.057547447 (30h0m54.528758187s) [upload=26406]: Upload finished. err: <nil> Duration: 5.795499978s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.751599501s trace: https://jaeger.dev.storj.io/trace/53db3ad570a94624 FAIL (2025-12-04 23:30:56 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 4 22:55:49.159528058 (30h0m25.630738787s) [upload=26406]: Starting upload Dec 4 22:55:49.159656774 (30h0m25.630867513s) [upload=26406]: Done starting upload Dec 4 22:55:49.159698914 (30h0m25.630909653s) [upload=26406]: Uploading segments... Dec 4 22:55:49.159881322 (30h0m25.631092061s) [upload=26406]: Got next segment. Inline: false Dec 4 22:55:49.159921538 (30h0m25.631132267s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 22:55:49.159927079 (30h0m25.631137808s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 22:55:49.407632244 (30h0m25.878842983s) [seg_pos={0 0}, upload=26406]: Begin upload segment... Dec 4 22:55:49.407685676 (30h0m25.878896415s) [seg_pos={0 0}, upload=26406]: Done begin upload segment. Dec 4 22:55:49.407698249 (30h0m25.878908988s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.407776349 (30h0m25.878987078s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.408596580 (30h0m25.879807319s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409349504 (30h0m25.880560243s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409963062 (30h0m25.881173801s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410210894 (30h0m25.881421623s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410541033 (30h0m25.881751772s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411049931 (30h0m25.88226068s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411451476 (30h0m25.882662215s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411905290 (30h0m25.883116039s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412457602 (30h0m25.883668331s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412740861 (30h0m25.8839516s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413363186 (30h0m25.884573926s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413762196 (30h0m25.884972935s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414199659 (30h0m25.885410398s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414579071 (30h0m25.88578981s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:50.491991030 (30h0m26.963201759s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 22:55:51.092442279 (30h0m27.563653008s) [upload=26406]: Next returned nil segment Dec 4 22:55:51.092455273 (30h0m27.563666012s) [upload=26406]: Waiting for error group managing segments... Dec 4 22:55:51.416315726 (30h0m27.887526465s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.416474488 (30h0m27.887685217s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.418547936 (30h0m27.889758665s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.431075520 (30h0m27.902286249s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.438237447 (30h0m27.909448186s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.444558713 (30h0m27.915769452s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.454863965 (30h0m27.926074704s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.500395572 (30h0m27.971606301s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.002667307 (30h0m28.473878056s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.056889902 (30h0m28.528100641s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.357024719 (30h0m28.828235448s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.423527005 (30h0m28.894737734s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759657335 (30h0m54.230868064s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759677794 (30h0m54.230888523s) [seg_pos={0 0}, upload=26406]: Segment reached optimal threshold of 13 pieces. Dec 4 22:56:17.759902682 (30h0m54.231113661s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection Dec 4 22:56:17.759916668 (30h0m54.231127407s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760005618 (30h0m54.231216347s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection Dec 4 22:56:17.760014334 (30h0m54.231225063s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760032339 (30h0m54.231243068s) [seg_pos={0 0}, upload=26406]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 22:56:17.760113693 (30h0m54.231324423s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 22:56:17.760119615 (30h0m54.231330344s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 22:56:18.057470049 (30h0m54.528680788s) [upload=26406]: Done uploading segments... Dec 4 22:56:18.057547447 (30h0m54.528758187s) [upload=26406]: Upload finished. err: <nil> Duration: 5.751627624s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.70773668s trace: https://jaeger.dev.storj.io/trace/53db3ad570a94624 FAIL (2025-12-04 23:30:21 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 4 22:55:49.159528058 (30h0m25.630738787s) [upload=26406]: Starting upload Dec 4 22:55:49.159656774 (30h0m25.630867513s) [upload=26406]: Done starting upload Dec 4 22:55:49.159698914 (30h0m25.630909653s) [upload=26406]: Uploading segments... Dec 4 22:55:49.159881322 (30h0m25.631092061s) [upload=26406]: Got next segment. Inline: false Dec 4 22:55:49.159921538 (30h0m25.631132267s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 22:55:49.159927079 (30h0m25.631137808s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 22:55:49.407632244 (30h0m25.878842983s) [seg_pos={0 0}, upload=26406]: Begin upload segment... Dec 4 22:55:49.407685676 (30h0m25.878896415s) [seg_pos={0 0}, upload=26406]: Done begin upload segment. Dec 4 22:55:49.407698249 (30h0m25.878908988s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.407776349 (30h0m25.878987078s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.408596580 (30h0m25.879807319s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409349504 (30h0m25.880560243s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409963062 (30h0m25.881173801s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410210894 (30h0m25.881421623s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410541033 (30h0m25.881751772s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411049931 (30h0m25.88226068s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411451476 (30h0m25.882662215s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411905290 (30h0m25.883116039s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412457602 (30h0m25.883668331s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412740861 (30h0m25.8839516s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413363186 (30h0m25.884573926s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413762196 (30h0m25.884972935s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414199659 (30h0m25.885410398s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414579071 (30h0m25.88578981s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:50.491991030 (30h0m26.963201759s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 22:55:51.092442279 (30h0m27.563653008s) [upload=26406]: Next returned nil segment Dec 4 22:55:51.092455273 (30h0m27.563666012s) [upload=26406]: Waiting for error group managing segments... Dec 4 22:55:51.416315726 (30h0m27.887526465s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.416474488 (30h0m27.887685217s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.418547936 (30h0m27.889758665s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.431075520 (30h0m27.902286249s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.438237447 (30h0m27.909448186s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.444558713 (30h0m27.915769452s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.454863965 (30h0m27.926074704s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.500395572 (30h0m27.971606301s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.002667307 (30h0m28.473878056s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.056889902 (30h0m28.528100641s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.357024719 (30h0m28.828235448s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.423527005 (30h0m28.894737734s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759657335 (30h0m54.230868064s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759677794 (30h0m54.230888523s) [seg_pos={0 0}, upload=26406]: Segment reached optimal threshold of 13 pieces. Dec 4 22:56:17.759902682 (30h0m54.231113661s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection Dec 4 22:56:17.759916668 (30h0m54.231127407s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760005618 (30h0m54.231216347s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection Dec 4 22:56:17.760014334 (30h0m54.231225063s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760032339 (30h0m54.231243068s) [seg_pos={0 0}, upload=26406]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 22:56:17.760113693 (30h0m54.231324423s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 22:56:17.760119615 (30h0m54.231330344s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 22:56:18.057470049 (30h0m54.528680788s) [upload=26406]: Done uploading segments... Dec 4 22:56:18.057547447 (30h0m54.528758187s) [upload=26406]: Upload finished. err: <nil> Duration: 5.707769773s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.812754529s trace: https://jaeger.dev.storj.io/trace/53db3ad570a94624 FAIL (2025-12-04 23:29:33 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 4 22:55:49.159528058 (30h0m25.630738787s) [upload=26406]: Starting upload Dec 4 22:55:49.159656774 (30h0m25.630867513s) [upload=26406]: Done starting upload Dec 4 22:55:49.159698914 (30h0m25.630909653s) [upload=26406]: Uploading segments... Dec 4 22:55:49.159881322 (30h0m25.631092061s) [upload=26406]: Got next segment. Inline: false Dec 4 22:55:49.159921538 (30h0m25.631132267s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 22:55:49.159927079 (30h0m25.631137808s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 22:55:49.407632244 (30h0m25.878842983s) [seg_pos={0 0}, upload=26406]: Begin upload segment... Dec 4 22:55:49.407685676 (30h0m25.878896415s) [seg_pos={0 0}, upload=26406]: Done begin upload segment. Dec 4 22:55:49.407698249 (30h0m25.878908988s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.407776349 (30h0m25.878987078s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.408596580 (30h0m25.879807319s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409349504 (30h0m25.880560243s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409963062 (30h0m25.881173801s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410210894 (30h0m25.881421623s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410541033 (30h0m25.881751772s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411049931 (30h0m25.88226068s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411451476 (30h0m25.882662215s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411905290 (30h0m25.883116039s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412457602 (30h0m25.883668331s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412740861 (30h0m25.8839516s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413363186 (30h0m25.884573926s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413762196 (30h0m25.884972935s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414199659 (30h0m25.885410398s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414579071 (30h0m25.88578981s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:50.491991030 (30h0m26.963201759s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 22:55:51.092442279 (30h0m27.563653008s) [upload=26406]: Next returned nil segment Dec 4 22:55:51.092455273 (30h0m27.563666012s) [upload=26406]: Waiting for error group managing segments... Dec 4 22:55:51.416315726 (30h0m27.887526465s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.416474488 (30h0m27.887685217s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.418547936 (30h0m27.889758665s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.431075520 (30h0m27.902286249s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.438237447 (30h0m27.909448186s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.444558713 (30h0m27.915769452s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.454863965 (30h0m27.926074704s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.500395572 (30h0m27.971606301s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.002667307 (30h0m28.473878056s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.056889902 (30h0m28.528100641s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.357024719 (30h0m28.828235448s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.423527005 (30h0m28.894737734s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759657335 (30h0m54.230868064s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759677794 (30h0m54.230888523s) [seg_pos={0 0}, upload=26406]: Segment reached optimal threshold of 13 pieces. Dec 4 22:56:17.759902682 (30h0m54.231113661s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection Dec 4 22:56:17.759916668 (30h0m54.231127407s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760005618 (30h0m54.231216347s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection Dec 4 22:56:17.760014334 (30h0m54.231225063s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760032339 (30h0m54.231243068s) [seg_pos={0 0}, upload=26406]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 22:56:17.760113693 (30h0m54.231324423s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 22:56:17.760119615 (30h0m54.231330344s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 22:56:18.057470049 (30h0m54.528680788s) [upload=26406]: Done uploading segments... Dec 4 22:56:18.057547447 (30h0m54.528758187s) [upload=26406]: Upload finished. err: <nil> Duration: 5.812784037s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.762113923s trace: https://jaeger.dev.storj.io/trace/53db3ad570a94624 FAIL (2025-12-04 23:28:35 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 4 22:55:49.159528058 (30h0m25.630738787s) [upload=26406]: Starting upload Dec 4 22:55:49.159656774 (30h0m25.630867513s) [upload=26406]: Done starting upload Dec 4 22:55:49.159698914 (30h0m25.630909653s) [upload=26406]: Uploading segments... Dec 4 22:55:49.159881322 (30h0m25.631092061s) [upload=26406]: Got next segment. Inline: false Dec 4 22:55:49.159921538 (30h0m25.631132267s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 22:55:49.159927079 (30h0m25.631137808s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 22:55:49.407632244 (30h0m25.878842983s) [seg_pos={0 0}, upload=26406]: Begin upload segment... Dec 4 22:55:49.407685676 (30h0m25.878896415s) [seg_pos={0 0}, upload=26406]: Done begin upload segment. Dec 4 22:55:49.407698249 (30h0m25.878908988s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.407776349 (30h0m25.878987078s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.408596580 (30h0m25.879807319s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409349504 (30h0m25.880560243s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409963062 (30h0m25.881173801s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410210894 (30h0m25.881421623s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410541033 (30h0m25.881751772s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411049931 (30h0m25.88226068s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411451476 (30h0m25.882662215s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411905290 (30h0m25.883116039s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412457602 (30h0m25.883668331s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412740861 (30h0m25.8839516s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413363186 (30h0m25.884573926s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413762196 (30h0m25.884972935s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414199659 (30h0m25.885410398s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414579071 (30h0m25.88578981s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:50.491991030 (30h0m26.963201759s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 22:55:51.092442279 (30h0m27.563653008s) [upload=26406]: Next returned nil segment Dec 4 22:55:51.092455273 (30h0m27.563666012s) [upload=26406]: Waiting for error group managing segments... Dec 4 22:55:51.416315726 (30h0m27.887526465s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.416474488 (30h0m27.887685217s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.418547936 (30h0m27.889758665s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.431075520 (30h0m27.902286249s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.438237447 (30h0m27.909448186s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.444558713 (30h0m27.915769452s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.454863965 (30h0m27.926074704s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.500395572 (30h0m27.971606301s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.002667307 (30h0m28.473878056s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.056889902 (30h0m28.528100641s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.357024719 (30h0m28.828235448s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.423527005 (30h0m28.894737734s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759657335 (30h0m54.230868064s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759677794 (30h0m54.230888523s) [seg_pos={0 0}, upload=26406]: Segment reached optimal threshold of 13 pieces. Dec 4 22:56:17.759902682 (30h0m54.231113661s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection Dec 4 22:56:17.759916668 (30h0m54.231127407s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760005618 (30h0m54.231216347s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection Dec 4 22:56:17.760014334 (30h0m54.231225063s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760032339 (30h0m54.231243068s) [seg_pos={0 0}, upload=26406]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 22:56:17.760113693 (30h0m54.231324423s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 22:56:17.760119615 (30h0m54.231330344s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 22:56:18.057470049 (30h0m54.528680788s) [upload=26406]: Done uploading segments... Dec 4 22:56:18.057547447 (30h0m54.528758187s) [upload=26406]: Upload finished. err: <nil> Duration: 5.762144112s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.970461748s trace: https://jaeger.dev.storj.io/trace/53db3ad570a94624 FAIL (2025-12-04 23:28:08 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 4 22:55:49.159528058 (30h0m25.630738787s) [upload=26406]: Starting upload Dec 4 22:55:49.159656774 (30h0m25.630867513s) [upload=26406]: Done starting upload Dec 4 22:55:49.159698914 (30h0m25.630909653s) [upload=26406]: Uploading segments... Dec 4 22:55:49.159881322 (30h0m25.631092061s) [upload=26406]: Got next segment. Inline: false Dec 4 22:55:49.159921538 (30h0m25.631132267s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 22:55:49.159927079 (30h0m25.631137808s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 22:55:49.407632244 (30h0m25.878842983s) [seg_pos={0 0}, upload=26406]: Begin upload segment... Dec 4 22:55:49.407685676 (30h0m25.878896415s) [seg_pos={0 0}, upload=26406]: Done begin upload segment. Dec 4 22:55:49.407698249 (30h0m25.878908988s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.407776349 (30h0m25.878987078s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.408596580 (30h0m25.879807319s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409349504 (30h0m25.880560243s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409963062 (30h0m25.881173801s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410210894 (30h0m25.881421623s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410541033 (30h0m25.881751772s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411049931 (30h0m25.88226068s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411451476 (30h0m25.882662215s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411905290 (30h0m25.883116039s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412457602 (30h0m25.883668331s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412740861 (30h0m25.8839516s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413363186 (30h0m25.884573926s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413762196 (30h0m25.884972935s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414199659 (30h0m25.885410398s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414579071 (30h0m25.88578981s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:50.491991030 (30h0m26.963201759s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 22:55:51.092442279 (30h0m27.563653008s) [upload=26406]: Next returned nil segment Dec 4 22:55:51.092455273 (30h0m27.563666012s) [upload=26406]: Waiting for error group managing segments... Dec 4 22:55:51.416315726 (30h0m27.887526465s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.416474488 (30h0m27.887685217s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.418547936 (30h0m27.889758665s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.431075520 (30h0m27.902286249s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.438237447 (30h0m27.909448186s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.444558713 (30h0m27.915769452s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.454863965 (30h0m27.926074704s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.500395572 (30h0m27.971606301s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.002667307 (30h0m28.473878056s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.056889902 (30h0m28.528100641s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.357024719 (30h0m28.828235448s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.423527005 (30h0m28.894737734s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759657335 (30h0m54.230868064s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759677794 (30h0m54.230888523s) [seg_pos={0 0}, upload=26406]: Segment reached optimal threshold of 13 pieces. Dec 4 22:56:17.759902682 (30h0m54.231113661s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection Dec 4 22:56:17.759916668 (30h0m54.231127407s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760005618 (30h0m54.231216347s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection Dec 4 22:56:17.760014334 (30h0m54.231225063s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760032339 (30h0m54.231243068s) [seg_pos={0 0}, upload=26406]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 22:56:17.760113693 (30h0m54.231324423s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 22:56:17.760119615 (30h0m54.231330344s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 22:56:18.057470049 (30h0m54.528680788s) [upload=26406]: Done uploading segments... Dec 4 22:56:18.057547447 (30h0m54.528758187s) [upload=26406]: Upload finished. err: <nil> Duration: 5.970485424s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.820665581s trace: https://jaeger.dev.storj.io/trace/53db3ad570a94624 FAIL (2025-12-04 23:27:48 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 4 22:55:49.159528058 (30h0m25.630738787s) [upload=26406]: Starting upload Dec 4 22:55:49.159656774 (30h0m25.630867513s) [upload=26406]: Done starting upload Dec 4 22:55:49.159698914 (30h0m25.630909653s) [upload=26406]: Uploading segments... Dec 4 22:55:49.159881322 (30h0m25.631092061s) [upload=26406]: Got next segment. Inline: false Dec 4 22:55:49.159921538 (30h0m25.631132267s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 22:55:49.159927079 (30h0m25.631137808s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 22:55:49.407632244 (30h0m25.878842983s) [seg_pos={0 0}, upload=26406]: Begin upload segment... Dec 4 22:55:49.407685676 (30h0m25.878896415s) [seg_pos={0 0}, upload=26406]: Done begin upload segment. Dec 4 22:55:49.407698249 (30h0m25.878908988s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.407776349 (30h0m25.878987078s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.408596580 (30h0m25.879807319s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409349504 (30h0m25.880560243s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409963062 (30h0m25.881173801s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410210894 (30h0m25.881421623s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410541033 (30h0m25.881751772s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411049931 (30h0m25.88226068s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411451476 (30h0m25.882662215s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411905290 (30h0m25.883116039s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412457602 (30h0m25.883668331s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412740861 (30h0m25.8839516s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413363186 (30h0m25.884573926s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413762196 (30h0m25.884972935s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414199659 (30h0m25.885410398s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414579071 (30h0m25.88578981s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:50.491991030 (30h0m26.963201759s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 22:55:51.092442279 (30h0m27.563653008s) [upload=26406]: Next returned nil segment Dec 4 22:55:51.092455273 (30h0m27.563666012s) [upload=26406]: Waiting for error group managing segments... Dec 4 22:55:51.416315726 (30h0m27.887526465s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.416474488 (30h0m27.887685217s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.418547936 (30h0m27.889758665s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.431075520 (30h0m27.902286249s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.438237447 (30h0m27.909448186s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.444558713 (30h0m27.915769452s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.454863965 (30h0m27.926074704s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.500395572 (30h0m27.971606301s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.002667307 (30h0m28.473878056s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.056889902 (30h0m28.528100641s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.357024719 (30h0m28.828235448s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.423527005 (30h0m28.894737734s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759657335 (30h0m54.230868064s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759677794 (30h0m54.230888523s) [seg_pos={0 0}, upload=26406]: Segment reached optimal threshold of 13 pieces. Dec 4 22:56:17.759902682 (30h0m54.231113661s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection Dec 4 22:56:17.759916668 (30h0m54.231127407s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760005618 (30h0m54.231216347s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection Dec 4 22:56:17.760014334 (30h0m54.231225063s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760032339 (30h0m54.231243068s) [seg_pos={0 0}, upload=26406]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 22:56:17.760113693 (30h0m54.231324423s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 22:56:17.760119615 (30h0m54.231330344s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 22:56:18.057470049 (30h0m54.528680788s) [upload=26406]: Done uploading segments... Dec 4 22:56:18.057547447 (30h0m54.528758187s) [upload=26406]: Upload finished. err: <nil> Duration: 5.820687923s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.736976154s trace: https://jaeger.dev.storj.io/trace/53db3ad570a94624 FAIL (2025-12-04 23:27:11 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 4 22:55:49.159528058 (30h0m25.630738787s) [upload=26406]: Starting upload Dec 4 22:55:49.159656774 (30h0m25.630867513s) [upload=26406]: Done starting upload Dec 4 22:55:49.159698914 (30h0m25.630909653s) [upload=26406]: Uploading segments... Dec 4 22:55:49.159881322 (30h0m25.631092061s) [upload=26406]: Got next segment. Inline: false Dec 4 22:55:49.159921538 (30h0m25.631132267s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 22:55:49.159927079 (30h0m25.631137808s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 22:55:49.407632244 (30h0m25.878842983s) [seg_pos={0 0}, upload=26406]: Begin upload segment... Dec 4 22:55:49.407685676 (30h0m25.878896415s) [seg_pos={0 0}, upload=26406]: Done begin upload segment. Dec 4 22:55:49.407698249 (30h0m25.878908988s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.407776349 (30h0m25.878987078s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.408596580 (30h0m25.879807319s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409349504 (30h0m25.880560243s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409963062 (30h0m25.881173801s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410210894 (30h0m25.881421623s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410541033 (30h0m25.881751772s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411049931 (30h0m25.88226068s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411451476 (30h0m25.882662215s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411905290 (30h0m25.883116039s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412457602 (30h0m25.883668331s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412740861 (30h0m25.8839516s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413363186 (30h0m25.884573926s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413762196 (30h0m25.884972935s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414199659 (30h0m25.885410398s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414579071 (30h0m25.88578981s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:50.491991030 (30h0m26.963201759s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 22:55:51.092442279 (30h0m27.563653008s) [upload=26406]: Next returned nil segment Dec 4 22:55:51.092455273 (30h0m27.563666012s) [upload=26406]: Waiting for error group managing segments... Dec 4 22:55:51.416315726 (30h0m27.887526465s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.416474488 (30h0m27.887685217s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.418547936 (30h0m27.889758665s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.431075520 (30h0m27.902286249s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.438237447 (30h0m27.909448186s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.444558713 (30h0m27.915769452s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.454863965 (30h0m27.926074704s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.500395572 (30h0m27.971606301s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.002667307 (30h0m28.473878056s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.056889902 (30h0m28.528100641s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.357024719 (30h0m28.828235448s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.423527005 (30h0m28.894737734s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759657335 (30h0m54.230868064s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759677794 (30h0m54.230888523s) [seg_pos={0 0}, upload=26406]: Segment reached optimal threshold of 13 pieces. Dec 4 22:56:17.759902682 (30h0m54.231113661s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection Dec 4 22:56:17.759916668 (30h0m54.231127407s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760005618 (30h0m54.231216347s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection Dec 4 22:56:17.760014334 (30h0m54.231225063s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760032339 (30h0m54.231243068s) [seg_pos={0 0}, upload=26406]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 22:56:17.760113693 (30h0m54.231324423s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 22:56:17.760119615 (30h0m54.231330344s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 22:56:18.057470049 (30h0m54.528680788s) [upload=26406]: Done uploading segments... Dec 4 22:56:18.057547447 (30h0m54.528758187s) [upload=26406]: Upload finished. err: <nil> Duration: 5.737007184s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.733311065s trace: https://jaeger.dev.storj.io/trace/53db3ad570a94624 FAIL (2025-12-04 23:26:21 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 4 22:55:49.159528058 (30h0m25.630738787s) [upload=26406]: Starting upload Dec 4 22:55:49.159656774 (30h0m25.630867513s) [upload=26406]: Done starting upload Dec 4 22:55:49.159698914 (30h0m25.630909653s) [upload=26406]: Uploading segments... Dec 4 22:55:49.159881322 (30h0m25.631092061s) [upload=26406]: Got next segment. Inline: false Dec 4 22:55:49.159921538 (30h0m25.631132267s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 22:55:49.159927079 (30h0m25.631137808s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 22:55:49.407632244 (30h0m25.878842983s) [seg_pos={0 0}, upload=26406]: Begin upload segment... Dec 4 22:55:49.407685676 (30h0m25.878896415s) [seg_pos={0 0}, upload=26406]: Done begin upload segment. Dec 4 22:55:49.407698249 (30h0m25.878908988s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.407776349 (30h0m25.878987078s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.408596580 (30h0m25.879807319s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409349504 (30h0m25.880560243s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409963062 (30h0m25.881173801s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410210894 (30h0m25.881421623s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410541033 (30h0m25.881751772s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411049931 (30h0m25.88226068s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411451476 (30h0m25.882662215s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411905290 (30h0m25.883116039s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412457602 (30h0m25.883668331s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412740861 (30h0m25.8839516s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413363186 (30h0m25.884573926s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413762196 (30h0m25.884972935s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414199659 (30h0m25.885410398s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414579071 (30h0m25.88578981s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:50.491991030 (30h0m26.963201759s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 22:55:51.092442279 (30h0m27.563653008s) [upload=26406]: Next returned nil segment Dec 4 22:55:51.092455273 (30h0m27.563666012s) [upload=26406]: Waiting for error group managing segments... Dec 4 22:55:51.416315726 (30h0m27.887526465s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.416474488 (30h0m27.887685217s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.418547936 (30h0m27.889758665s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.431075520 (30h0m27.902286249s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.438237447 (30h0m27.909448186s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.444558713 (30h0m27.915769452s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.454863965 (30h0m27.926074704s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.500395572 (30h0m27.971606301s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.002667307 (30h0m28.473878056s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.056889902 (30h0m28.528100641s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.357024719 (30h0m28.828235448s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.423527005 (30h0m28.894737734s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759657335 (30h0m54.230868064s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759677794 (30h0m54.230888523s) [seg_pos={0 0}, upload=26406]: Segment reached optimal threshold of 13 pieces. Dec 4 22:56:17.759902682 (30h0m54.231113661s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection Dec 4 22:56:17.759916668 (30h0m54.231127407s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760005618 (30h0m54.231216347s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection Dec 4 22:56:17.760014334 (30h0m54.231225063s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760032339 (30h0m54.231243068s) [seg_pos={0 0}, upload=26406]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 22:56:17.760113693 (30h0m54.231324423s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 22:56:17.760119615 (30h0m54.231330344s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 22:56:18.057470049 (30h0m54.528680788s) [upload=26406]: Done uploading segments... Dec 4 22:56:18.057547447 (30h0m54.528758187s) [upload=26406]: Upload finished. err: <nil> Duration: 5.733339109s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.647453037s trace: https://jaeger.dev.storj.io/trace/53db3ad570a94624 FAIL (2025-12-04 23:25:29 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 4 22:55:49.159528058 (30h0m25.630738787s) [upload=26406]: Starting upload Dec 4 22:55:49.159656774 (30h0m25.630867513s) [upload=26406]: Done starting upload Dec 4 22:55:49.159698914 (30h0m25.630909653s) [upload=26406]: Uploading segments... Dec 4 22:55:49.159881322 (30h0m25.631092061s) [upload=26406]: Got next segment. Inline: false Dec 4 22:55:49.159921538 (30h0m25.631132267s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 22:55:49.159927079 (30h0m25.631137808s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 22:55:49.407632244 (30h0m25.878842983s) [seg_pos={0 0}, upload=26406]: Begin upload segment... Dec 4 22:55:49.407685676 (30h0m25.878896415s) [seg_pos={0 0}, upload=26406]: Done begin upload segment. Dec 4 22:55:49.407698249 (30h0m25.878908988s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.407776349 (30h0m25.878987078s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.408596580 (30h0m25.879807319s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409349504 (30h0m25.880560243s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409963062 (30h0m25.881173801s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410210894 (30h0m25.881421623s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410541033 (30h0m25.881751772s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411049931 (30h0m25.88226068s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411451476 (30h0m25.882662215s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411905290 (30h0m25.883116039s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412457602 (30h0m25.883668331s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412740861 (30h0m25.8839516s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413363186 (30h0m25.884573926s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413762196 (30h0m25.884972935s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414199659 (30h0m25.885410398s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414579071 (30h0m25.88578981s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:50.491991030 (30h0m26.963201759s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 22:55:51.092442279 (30h0m27.563653008s) [upload=26406]: Next returned nil segment Dec 4 22:55:51.092455273 (30h0m27.563666012s) [upload=26406]: Waiting for error group managing segments... Dec 4 22:55:51.416315726 (30h0m27.887526465s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.416474488 (30h0m27.887685217s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.418547936 (30h0m27.889758665s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.431075520 (30h0m27.902286249s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.438237447 (30h0m27.909448186s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.444558713 (30h0m27.915769452s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.454863965 (30h0m27.926074704s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.500395572 (30h0m27.971606301s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.002667307 (30h0m28.473878056s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.056889902 (30h0m28.528100641s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.357024719 (30h0m28.828235448s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.423527005 (30h0m28.894737734s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759657335 (30h0m54.230868064s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759677794 (30h0m54.230888523s) [seg_pos={0 0}, upload=26406]: Segment reached optimal threshold of 13 pieces. Dec 4 22:56:17.759902682 (30h0m54.231113661s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection Dec 4 22:56:17.759916668 (30h0m54.231127407s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760005618 (30h0m54.231216347s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection Dec 4 22:56:17.760014334 (30h0m54.231225063s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760032339 (30h0m54.231243068s) [seg_pos={0 0}, upload=26406]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 22:56:17.760113693 (30h0m54.231324423s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 22:56:17.760119615 (30h0m54.231330344s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 22:56:18.057470049 (30h0m54.528680788s) [upload=26406]: Done uploading segments... Dec 4 22:56:18.057547447 (30h0m54.528758187s) [upload=26406]: Upload finished. err: <nil> Duration: 5.647539701s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.733731378s trace: https://jaeger.dev.storj.io/trace/53db3ad570a94624 FAIL (2025-12-04 23:24:33 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 4 22:55:49.159528058 (30h0m25.630738787s) [upload=26406]: Starting upload Dec 4 22:55:49.159656774 (30h0m25.630867513s) [upload=26406]: Done starting upload Dec 4 22:55:49.159698914 (30h0m25.630909653s) [upload=26406]: Uploading segments... Dec 4 22:55:49.159881322 (30h0m25.631092061s) [upload=26406]: Got next segment. Inline: false Dec 4 22:55:49.159921538 (30h0m25.631132267s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 22:55:49.159927079 (30h0m25.631137808s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 22:55:49.407632244 (30h0m25.878842983s) [seg_pos={0 0}, upload=26406]: Begin upload segment... Dec 4 22:55:49.407685676 (30h0m25.878896415s) [seg_pos={0 0}, upload=26406]: Done begin upload segment. Dec 4 22:55:49.407698249 (30h0m25.878908988s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.407776349 (30h0m25.878987078s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.408596580 (30h0m25.879807319s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409349504 (30h0m25.880560243s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409963062 (30h0m25.881173801s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410210894 (30h0m25.881421623s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410541033 (30h0m25.881751772s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411049931 (30h0m25.88226068s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411451476 (30h0m25.882662215s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411905290 (30h0m25.883116039s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412457602 (30h0m25.883668331s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412740861 (30h0m25.8839516s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413363186 (30h0m25.884573926s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413762196 (30h0m25.884972935s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414199659 (30h0m25.885410398s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414579071 (30h0m25.88578981s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:50.491991030 (30h0m26.963201759s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 22:55:51.092442279 (30h0m27.563653008s) [upload=26406]: Next returned nil segment Dec 4 22:55:51.092455273 (30h0m27.563666012s) [upload=26406]: Waiting for error group managing segments... Dec 4 22:55:51.416315726 (30h0m27.887526465s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.416474488 (30h0m27.887685217s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.418547936 (30h0m27.889758665s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.431075520 (30h0m27.902286249s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.438237447 (30h0m27.909448186s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.444558713 (30h0m27.915769452s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.454863965 (30h0m27.926074704s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.500395572 (30h0m27.971606301s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.002667307 (30h0m28.473878056s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.056889902 (30h0m28.528100641s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.357024719 (30h0m28.828235448s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.423527005 (30h0m28.894737734s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759657335 (30h0m54.230868064s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759677794 (30h0m54.230888523s) [seg_pos={0 0}, upload=26406]: Segment reached optimal threshold of 13 pieces. Dec 4 22:56:17.759902682 (30h0m54.231113661s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection Dec 4 22:56:17.759916668 (30h0m54.231127407s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760005618 (30h0m54.231216347s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection Dec 4 22:56:17.760014334 (30h0m54.231225063s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760032339 (30h0m54.231243068s) [seg_pos={0 0}, upload=26406]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 22:56:17.760113693 (30h0m54.231324423s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 22:56:17.760119615 (30h0m54.231330344s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 22:56:18.057470049 (30h0m54.528680788s) [upload=26406]: Done uploading segments... Dec 4 22:56:18.057547447 (30h0m54.528758187s) [upload=26406]: Upload finished. err: <nil> Duration: 5.733778799s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.700704829s trace: https://jaeger.dev.storj.io/trace/53db3ad570a94624 FAIL (2025-12-04 23:23:24 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 4 22:55:49.159528058 (30h0m25.630738787s) [upload=26406]: Starting upload Dec 4 22:55:49.159656774 (30h0m25.630867513s) [upload=26406]: Done starting upload Dec 4 22:55:49.159698914 (30h0m25.630909653s) [upload=26406]: Uploading segments... Dec 4 22:55:49.159881322 (30h0m25.631092061s) [upload=26406]: Got next segment. Inline: false Dec 4 22:55:49.159921538 (30h0m25.631132267s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 22:55:49.159927079 (30h0m25.631137808s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 22:55:49.407632244 (30h0m25.878842983s) [seg_pos={0 0}, upload=26406]: Begin upload segment... Dec 4 22:55:49.407685676 (30h0m25.878896415s) [seg_pos={0 0}, upload=26406]: Done begin upload segment. Dec 4 22:55:49.407698249 (30h0m25.878908988s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.407776349 (30h0m25.878987078s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.408596580 (30h0m25.879807319s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409349504 (30h0m25.880560243s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409963062 (30h0m25.881173801s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410210894 (30h0m25.881421623s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410541033 (30h0m25.881751772s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411049931 (30h0m25.88226068s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411451476 (30h0m25.882662215s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411905290 (30h0m25.883116039s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412457602 (30h0m25.883668331s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412740861 (30h0m25.8839516s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413363186 (30h0m25.884573926s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413762196 (30h0m25.884972935s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414199659 (30h0m25.885410398s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414579071 (30h0m25.88578981s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:50.491991030 (30h0m26.963201759s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 22:55:51.092442279 (30h0m27.563653008s) [upload=26406]: Next returned nil segment Dec 4 22:55:51.092455273 (30h0m27.563666012s) [upload=26406]: Waiting for error group managing segments... Dec 4 22:55:51.416315726 (30h0m27.887526465s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.416474488 (30h0m27.887685217s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.418547936 (30h0m27.889758665s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.431075520 (30h0m27.902286249s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.438237447 (30h0m27.909448186s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.444558713 (30h0m27.915769452s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.454863965 (30h0m27.926074704s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.500395572 (30h0m27.971606301s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.002667307 (30h0m28.473878056s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.056889902 (30h0m28.528100641s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.357024719 (30h0m28.828235448s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.423527005 (30h0m28.894737734s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759657335 (30h0m54.230868064s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759677794 (30h0m54.230888523s) [seg_pos={0 0}, upload=26406]: Segment reached optimal threshold of 13 pieces. Dec 4 22:56:17.759902682 (30h0m54.231113661s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection Dec 4 22:56:17.759916668 (30h0m54.231127407s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760005618 (30h0m54.231216347s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection Dec 4 22:56:17.760014334 (30h0m54.231225063s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760032339 (30h0m54.231243068s) [seg_pos={0 0}, upload=26406]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 22:56:17.760113693 (30h0m54.231324423s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 22:56:17.760119615 (30h0m54.231330344s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 22:56:18.057470049 (30h0m54.528680788s) [upload=26406]: Done uploading segments... Dec 4 22:56:18.057547447 (30h0m54.528758187s) [upload=26406]: Upload finished. err: <nil> Duration: 5.70073121s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.697409979s trace: https://jaeger.dev.storj.io/trace/53db3ad570a94624 FAIL (2025-12-04 23:22:23 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 4 22:55:49.159528058 (30h0m25.630738787s) [upload=26406]: Starting upload Dec 4 22:55:49.159656774 (30h0m25.630867513s) [upload=26406]: Done starting upload Dec 4 22:55:49.159698914 (30h0m25.630909653s) [upload=26406]: Uploading segments... Dec 4 22:55:49.159881322 (30h0m25.631092061s) [upload=26406]: Got next segment. Inline: false Dec 4 22:55:49.159921538 (30h0m25.631132267s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 22:55:49.159927079 (30h0m25.631137808s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 22:55:49.407632244 (30h0m25.878842983s) [seg_pos={0 0}, upload=26406]: Begin upload segment... Dec 4 22:55:49.407685676 (30h0m25.878896415s) [seg_pos={0 0}, upload=26406]: Done begin upload segment. Dec 4 22:55:49.407698249 (30h0m25.878908988s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.407776349 (30h0m25.878987078s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.408596580 (30h0m25.879807319s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409349504 (30h0m25.880560243s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409963062 (30h0m25.881173801s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410210894 (30h0m25.881421623s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410541033 (30h0m25.881751772s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411049931 (30h0m25.88226068s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411451476 (30h0m25.882662215s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411905290 (30h0m25.883116039s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412457602 (30h0m25.883668331s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412740861 (30h0m25.8839516s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413363186 (30h0m25.884573926s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413762196 (30h0m25.884972935s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414199659 (30h0m25.885410398s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414579071 (30h0m25.88578981s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:50.491991030 (30h0m26.963201759s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 22:55:51.092442279 (30h0m27.563653008s) [upload=26406]: Next returned nil segment Dec 4 22:55:51.092455273 (30h0m27.563666012s) [upload=26406]: Waiting for error group managing segments... Dec 4 22:55:51.416315726 (30h0m27.887526465s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.416474488 (30h0m27.887685217s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.418547936 (30h0m27.889758665s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.431075520 (30h0m27.902286249s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.438237447 (30h0m27.909448186s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.444558713 (30h0m27.915769452s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.454863965 (30h0m27.926074704s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.500395572 (30h0m27.971606301s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.002667307 (30h0m28.473878056s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.056889902 (30h0m28.528100641s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.357024719 (30h0m28.828235448s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.423527005 (30h0m28.894737734s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759657335 (30h0m54.230868064s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759677794 (30h0m54.230888523s) [seg_pos={0 0}, upload=26406]: Segment reached optimal threshold of 13 pieces. Dec 4 22:56:17.759902682 (30h0m54.231113661s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection Dec 4 22:56:17.759916668 (30h0m54.231127407s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760005618 (30h0m54.231216347s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection Dec 4 22:56:17.760014334 (30h0m54.231225063s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760032339 (30h0m54.231243068s) [seg_pos={0 0}, upload=26406]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 22:56:17.760113693 (30h0m54.231324423s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 22:56:17.760119615 (30h0m54.231330344s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 22:56:18.057470049 (30h0m54.528680788s) [upload=26406]: Done uploading segments... Dec 4 22:56:18.057547447 (30h0m54.528758187s) [upload=26406]: Upload finished. err: <nil> Duration: 5.697444143s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.678148119s trace: https://jaeger.dev.storj.io/trace/53db3ad570a94624 FAIL (2025-12-04 23:21:56 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 4 22:55:49.159528058 (30h0m25.630738787s) [upload=26406]: Starting upload Dec 4 22:55:49.159656774 (30h0m25.630867513s) [upload=26406]: Done starting upload Dec 4 22:55:49.159698914 (30h0m25.630909653s) [upload=26406]: Uploading segments... Dec 4 22:55:49.159881322 (30h0m25.631092061s) [upload=26406]: Got next segment. Inline: false Dec 4 22:55:49.159921538 (30h0m25.631132267s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 22:55:49.159927079 (30h0m25.631137808s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 22:55:49.407632244 (30h0m25.878842983s) [seg_pos={0 0}, upload=26406]: Begin upload segment... Dec 4 22:55:49.407685676 (30h0m25.878896415s) [seg_pos={0 0}, upload=26406]: Done begin upload segment. Dec 4 22:55:49.407698249 (30h0m25.878908988s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.407776349 (30h0m25.878987078s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.408596580 (30h0m25.879807319s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409349504 (30h0m25.880560243s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409963062 (30h0m25.881173801s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410210894 (30h0m25.881421623s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410541033 (30h0m25.881751772s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411049931 (30h0m25.88226068s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411451476 (30h0m25.882662215s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411905290 (30h0m25.883116039s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412457602 (30h0m25.883668331s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412740861 (30h0m25.8839516s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413363186 (30h0m25.884573926s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413762196 (30h0m25.884972935s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414199659 (30h0m25.885410398s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414579071 (30h0m25.88578981s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:50.491991030 (30h0m26.963201759s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 22:55:51.092442279 (30h0m27.563653008s) [upload=26406]: Next returned nil segment Dec 4 22:55:51.092455273 (30h0m27.563666012s) [upload=26406]: Waiting for error group managing segments... Dec 4 22:55:51.416315726 (30h0m27.887526465s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.416474488 (30h0m27.887685217s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.418547936 (30h0m27.889758665s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.431075520 (30h0m27.902286249s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.438237447 (30h0m27.909448186s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.444558713 (30h0m27.915769452s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.454863965 (30h0m27.926074704s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.500395572 (30h0m27.971606301s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.002667307 (30h0m28.473878056s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.056889902 (30h0m28.528100641s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.357024719 (30h0m28.828235448s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.423527005 (30h0m28.894737734s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759657335 (30h0m54.230868064s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759677794 (30h0m54.230888523s) [seg_pos={0 0}, upload=26406]: Segment reached optimal threshold of 13 pieces. Dec 4 22:56:17.759902682 (30h0m54.231113661s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection Dec 4 22:56:17.759916668 (30h0m54.231127407s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760005618 (30h0m54.231216347s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection Dec 4 22:56:17.760014334 (30h0m54.231225063s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760032339 (30h0m54.231243068s) [seg_pos={0 0}, upload=26406]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 22:56:17.760113693 (30h0m54.231324423s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 22:56:17.760119615 (30h0m54.231330344s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 22:56:18.057470049 (30h0m54.528680788s) [upload=26406]: Done uploading segments... Dec 4 22:56:18.057547447 (30h0m54.528758187s) [upload=26406]: Upload finished. err: <nil> Duration: 5.678336447s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.901664934s trace: https://jaeger.dev.storj.io/trace/53db3ad570a94624 FAIL (2025-12-04 23:21:36 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 4 22:55:49.159528058 (30h0m25.630738787s) [upload=26406]: Starting upload Dec 4 22:55:49.159656774 (30h0m25.630867513s) [upload=26406]: Done starting upload Dec 4 22:55:49.159698914 (30h0m25.630909653s) [upload=26406]: Uploading segments... Dec 4 22:55:49.159881322 (30h0m25.631092061s) [upload=26406]: Got next segment. Inline: false Dec 4 22:55:49.159921538 (30h0m25.631132267s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 22:55:49.159927079 (30h0m25.631137808s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 22:55:49.407632244 (30h0m25.878842983s) [seg_pos={0 0}, upload=26406]: Begin upload segment... Dec 4 22:55:49.407685676 (30h0m25.878896415s) [seg_pos={0 0}, upload=26406]: Done begin upload segment. Dec 4 22:55:49.407698249 (30h0m25.878908988s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.407776349 (30h0m25.878987078s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.408596580 (30h0m25.879807319s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409349504 (30h0m25.880560243s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409963062 (30h0m25.881173801s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410210894 (30h0m25.881421623s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410541033 (30h0m25.881751772s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411049931 (30h0m25.88226068s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411451476 (30h0m25.882662215s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411905290 (30h0m25.883116039s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412457602 (30h0m25.883668331s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412740861 (30h0m25.8839516s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413363186 (30h0m25.884573926s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413762196 (30h0m25.884972935s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414199659 (30h0m25.885410398s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414579071 (30h0m25.88578981s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:50.491991030 (30h0m26.963201759s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 22:55:51.092442279 (30h0m27.563653008s) [upload=26406]: Next returned nil segment Dec 4 22:55:51.092455273 (30h0m27.563666012s) [upload=26406]: Waiting for error group managing segments... Dec 4 22:55:51.416315726 (30h0m27.887526465s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.416474488 (30h0m27.887685217s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.418547936 (30h0m27.889758665s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.431075520 (30h0m27.902286249s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.438237447 (30h0m27.909448186s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.444558713 (30h0m27.915769452s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.454863965 (30h0m27.926074704s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.500395572 (30h0m27.971606301s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.002667307 (30h0m28.473878056s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.056889902 (30h0m28.528100641s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.357024719 (30h0m28.828235448s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.423527005 (30h0m28.894737734s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759657335 (30h0m54.230868064s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759677794 (30h0m54.230888523s) [seg_pos={0 0}, upload=26406]: Segment reached optimal threshold of 13 pieces. Dec 4 22:56:17.759902682 (30h0m54.231113661s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection Dec 4 22:56:17.759916668 (30h0m54.231127407s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760005618 (30h0m54.231216347s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection Dec 4 22:56:17.760014334 (30h0m54.231225063s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760032339 (30h0m54.231243068s) [seg_pos={0 0}, upload=26406]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 22:56:17.760113693 (30h0m54.231324423s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 22:56:17.760119615 (30h0m54.231330344s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 22:56:18.057470049 (30h0m54.528680788s) [upload=26406]: Done uploading segments... Dec 4 22:56:18.057547447 (30h0m54.528758187s) [upload=26406]: Upload finished. err: <nil> Duration: 5.901685894s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.660521046s trace: https://jaeger.dev.storj.io/trace/53db3ad570a94624 FAIL (2025-12-04 23:20:25 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 4 22:55:49.159528058 (30h0m25.630738787s) [upload=26406]: Starting upload Dec 4 22:55:49.159656774 (30h0m25.630867513s) [upload=26406]: Done starting upload Dec 4 22:55:49.159698914 (30h0m25.630909653s) [upload=26406]: Uploading segments... Dec 4 22:55:49.159881322 (30h0m25.631092061s) [upload=26406]: Got next segment. Inline: false Dec 4 22:55:49.159921538 (30h0m25.631132267s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 22:55:49.159927079 (30h0m25.631137808s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 22:55:49.407632244 (30h0m25.878842983s) [seg_pos={0 0}, upload=26406]: Begin upload segment... Dec 4 22:55:49.407685676 (30h0m25.878896415s) [seg_pos={0 0}, upload=26406]: Done begin upload segment. Dec 4 22:55:49.407698249 (30h0m25.878908988s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.407776349 (30h0m25.878987078s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.408596580 (30h0m25.879807319s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409349504 (30h0m25.880560243s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.409963062 (30h0m25.881173801s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410210894 (30h0m25.881421623s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.410541033 (30h0m25.881751772s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411049931 (30h0m25.88226068s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411451476 (30h0m25.882662215s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.411905290 (30h0m25.883116039s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412457602 (30h0m25.883668331s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.412740861 (30h0m25.8839516s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413363186 (30h0m25.884573926s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.413762196 (30h0m25.884972935s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414199659 (30h0m25.885410398s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:49.414579071 (30h0m25.88578981s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Uploading piece... Dec 4 22:55:50.491991030 (30h0m26.963201759s) [noise=true , address=77.90.31.19:4841 , piece_id=PADT5VO47XC7PBZJ46XJWERIJXFXB7DKAFMADRRQL6HQ2XMAAN6A, seg_pos={0 0}, upload=26406]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 4 22:55:51.092442279 (30h0m27.563653008s) [upload=26406]: Next returned nil segment Dec 4 22:55:51.092455273 (30h0m27.563666012s) [upload=26406]: Waiting for error group managing segments... Dec 4 22:55:51.416315726 (30h0m27.887526465s) [noise=true , address=145.79.227.9:4366 , piece_id=SXJDR3YBIC76NWAQ6YM5AQMRG2W4CV7NUIKHEQC6AUY7VFNMRIKA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.416474488 (30h0m27.887685217s) [noise=true , address=145.79.227.44:4194 , piece_id=W4NTFDJ4NCX7Q4JPYJRTTMVWTZWLH2LQMPBCOI33MD6KJOBHPW5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.418547936 (30h0m27.889758665s) [noise=true , address=145.79.227.29:4942 , piece_id=7TARTXXAUPYWMCXCA54H253QXH3DJAVBUCLRM5YPPSKC3IKC7ZWA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.431075520 (30h0m27.902286249s) [noise=true , address=145.79.227.22:4510 , piece_id=4UPHNABZN4NQZ3L62S4HBFKRXQD2F22NZ4J4FZKMY5LOEFU34X5Q, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.438237447 (30h0m27.909448186s) [noise=true , address=145.79.227.8:4329 , piece_id=IKXNC772PKB5QFBLO5SLXHDWCVZ55PQRF4RSW2XPGKZ4IEHJAULA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.444558713 (30h0m27.915769452s) [noise=true , address=145.79.227.25:4619 , piece_id=WXEUBRQX2YAHWEQ7I5BUR5PJKYF2M2OSAVZT55SKTHL4Q7HHTOCA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.454863965 (30h0m27.926074704s) [noise=true , address=145.79.227.21:4474 , piece_id=BACAAGU3N2BDZNHJTWQQR3DIHU5RZGBKJ4TSLVZ77R63OBATWDZQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:51.500395572 (30h0m27.971606301s) [noise=true , address=45.91.116.4:4940 , piece_id=LFCDPANIUGQMYOA2W5UCCX3CIAHC7424IWBRRAJGSXFTG6ND7JCQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.002667307 (30h0m28.473878056s) [noise=true , address=145.79.227.16:4911 , piece_id=ZJO5RN6X552GVT3IOOZX4DV5ZELHPQHBBDSA6YVBK4IZQDTPO3JQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.056889902 (30h0m28.528100641s) [noise=true , address=145.79.227.26:4654 , piece_id=UJOVD3QRYPPLO7LMSI7L25O3NGLCXDZPKYCKRCMJM76MRJFBH4HQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.357024719 (30h0m28.828235448s) [noise=true , address=145.79.227.34:4905 , piece_id=W4TCEKHRKNOIQ4BYI36KRICWDNJXHQ5ZVWR6Y3ATBNIS3IP5CDYQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:55:52.423527005 (30h0m28.894737734s) [noise=true , address=145.79.227.41:4087 , piece_id=KE5262VIE27755LNEN4RISTKG4INSZIJ4IXVYVLG44TTZ34N23WQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759657335 (30h0m54.230868064s) [noise=true , address=145.79.227.33:4869 , piece_id=SEWLHK2A7OKFIUQT3QQRFKSMHLBGCI7GY2QALAFUJFFBGAIRQBDQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: <nil> Dec 4 22:56:17.759677794 (30h0m54.230888523s) [seg_pos={0 0}, upload=26406]: Segment reached optimal threshold of 13 pieces. Dec 4 22:56:17.759902682 (30h0m54.231113661s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43536->145.79.227.42:4121: use of closed network connection Dec 4 22:56:17.759916668 (30h0m54.231127407s) [noise=true , address=145.79.227.42:4121 , piece_id=XH3BRK76WULUWEEMY7ISE6ZISFYQ7Q2HCHR4BC3FMXXJI7MBYSOQ, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760005618 (30h0m54.231216347s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50122->145.79.227.38:4977: use of closed network connection Dec 4 22:56:17.760014334 (30h0m54.231225063s) [noise=true , address=145.79.227.38:4977 , piece_id=3X2FM6XGI3KKUYTQAEH3H5O4FNPMVGQITQ4NJGM2GK6UIL3FF6JA, seg_pos={0 0}, upload=26406]: Long tail cancellation due to optimal threshold Dec 4 22:56:17.760032339 (30h0m54.231243068s) [seg_pos={0 0}, upload=26406]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 22:56:17.760113693 (30h0m54.231324423s) [upload=26406]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 22:56:17.760119615 (30h0m54.231330344s) [upload=26406]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 22:56:18.057470049 (30h0m54.528680788s) [upload=26406]: Done uploading segments... Dec 4 22:56:18.057547447 (30h0m54.528758187s) [upload=26406]: Upload finished. err: <nil> Duration: 5.660589646s core uplink us-select-1

Orbiter Check History

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)