Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........................................X.................X.] duration: 3.472431446s trace: https://jaeger.dev.storj.io/trace/61feaa3186d863f0 FAIL (2025-12-04 14:22:08 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.47246435s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........................................X.................X] duration: 5.0019713s trace: https://jaeger.dev.storj.io/trace/61feaa3186d863f0 FAIL (2025-12-04 14:21:31 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001996399s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................X.................] duration: 3.446365941s trace: https://jaeger.dev.storj.io/trace/14069ca4e24a6dac FAIL (2025-12-04 14:20:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.446394728s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................X................] duration: 3.532531002s trace: https://jaeger.dev.storj.io/trace/14069ca4e24a6dac FAIL (2025-12-04 14:20:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.532572943s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................................X...............] duration: 3.454859199s trace: https://jaeger.dev.storj.io/trace/14069ca4e24a6dac FAIL (2025-12-04 14:19:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.454893695s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................................X..............] duration: 3.530962728s trace: https://jaeger.dev.storj.io/trace/14069ca4e24a6dac FAIL (2025-12-04 14:18:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.530991141s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................................X.............] duration: 3.574951733s trace: https://jaeger.dev.storj.io/trace/14069ca4e24a6dac FAIL (2025-12-04 14:17:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.574975629s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................................X............] duration: 3.495128609s trace: https://jaeger.dev.storj.io/trace/14069ca4e24a6dac FAIL (2025-12-04 14:16:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.495157814s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................................X...........] duration: 3.481174258s trace: https://jaeger.dev.storj.io/trace/14069ca4e24a6dac FAIL (2025-12-04 14:16:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.481206388s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................................X..........] duration: 3.563567534s trace: https://jaeger.dev.storj.io/trace/14069ca4e24a6dac FAIL (2025-12-04 14:14:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.563599566s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................................X.........] duration: 3.650281484s trace: https://jaeger.dev.storj.io/trace/14069ca4e24a6dac FAIL (2025-12-04 14:14:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.650418956s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................................X........] duration: 5.001362483s trace: https://jaeger.dev.storj.io/trace/14069ca4e24a6dac FAIL (2025-12-04 14:13:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001394854s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 3.431108666s trace: https://jaeger.dev.storj.io/trace/14069ca4e24a6dac FAIL (2025-12-04 14:12:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.431138563s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................................X......] duration: 3.537315106s trace: https://jaeger.dev.storj.io/trace/14069ca4e24a6dac FAIL (2025-12-04 14:12:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.537344682s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................................X.....] duration: 3.522864021s trace: https://jaeger.dev.storj.io/trace/14069ca4e24a6dac FAIL (2025-12-04 14:12:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.522893135s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................................X....] duration: 3.4413916s trace: https://jaeger.dev.storj.io/trace/14069ca4e24a6dac FAIL (2025-12-04 14:11:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.44141786s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................................X...] duration: 3.509808568s trace: https://jaeger.dev.storj.io/trace/14069ca4e24a6dac FAIL (2025-12-04 14:10:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.50983673s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................................X..] duration: 3.579170024s trace: https://jaeger.dev.storj.io/trace/14069ca4e24a6dac FAIL (2025-12-04 14:10:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.579197996s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................................X.] duration: 3.604371334s trace: https://jaeger.dev.storj.io/trace/14069ca4e24a6dac FAIL (2025-12-04 14:10:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.604393115s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.001112715s trace: https://jaeger.dev.storj.io/trace/14069ca4e24a6dac FAIL (2025-12-04 14:09:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001135227s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.723280638s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 12:10:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.723312137s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.657335677s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 12:10:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.657363269s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.798421663s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 12:09:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.798462339s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.647174181s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 12:08:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.647200921s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.728633343s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 12:07:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.728647931s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.861385927s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 12:07:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.861423539s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.729854445s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 12:06:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.729880005s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.672864229s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 12:06:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.672904434s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.755575247s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 12:05:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.755670138s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.74462835s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 12:04:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.744655183s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.827722666s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 12:04:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.827750058s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.727574447s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 12:03:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.727622329s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.741008436s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 12:02:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.741038844s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.791684806s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 12:01:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.791711215s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.741840159s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 12:00:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.741933368s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.649977905s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 12:00:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.650018263s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.714040815s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 11:59:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.714068408s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.743938946s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 11:58:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.743968202s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.707775676s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 11:57:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.707804571s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.615412123s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 11:57:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.615438873s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.735374565s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 11:56:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.735402347s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.642506855s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 11:56:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.642539587s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.721876574s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 11:55:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.721903165s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.701740993s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 11:54:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.701770207s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.741250721s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 11:53:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.741370389s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.803817937s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 11:53:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.803851222s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.649965003s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 11:52:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.64998875s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.656882485s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 11:51:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.656909586s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.706767784s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 11:50:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.70679705s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.751674068s trace: https://jaeger.dev.storj.io/trace/7723de6c40e51b6f FAIL (2025-12-04 11:50:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, Piece ID: QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A) context deadline exceeded; piecestore: (Node ID: 1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, Piece ID: IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ) context deadline exceeded; piecestore: (Node ID: 1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, Piece ID: WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA) context deadline exceeded; piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: 6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA) context deadline exceeded ======LOGS====== Dec 4 11:24:55.690968696 (18h29m32.162179425s) [upload=16258]: Starting upload Dec 4 11:24:55.691048709 (18h29m32.162259438s) [upload=16258]: Done starting upload Dec 4 11:24:55.691092242 (18h29m32.162302971s) [upload=16258]: Uploading segments... Dec 4 11:24:55.691259100 (18h29m32.162469829s) [upload=16258]: Got next segment. Inline: false Dec 4 11:24:55.691304536 (18h29m32.162515275s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 11:24:55.691310617 (18h29m32.162521346s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 11:24:56.255206122 (18h29m32.726416861s) [seg_pos={0 0}, upload=16258]: Begin upload segment... Dec 4 11:24:56.255264473 (18h29m32.726475202s) [seg_pos={0 0}, upload=16258]: Done begin upload segment. Dec 4 11:24:56.255296153 (18h29m32.726506903s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255311422 (18h29m32.726522161s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.255690244 (18h29m32.726900983s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256107468 (18h29m32.727318197s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256161762 (18h29m32.727372501s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256444059 (18h29m32.727654788s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256561893 (18h29m32.727772632s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256752326 (18h29m32.727963056s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.256943550 (18h29m32.728154289s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257065473 (18h29m32.728276202s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257338463 (18h29m32.728549192s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257353271 (18h29m32.728564s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.257906955 (18h29m32.729117694s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258382401 (18h29m32.72959313s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.258810796 (18h29m32.730021535s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:56.259065741 (18h29m32.73027648s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Uploading piece... Dec 4 11:24:57.270079201 (18h29m33.74128993s) [noise=true , address=77.90.31.16:4733 , piece_id=J7CEYJBOM44KP6ZGJPJ7PMRFPBN3CWYPIELJANX4MTB7BET6DI7Q, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:57.342671161 (18h29m33.81388189s) [noise=true , address=77.90.31.16:4735 , piece_id=LJ644J7I3U3KATJ4BFXEUHIT5IDWTD74DH2ET6IPXVM43LYL5AVA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: storage node overloaded, request limit: 2000 Dec 4 11:24:58.040132895 (18h29m34.511343624s) [upload=16258]: Next returned nil segment Dec 4 11:24:58.040146260 (18h29m34.511356989s) [upload=16258]: Waiting for error group managing segments... Dec 4 11:24:58.394317316 (18h29m34.865528045s) [noise=true , address=77.90.31.11:4553 , piece_id=QFSHIVQFZIBERB7237NLKRYZIARGEVJUO7HHK65SD3CDL3VFKZMQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.437280447 (18h29m34.908491186s) [noise=true , address=145.79.227.34:4906 , piece_id=XA4RA2S3VZV6F4RYNCFL6SKYIFTRVXYBDNMRSWMZ5WAAZGLK2SZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.439793622 (18h29m34.911004351s) [noise=true , address=145.79.227.24:4583 , piece_id=M34WD6CYMRMEYLDGMUVRGQGO5IFTAT53SJWBJ7Y24BUUVW4554KA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.442618912 (18h29m34.913829651s) [noise=true , address=145.79.227.26:4655 , piece_id=WUTFUBBLDCRYEZCTRMYGWJIYUQKCY4NFZN7FSUYNGFEZPWKSYPZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.477220100 (18h29m34.94843084s) [noise=true , address=145.79.227.42:4121 , piece_id=76NNUASGOR4Y3SLOUXSKUJ6WI424MPWZ7UPHAMGDBMQDT2CDFBTQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.484798822 (18h29m34.956009561s) [noise=true , address=145.79.227.28:4726 , piece_id=YTHSARDI3MHM6NZAWOCXUZJDCG7Y32TRT4AHVONSMTPJ66YDOUDQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:58.865178914 (18h29m35.336389643s) [noise=true , address=145.79.227.44:4193 , piece_id=IPUULT3TSB5EDR73LXLGSDI7FLJMU5SC2THHIB6WGZQH44DYXEFQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.222427280 (18h29m35.693638009s) [noise=true , address=145.79.227.32:4836 , piece_id=JXJS6T4LN6I7HYBFF7EVGT6TIFCAASANB4T46VLC753GJIFZEVZA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.238063816 (18h29m35.709274545s) [noise=true , address=145.79.227.39:4014 , piece_id=IE6JQ6R2R2XSYD6E5T6WJRRCUYJ4VVFFP7J7P5IPCDHBLDWNUBWQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.261333716 (18h29m35.732544455s) [noise=true , address=145.79.227.27:4689 , piece_id=QRFOWQRODVNOFQY7D77DMI2OXFQY6UI6HP4VFIEO4WUNRLZTLI2A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.448171725 (18h29m35.919382464s) [noise=true , address=145.79.227.31:4799 , piece_id=6NPI6YDEMFRUZ3RP6SM5BE2UFVNN5ZYNMSRZDLORJV6AQQEZRJCA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:24:59.546850579 (18h29m36.018061318s) [noise=true , address=145.79.227.40:4049 , piece_id=NZSUYLDQX3JOUY4FLOBSVLAHY5GBSVCROGE7UTCQQFZAYIQIMO4A, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484505090 (18h29m36.955715829s) [noise=true , address=145.79.227.38:4978 , piece_id=F4G5JZRPL76MHRVJANJG4AYNLIGXRAPYACTXQHOZLGIE3TDWWPGA, seg_pos={0 0}, upload=16258]: Done uploading piece. err: <nil> Dec 4 11:25:00.484530839 (18h29m36.955741578s) [seg_pos={0 0}, upload=16258]: Segment reached optimal threshold of 13 pieces. Dec 4 11:25:00.484803288 (18h29m36.956014027s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57322->145.79.227.8:4329: use of closed network connection Dec 4 11:25:00.484813227 (18h29m36.956023966s) [noise=true , address=145.79.227.8:4329 , piece_id=GNLWAH2PRHWYDX7G2FFM54LLUOJYSBDLH2MZLHLP4UOJ5XB5NZJQ, seg_pos={0 0}, upload=16258]: Long tail cancellation due to optimal threshold Dec 4 11:25:00.484862521 (18h29m36.95607325s) [seg_pos={0 0}, upload=16258]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 11:25:00.484945649 (18h29m36.956156378s) [upload=16258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 11:25:00.484952552 (18h29m36.956163281s) [upload=16258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 11:25:00.802002411 (18h29m37.27321315s) [upload=16258]: Done uploading segments... Dec 4 11:25:00.802027419 (18h29m37.273238158s) [upload=16258]: Upload finished. err: <nil> Duration: 5.751696541s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)