Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 427918 reports (Page 7964 of 8559)
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [..................................................X.........]
duration: 
6.333767658s
FAIL (2025-07-21 02:33:57 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.333795833s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)
antiblip: 
1 out of 60 is failed (threshold=3) [.................................................X..........]
duration: 
2.443956502s
FAIL (2025-07-21 02:33:56 UTC) 
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 2.443977934s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [...................................................X........]
duration: 
6.561550836s
FAIL (2025-07-21 02:33:36 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.56157955s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)
antiblip: 
1 out of 60 is failed (threshold=3) [..................................................X.........]
duration: 
2.360032149s
FAIL (2025-07-21 02:33:26 UTC) 
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 2.360081001s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [....................................................X.......]
duration: 
6.828303056s
FAIL (2025-07-21 02:33:15 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.828324457s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)
antiblip: 
1 out of 60 is failed (threshold=3) [...................................................X........]
duration: 
2.515725429s
FAIL (2025-07-21 02:32:28 UTC) 
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 2.515749927s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [.....................................................X......]
duration: 
6.509143385s
FAIL (2025-07-21 02:32:15 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.509158153s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)
antiblip: 
1 out of 60 is failed (threshold=3) [....................................................X.......]
duration: 
2.415516307s
FAIL (2025-07-21 02:32:07 UTC) 
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 2.415540622s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [......................................................X.....]
duration: 
6.215385385s
FAIL (2025-07-21 02:31:54 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.215407477s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)
antiblip: 
1 out of 60 is failed (threshold=3) [.....................................................X......]
duration: 
2.428163033s
FAIL (2025-07-21 02:31:51 UTC) 
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 2.42818725s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)
antiblip: 
1 out of 60 is failed (threshold=3) [......................................................X.....]
duration: 
2.496721776s
FAIL (2025-07-21 02:30:43 UTC) 
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 2.496744981s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [.......................................................X....]
duration: 
6.589248319s
FAIL (2025-07-21 02:30:42 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.589269219s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [........................................................X...]
duration: 
6.430570532s
FAIL (2025-07-21 02:30:15 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.430706982s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)
antiblip: 
1 out of 60 is failed (threshold=3) [.......................................................X....]
duration: 
2.478891031s
FAIL (2025-07-21 02:30:14 UTC) 
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 2.478907432s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)
antiblip: 
1 out of 60 is failed (threshold=3) [........................................................X...]
duration: 
2.641348851s
FAIL (2025-07-21 02:29:48 UTC) 
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 2.641364562s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [.........................................................X..]
duration: 
6.497619175s
FAIL (2025-07-21 02:29:25 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.497646087s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)
antiblip: 
1 out of 60 is failed (threshold=3) [.........................................................X..]
duration: 
2.488220571s
FAIL (2025-07-21 02:28:33 UTC) 
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 2.488240629s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [..........................................................X.]
duration: 
6.436189791s
FAIL (2025-07-21 02:28:27 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.436303959s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)
antiblip: 
1 out of 60 is failed (threshold=3) [..........................................................X.]
duration: 
2.563963201s
FAIL (2025-07-21 02:28:18 UTC) 
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 2.563984151s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)
antiblip: 
1 out of 60 is failed (threshold=3) [...........................................................X]
duration: 
30.000368183s
FAIL (2025-07-21 02:27:56 UTC) 
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 30.000396418s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [...........................................................X]
duration: 
30.001301968s
FAIL (2025-07-21 02:27:50 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.00131909s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [X...........................................................]
duration: 
5.71013361s
FAIL (2025-07-21 01:38:55 UTC) 
orbiter warning: 
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments...
Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil>
Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload
Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload
Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments...
Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false
Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment...
Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment.
Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment
Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments...
Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded
Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection
Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments...
Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 5.71015523s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [.X..........................................................]
duration: 
5.71110025s
FAIL (2025-07-21 01:38:18 UTC) 
orbiter warning: 
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments...
Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil>
Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload
Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload
Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments...
Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false
Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment...
Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment.
Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment
Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments...
Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded
Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection
Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments...
Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 5.711130347s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [..X.........................................................]
duration: 
5.718809429s
FAIL (2025-07-21 01:37:52 UTC) 
orbiter warning: 
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments...
Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil>
Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload
Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload
Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments...
Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false
Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment...
Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment.
Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment
Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments...
Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded
Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection
Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments...
Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 5.718834447s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [...X........................................................]
duration: 
5.858330384s
FAIL (2025-07-21 01:36:54 UTC) 
orbiter warning: 
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments...
Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil>
Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload
Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload
Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments...
Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false
Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment...
Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment.
Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment
Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments...
Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded
Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection
Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments...
Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 5.858356925s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [....X.......................................................]
duration: 
5.766423852s
FAIL (2025-07-21 01:35:51 UTC) 
orbiter warning: 
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments...
Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil>
Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload
Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload
Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments...
Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false
Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment...
Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment.
Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment
Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments...
Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded
Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection
Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments...
Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 5.766453209s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [.....X......................................................]
duration: 
7.149755892s
FAIL (2025-07-21 01:35:12 UTC) 
orbiter warning: 
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments...
Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil>
Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload
Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload
Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments...
Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false
Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment...
Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment.
Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment
Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments...
Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded
Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection
Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments...
Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 7.14977638s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [......X.....................................................]
duration: 
5.857314512s
FAIL (2025-07-21 01:34:25 UTC) 
orbiter warning: 
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments...
Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil>
Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload
Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload
Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments...
Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false
Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment...
Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment.
Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment
Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments...
Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded
Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection
Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments...
Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 5.857343376s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [.......X....................................................]
duration: 
5.830278493s
FAIL (2025-07-21 01:33:27 UTC) 
orbiter warning: 
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments...
Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil>
Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload
Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload
Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments...
Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false
Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment...
Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment.
Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment
Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments...
Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded
Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection
Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments...
Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 5.830326004s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [........X...................................................]
duration: 
5.826572371s
FAIL (2025-07-21 01:33:09 UTC) 
orbiter warning: 
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments...
Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil>
Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload
Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload
Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments...
Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false
Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment...
Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment.
Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment
Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments...
Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded
Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection
Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments...
Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 5.826600256s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [.........X..................................................]
duration: 
5.89484386s
FAIL (2025-07-21 01:32:08 UTC) 
orbiter warning: 
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments...
Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil>
Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload
Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload
Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments...
Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false
Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment...
Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment.
Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment
Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments...
Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded
Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection
Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments...
Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 5.894873025s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [..........X.................................................]
duration: 
5.95024926s
FAIL (2025-07-21 01:31:39 UTC) 
orbiter warning: 
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments...
Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil>
Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload
Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload
Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments...
Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false
Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment...
Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment.
Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment
Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments...
Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded
Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection
Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments...
Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 5.950275963s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [...........X................................................]
duration: 
5.807763112s
FAIL (2025-07-21 01:31:13 UTC) 
orbiter warning: 
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments...
Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil>
Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload
Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload
Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments...
Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false
Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment...
Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment.
Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment
Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments...
Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded
Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection
Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments...
Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 5.807799222s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [............X...............................................]
duration: 
5.773209985s
FAIL (2025-07-21 01:30:32 UTC) 
orbiter warning: 
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments...
Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil>
Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload
Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload
Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments...
Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false
Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment...
Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment.
Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment
Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments...
Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded
Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection
Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments...
Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 5.773237057s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [.............X..............................................]
duration: 
5.695807102s
FAIL (2025-07-21 01:30:15 UTC) 
orbiter warning: 
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments...
Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil>
Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload
Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload
Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments...
Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false
Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment...
Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment.
Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment
Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments...
Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded
Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection
Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments...
Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 5.695835767s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [..............X.............................................]
duration: 
5.838366387s
FAIL (2025-07-21 01:29:47 UTC) 
orbiter warning: 
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments...
Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil>
Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload
Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload
Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments...
Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false
Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment...
Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment.
Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment
Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments...
Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded
Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection
Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments...
Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 5.838393499s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [...............X............................................]
duration: 
5.729578837s
FAIL (2025-07-21 01:28:32 UTC) 
orbiter warning: 
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments...
Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil>
Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload
Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload
Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments...
Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false
Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment...
Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment.
Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment
Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments...
Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded
Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection
Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments...
Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 5.729608193s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [................X...........................................]
duration: 
5.733263224s
FAIL (2025-07-21 01:27:52 UTC) 
orbiter warning: 
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments...
Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil>
Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload
Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload
Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments...
Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false
Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment...
Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment.
Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment
Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments...
Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded
Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection
Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments...
Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 5.733295776s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [.................X..........................................]
duration: 
5.987918116s
FAIL (2025-07-21 01:26:50 UTC) 
orbiter warning: 
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments...
Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil>
Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload
Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload
Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments...
Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false
Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment...
Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment.
Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment
Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments...
Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded
Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection
Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments...
Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 5.98794576s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [..................X.........................................]
duration: 
5.742652387s
FAIL (2025-07-21 01:26:02 UTC) 
orbiter warning: 
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments...
Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil>
Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload
Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload
Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments...
Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false
Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment...
Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment.
Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment
Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments...
Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded
Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection
Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments...
Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 5.742688065s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [...................X........................................]
duration: 
5.78286991s
FAIL (2025-07-21 01:25:09 UTC) 
orbiter warning: 
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments...
Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil>
Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload
Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload
Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments...
Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false
Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment...
Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment.
Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment
Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments...
Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded
Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection
Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments...
Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 5.782899435s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [....................X.......................................]
duration: 
5.806962121s
FAIL (2025-07-21 01:24:02 UTC) 
orbiter warning: 
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments...
Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil>
Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload
Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload
Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments...
Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false
Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment...
Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment.
Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment
Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments...
Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded
Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection
Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments...
Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 5.806989774s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [.....................X......................................]
duration: 
5.865472399s
FAIL (2025-07-21 01:23:42 UTC) 
orbiter warning: 
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments...
Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil>
Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload
Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload
Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments...
Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false
Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment...
Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment.
Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment
Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments...
Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded
Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection
Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments...
Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 5.865497327s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [......................X.....................................]
duration: 
6.01292895s
FAIL (2025-07-21 01:23:14 UTC) 
orbiter warning: 
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments...
Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil>
Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload
Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload
Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments...
Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false
Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment...
Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment.
Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment
Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments...
Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded
Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection
Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments...
Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 6.012954768s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [.......................X....................................]
duration: 
5.907752886s
FAIL (2025-07-21 01:22:02 UTC) 
orbiter warning: 
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments...
Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil>
Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload
Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload
Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments...
Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false
Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment...
Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment.
Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment
Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments...
Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded
Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection
Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments...
Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 5.907776402s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [........................X...................................]
duration: 
6.360403056s
FAIL (2025-07-21 01:21:14 UTC) 
orbiter warning: 
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments...
Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil>
Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload
Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload
Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments...
Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false
Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment...
Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment.
Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment
Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments...
Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded
Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection
Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments...
Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 6.360435568s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [.........................X..................................]
duration: 
6.126805288s
FAIL (2025-07-21 01:20:04 UTC) 
orbiter warning: 
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments...
Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil>
Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload
Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload
Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments...
Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false
Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment...
Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment.
Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment
Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments...
Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded
Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection
Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments...
Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 6.126833069s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [..........................X.................................]
duration: 
5.849297647s
FAIL (2025-07-21 01:19:12 UTC) 
orbiter warning: 
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments...
Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil>
Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload
Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload
Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments...
Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false
Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment...
Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment.
Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment
Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments...
Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded
Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection
Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments...
Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 5.84932554s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [...........................X................................]
duration: 
7.815417935s
FAIL (2025-07-21 01:18:57 UTC) 
orbiter warning: 
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments...
Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil>
Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload
Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload
Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments...
Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false
Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment...
Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment.
Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment
Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments...
Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded
Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection
Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments...
Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 7.815442992s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [............................X...............................]
duration: 
6.148574368s
FAIL (2025-07-21 01:17:41 UTC) 
orbiter warning: 
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
g_pos={0 0}, upload=39980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Jul 21 00:55:56.746681935 (50h5m3.11926129s) [seg_pos={0 0}, upload=39980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 21 00:55:56.746757239 (50h5m3.119336595s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 21 00:55:56.746762169 (50h5m3.119341524s) [upload=39980]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 21 00:55:57.046171622 (50h5m3.418750977s) [upload=39980]: Done uploading segments...
Jul 21 00:55:57.046196299 (50h5m3.418775644s) [upload=39980]: Upload finished. err: <nil>
Jul 21 00:56:27.636645734 (50h5m34.009225089s) [upload=39989]: Starting upload
Jul 21 00:56:27.636705699 (50h5m34.009285054s) [upload=39989]: Done starting upload
Jul 21 00:56:27.636796402 (50h5m34.009375757s) [upload=39989]: Uploading segments...
Jul 21 00:56:27.636886103 (50h5m34.009465458s) [upload=39989]: Got next segment. Inline: false
Jul 21 00:56:27.636919067 (50h5m34.009498422s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 21 00:56:27.636923776 (50h5m34.009503131s) [upload=39989]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 21 00:56:28.204005298 (50h5m34.576584653s) [seg_pos={0 0}, upload=39989]: Begin upload segment...
Jul 21 00:56:28.204048731 (50h5m34.576628086s) [seg_pos={0 0}, upload=39989]: Done begin upload segment.
Jul 21 00:56:28.204070342 (50h5m34.576649697s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204109337 (50h5m34.576688702s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204436393 (50h5m34.577015738s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204688215 (50h5m34.57726756s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.204849283 (50h5m34.577428638s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205034387 (50h5m34.577613742s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205315285 (50h5m34.57789464s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205391610 (50h5m34.577970965s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205597615 (50h5m34.57817697s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205914050 (50h5m34.578493425s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.205988502 (50h5m34.578567857s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206209345 (50h5m34.5787887s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206353821 (50h5m34.578933176s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.206434055 (50h5m34.57901341s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.207809396 (50h5m34.580388741s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:28.208121774 (50h5m34.580701129s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Uploading piece...
Jul 21 00:56:30.032149002 (50h5m36.404728357s) [upload=39989]: Next returned nil segment
Jul 21 00:56:30.032170763 (50h5m36.404750118s) [upload=39989]: Waiting for error group managing segments...
Jul 21 00:56:30.342362436 (50h5m36.714941781s) [noise=true , address=104.36.82.8:4970     , piece_id=K6DJ6ASXU6ZYXJ7TUI6SYEEGAT7E3BULC47PM6JDRM3ZDAS7MCPQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:30.366992030 (50h5m36.739571375s) [noise=true , address=104.36.82.16:4295    , piece_id=6H4GQ4H72CI5VNSRQVMIQSD3ZU3TQMJZLKRFBPHUF4Q5Q6WAQMAQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:32.784102689 (50h5m39.156682044s) [noise=true , address=45.91.116.10:4082    , piece_id=LVAV5DTFVSFK5YIKNBEB6RYSRJWXU57SYSJQJKHDYCBYIXEAPBXA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.496717691 (50h5m40.869297046s) [noise=true , address=77.90.31.21:4877     , piece_id=C74EYJQWPNLEKC5JXVIGPQ6MYEEQXVUCO5IDJFHE37EERM3KJDQA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:34.604390252 (50h5m40.976969617s) [noise=true , address=77.90.31.17:4772     , piece_id=X5WBW2IH6PETUBC7KMY4NXLNYKSFXALD7TK2DN7QMW6DAFIUECBA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.064876482 (50h5m43.437455837s) [noise=true , address=77.90.31.19:4842     , piece_id=3RQ6DI7XZJY66FS26TTXXX7TJ2IRJEPHX3LXX6MKJ7UKRBOSF74Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:37.513875074 (50h5m43.886454429s) [noise=true , address=77.90.31.12:4589     , piece_id=5U25ORO35UYRA7CEIKYZPF3VGH6TK42PFW3G37E53QKSTHNTS6AA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:38.575486876 (50h5m44.948066231s) [noise=true , address=45.91.116.13:4154    , piece_id=KIVXBCZVMWSFL2GRCE4YZMTKPGD6PKZEU52UHXNHEMH7324SGF4Q, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.076664683 (50h5m45.449244088s) [noise=true , address=45.91.116.11:4119    , piece_id=AOQZJPHEX3YEZTBZ2J5HOIKI4S5Q5GQDUFO4AO2UDLN6TDXDR5RA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:39.503217481 (50h5m45.875796836s) [noise=true , address=45.91.116.9:4263     , piece_id=GF57RAN4FSINPFCFPRAA2LQLDIODCWDMFOAOSXKOTNUS2427OWTQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:40.994766806 (50h5m47.367346191s) [noise=true , address=45.91.116.7:4010     , piece_id=T6OEGJ24I6SYZC6IPGNZW6XFB5KONXDE2XUTKZXCSBFZ6VCIQKQQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: <nil>
Jul 21 00:56:41.826026852 (50h5m48.198606207s) [noise=true , address=77.90.31.18:4806     , piece_id=QGO562OFWGBM2Q5EBEARH337O5WUCBRI2LEBIZA2D34WTVGLUCNQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826111363 (50h5m48.198690718s) [noise=true , address=104.36.82.3:4765     , piece_id=NM77W7AVCSKG3DLZFTYSABT3JJTOTAQQ5TLF42XHBCF2PPBQ2JGQ, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 21 00:56:41.826125932 (50h5m48.198705287s) [noise=true , address=104.36.82.6:4875     , piece_id=GINC23262ZBV3XQZCFFRWAUSLFNQSUPJLBZ6ERKI5EE7TYOUY2KA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded
Jul 21 00:56:41.826192939 (50h5m48.198772405s) [noise=true , address=45.91.116.8:4048     , piece_id=GZIL4XBEWF3PJ4TNR66UHLQQDSDVSLFGJYOIYO6HZTCHSB2OUSEA, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: expected piece hash; context deadline exceeded; EOF
Jul 21 00:56:41.826204192 (50h5m48.198783547s) [noise=true , address=45.91.116.12:4227    , piece_id=VVL64AHJAORPUHUU2CSPQOPMD3G4CLGRPIBDANUP4XSYEGYMGN3A, seg_pos={0 0}, upload=39989]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51298->45.91.116.12:4227: use of closed network connection
Jul 21 00:56:41.826227535 (50h5m48.19880689s) [seg_pos={0 0}, upload=39989]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 21 00:56:42.066809032 (50h5m48.439388387s) [upload=39989]: Done uploading segments...
Jul 21 00:56:42.066835062 (50h5m48.439414407s) [upload=39989]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 6.148603014s core uplink us-select-1