Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
847.726194ms target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410
FAIL (2025-06-26 09:23:48 UTC)
orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: connect: no route to host
Duration: 847.757523ms
edge
gateway
vhost
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
718.26635ms target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410
FAIL (2025-06-26 09:23:02 UTC)
orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: connect: no route to host
Duration: 718.296517ms
edge
gateway
vhost
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
712.713124ms target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410
FAIL (2025-06-26 09:21:47 UTC)
orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: connect: no route to host
Duration: 712.745236ms
edge
gateway
vhost
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
720.075542ms target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410
FAIL (2025-06-26 09:20:45 UTC)
orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: connect: no route to host
Duration: 720.100791ms
edge
gateway
vhost
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
704.979576ms target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410
FAIL (2025-06-26 09:19:33 UTC)
orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: connect: no route to host
Duration: 705.0077ms
edge
gateway
vhost
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
705.897353ms target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410
FAIL (2025-06-26 09:19:13 UTC)
orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: connect: no route to host
Duration: 705.924646ms
edge
gateway
vhost
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
757.234218ms target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410
FAIL (2025-06-26 09:18:56 UTC)
orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: connect: no route to host
Duration: 757.261592ms
edge
gateway
vhost
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
237.187027ms target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410
FAIL (2025-06-26 09:17:47 UTC)
orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: connect: no route to host
Duration: 237.218899ms
edge
gateway
vhost
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
5.988161916s
FAIL (2025-06-26 07:00:03 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.988185853s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
5.854425612s
FAIL (2025-06-26 06:59:17 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.854449459s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
5.996236052s
FAIL (2025-06-26 06:58:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.996268324s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
6.075690649s
FAIL (2025-06-26 06:57:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.075708854s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
5.911628105s
FAIL (2025-06-26 06:56:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.911651171s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
5.545114232s
FAIL (2025-06-26 06:56:31 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
equestItem_ObjectBegin
Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment...
Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment.
Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment
Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments...
Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments...
Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 5.545140371s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
6.191998938s
FAIL (2025-06-26 06:56:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.192012254s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [XX..........................................................] duration:
5.635273737s
FAIL (2025-06-26 06:55:26 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
equestItem_ObjectBegin
Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment...
Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment.
Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment
Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments...
Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments...
Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 5.635299117s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
5.943537029s
FAIL (2025-06-26 06:55:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.943561216s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.XX.........................................................] duration:
5.861124046s
FAIL (2025-06-26 06:55:03 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
equestItem_ObjectBegin
Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment...
Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment.
Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment
Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments...
Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments...
Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 5.861154265s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
5.968665542s
FAIL (2025-06-26 06:54:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.968683686s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..XX........................................................] duration:
5.71095286s
FAIL (2025-06-26 06:54:14 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
equestItem_ObjectBegin
Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment...
Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment.
Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment
Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments...
Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments...
Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 5.710975856s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
5.92367452s
FAIL (2025-06-26 06:54:13 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.923707734s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
5.91216157s
FAIL (2025-06-26 06:53:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.912175396s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...XX.......................................................] duration:
5.627814011s
FAIL (2025-06-26 06:53:15 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
equestItem_ObjectBegin
Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment...
Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment.
Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment
Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments...
Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments...
Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 5.627848506s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [....XX......................................................] duration:
5.636692005s
FAIL (2025-06-26 06:52:56 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
equestItem_ObjectBegin
Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment...
Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment.
Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment
Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments...
Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments...
Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 5.636718696s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
5.921648223s
FAIL (2025-06-26 06:52:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.921670788s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....XX.....................................................] duration:
5.656904182s
FAIL (2025-06-26 06:52:02 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
equestItem_ObjectBegin
Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment...
Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment.
Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment
Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments...
Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments...
Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 5.656943337s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [......XX....................................................] duration:
5.641328553s
FAIL (2025-06-26 06:51:43 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
equestItem_ObjectBegin
Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment...
Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment.
Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment
Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments...
Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments...
Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 5.641353832s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
6.170311886s
FAIL (2025-06-26 06:51:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.170339269s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
5.9965031s
FAIL (2025-06-26 06:50:53 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.996524812s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......XX...................................................] duration:
5.734707035s
FAIL (2025-06-26 06:50:30 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
equestItem_ObjectBegin
Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment...
Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment.
Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment
Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments...
Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments...
Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 5.734729549s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
6.250452301s
FAIL (2025-06-26 06:49:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.250479493s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [........XX..................................................] duration:
5.633409817s
FAIL (2025-06-26 06:49:28 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
equestItem_ObjectBegin
Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment...
Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment.
Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment
Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments...
Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments...
Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 5.633443511s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
6.184541079s
FAIL (2025-06-26 06:49:10 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.184902739s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........XX.................................................] duration:
5.69472496s
FAIL (2025-06-26 06:48:52 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
equestItem_ObjectBegin
Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment...
Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment.
Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment
Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments...
Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments...
Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 5.694754938s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
6.143782895s
FAIL (2025-06-26 06:48:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.143808435s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........XX................................................] duration:
5.781382043s
FAIL (2025-06-26 06:47:53 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
equestItem_ObjectBegin
Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment...
Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment.
Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment
Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments...
Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments...
Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 5.781404687s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
6.11998763s
FAIL (2025-06-26 06:47:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.12000811s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...........XX...............................................] duration:
5.656878711s
FAIL (2025-06-26 06:46:46 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
equestItem_ObjectBegin
Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment...
Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment.
Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment
Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments...
Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments...
Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 5.656897447s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
6.066688051s
FAIL (2025-06-26 06:46:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.066705075s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [............XX..............................................] duration:
5.667499801s
FAIL (2025-06-26 06:46:14 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
equestItem_ObjectBegin
Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment...
Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment.
Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment
Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments...
Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments...
Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 5.667522906s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................] duration:
6.210053766s
FAIL (2025-06-26 06:45:43 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.210075467s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
5.920595857s
FAIL (2025-06-26 06:45:27 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.920616427s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............XX.............................................] duration:
5.830116097s
FAIL (2025-06-26 06:45:14 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
equestItem_ObjectBegin
Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment...
Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment.
Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment
Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments...
Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments...
Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 5.830135575s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..............XX............................................] duration:
5.614047138s
FAIL (2025-06-26 06:44:32 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
equestItem_ObjectBegin
Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment...
Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment.
Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment
Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments...
Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments...
Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 5.614066034s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
6.368577602s
FAIL (2025-06-26 06:44:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.368593332s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
5.674551727s
FAIL (2025-06-26 06:44:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.674576796s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...............XX...........................................] duration:
5.659928595s
FAIL (2025-06-26 06:43:58 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
equestItem_ObjectBegin
Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment...
Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment.
Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment
Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments...
Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments...
Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 5.65995648s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [................XX..........................................] duration:
5.782334526s
FAIL (2025-06-26 06:43:29 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
equestItem_ObjectBegin
Jun 26 06:08:25.210115727 (68h57m59.980757062s) [upload=55050]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 26 06:08:25.442516411 (68h58m0.213157746s) [seg_pos={0 0}, upload=55050]: Begin upload segment...
Jun 26 06:08:25.442566328 (68h58m0.213207663s) [seg_pos={0 0}, upload=55050]: Done begin upload segment.
Jun 26 06:08:25.442596487 (68h58m0.213237822s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.442637426 (68h58m0.213278751s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.442972394 (68h58m0.21361373s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443021710 (68h58m0.213663045s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443216437 (68h58m0.213857772s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443270753 (68h58m0.213912088s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443516920 (68h58m0.214158245s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443527560 (68h58m0.214168895s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443796711 (68h58m0.214438036s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.443918827 (68h58m0.214560162s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444032658 (68h58m0.214673983s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444169914 (68h58m0.214811249s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444237415 (68h58m0.21487875s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444370072 (68h58m0.215011407s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444626458 (68h58m0.215267794s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:25.444866233 (68h58m0.215507568s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Uploading piece...
Jun 26 06:08:27.054872821 (68h58m1.825514156s) [upload=55050]: Next returned nil segment
Jun 26 06:08:27.054890315 (68h58m1.82553165s) [upload=55050]: Waiting for error group managing segments...
Jun 26 06:08:27.351983973 (68h58m2.122625309s) [noise=true , address=104.36.82.12:4149 , piece_id=ZE3NN2YTGDIVOYRQGYMWC2ZZYGI7XZKXIJT72GGNIM7XYY4HUAZQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.374682705 (68h58m2.14532403s) [noise=true , address=104.36.82.5:4838 , piece_id=5ADR6Z7Z6EGK3PV4EX7QTFLXEQPVVRVI2RTMQUB3LNBNU4UOSUWA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.425374895 (68h58m2.19601623s) [noise=true , address=104.36.82.16:4295 , piece_id=HKWZAQSTL4IWHPZJ5EWZGSIFRWP3OTOG2FYUKZXDQEJ3JOFJDOJQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.430685108 (68h58m2.201326463s) [noise=true , address=104.36.82.12:4150 , piece_id=SNNNYATDSDBZQM7T3R3F4CJRJC5NGFFNWENTQCYRWIXFSUG7RP3Q, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.452240948 (68h58m2.222882283s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5ZXKPEVHQZEBEV45Q72WDEVLNAN5T2K2NVEJSYKNJBXLPDY4TAQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.474506019 (68h58m2.245147354s) [noise=true , address=104.36.82.6:4876 , piece_id=4QNHS22YK45HHYN4ALFZIQEYQHADRTSWBFDI7C72OLZQMKYYUF6A, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:27.479932890 (68h58m2.250574225s) [noise=true , address=104.36.82.15:4259 , piece_id=7AHX4WUJ5B6ZS3MVPL3GD6IK3BTX5LKEMYQF5YLH6UQGJ4YOAIIQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: <nil>
Jun 26 06:08:54.563692569 (68h58m29.334333904s) [noise=true , address=77.90.31.12:4592 , piece_id=COJJXQR5ET5HQJW24RZRDYEVCFKSY3RWKBQRPJF4EPHE52IFCMBA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563713720 (68h58m29.334355055s) [noise=true , address=77.90.31.11:4553 , piece_id=QZTKXHKIMPC22UPFNMLEB52KORWZN2C6CYXAEQBC64MSW2K6QGUA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:48346->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563747997 (68h58m29.334389332s) [noise=true , address=77.90.31.13:4627 , piece_id=HBW4YWRTQ66PTR7LUMQTEYEOQA7CW4UXOQQBBCETC6TZ6VP753DQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:43950->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563774217 (68h58m29.334415552s) [noise=true , address=45.91.116.7:4011 , piece_id=IYHPAOTZIPJLELYUOF6OJRLJOINUA5YWE7ZL774GL7PTEXHHCLOQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:48912->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563781672 (68h58m29.334422997s) [noise=true , address=45.91.116.12:5227 , piece_id=33DCWMSL33KSCX5QFEWT7RNRNMVMWPFXB4FYLDOINLVRK3FMEZFQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:5227): protocol: write tcp 188.166.225.170:54296->45.91.116.12:5227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563803544 (68h58m29.334444879s) [noise=true , address=45.91.116.10:4083 , piece_id=4TEGVR7HFKMJV2JFGKYPESM6YRY5W5W6DSHRIXI5XZU62E6FPNGQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48734->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563811880 (68h58m29.334453205s) [noise=true , address=45.91.116.13:4154 , piece_id=R4ZJM3VV5T2KYYY6G7F5XULEP2VQOHFJFE6KME2JKLHG6LJTOLNA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 188.166.225.170:58068->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563817461 (68h58m29.334458796s) [noise=true , address=77.90.31.16:4735 , piece_id=PLQVRJQWCEHLWIO546KJQK77V66RVB3G5SH32ZVX3REX3TDGKQZA, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:34792->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563833041 (68h58m29.334474366s) [noise=true , address=45.91.116.8:4048 , piece_id=ZCOWVVSF33TPJ47GUOJOODVMBMEERCZGYMNYSUSW7JVJ7GHDAVTQ, seg_pos={0 0}, upload=55050]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:49514->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jun 26 06:08:54.563850485 (68h58m29.33449182s) [seg_pos={0 0}, upload=55050]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Jun 26 06:08:54.790997193 (68h58m29.561638538s) [upload=55050]: Done uploading segments...
Jun 26 06:08:54.791025188 (68h58m29.561666553s) [upload=55050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 5.782361769s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
5.974143247s
FAIL (2025-06-26 06:43:27 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.974167463s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
5.716114325s
FAIL (2025-06-26 06:43:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.716131768s
core
uplink
us-select-1
us1-placement-20
|