Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [................X........X..................................] duration:
3.482179589s trace:
https://jaeger.dev.storj.io/trace/538482b67f052294
FAIL (2025-11-09 00:46:48 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.48220589s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
5.964709301s trace:
https://jaeger.dev.storj.io/trace/15a102436a900d91
FAIL (2025-11-09 00:46:42 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 4)
======LOGS======
pload=201217]: Uploading piece...
Nov 9 00:15:07.882409862 (228h34m39.578258615s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.882737906 (228h34m39.578586659s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883235645 (228h34m39.579084398s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883270351 (228h34m39.579119104s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883583637 (228h34m39.57943239s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883903466 (228h34m39.579752219s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:09.669212422 (228h34m41.365061175s) [upload=201217]: Next returned nil segment
Nov 9 00:15:09.669230136 (228h34m41.365078889s) [upload=201217]: Waiting for error group managing segments...
Nov 9 00:15:09.979393119 (228h34m41.675241862s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:09.982783635 (228h34m41.678632388s) [noise=true , address=45.91.116.9:4263 , piece_id=Z7MGGEQ63C5QOKI7BAAQCCJPJPLBM2MG2MBROD7GL4RRNCFVJJOQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:10.004031905 (228h34m41.699880668s) [noise=true , address=45.91.116.6:4973 , piece_id=2R4VUB2FG7IIY3KSBYSZFU7VTBGNS662DQ2AFOH7KDHNYSBGNUBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:10.072857382 (228h34m41.768706145s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:36.939056286 (228h35m8.634905039s) [noise=true , address=145.79.227.28:4726 , piece_id=EGLGRHWIZETHAYGPUIMSPEGOB3HXC2YMC32EGJIXSW7TQ4KGM2GA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection
Nov 9 00:15:36.939107635 (228h35m8.634956398s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection
Nov 9 00:15:36.939340538 (228h35m8.635189292s) [noise=true , address=145.79.227.10:4401 , piece_id=FKLG63DNHBF43ZT3MDH2D6GEDURMW6XA335EZND2LSQITYI5TYHQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection
Nov 9 00:15:36.939371116 (228h35m8.635219879s) [noise=true , address=145.79.227.15:4876 , piece_id=5XA3I6VEOLUYSQK374SW52H5LNMHEZX2HVOC6L6FFMAKGJV4RXTQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection
Nov 9 00:15:36.939473351 (228h35m8.635322104s) [noise=true , address=145.79.227.21:4473 , piece_id=3UNC3F6P7ERWHHZQTVSCIP4SBO6XL72I5F4ID5SOCIKTRFEWM6XQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection
Nov 9 00:15:36.939551320 (228h35m8.635400073s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: expected piece hash; context deadline exceeded; EOF
Nov 9 00:15:36.939605372 (228h35m8.635454125s) [noise=true , address=145.79.227.8:4330 , piece_id=M7O6MM4DRNV7Y63AGYACAKZ6674HNG777XVUGEAZ7XVEDIBBYTVQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection
Nov 9 00:15:36.939692238 (228h35m8.635540991s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection
Nov 9 00:15:36.939816855 (228h35m8.635665608s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection
Nov 9 00:15:36.939964185 (228h35m8.635812938s) [noise=true , address=145.79.227.9:4365 , piece_id=GYEVA6LDJIQJJ6AVSTAXAVD2JDQ6CBQERQJJMBTW5OV4EZMVTOKA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection
Nov 9 00:15:36.940093301 (228h35m8.635942054s) [noise=true , address=145.79.227.13:4804 , piece_id=MTPRWDDQQ5LMOTNCPOGKDH5IA57BICDAWSNOQ3H6BEBSZJSDOICQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection
Nov 9 00:15:36.940217358 (228h35m8.636066111s) [noise=true , address=145.79.227.25:4618 , piece_id=J7WJIE3QGIMDMAPSGEDMOUN23GX356DCB3LGFN7XJUNYK6CCLTMA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection
Nov 9 00:15:36.940262043 (228h35m8.636110786s) [seg_pos={0 0}, upload=201217]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil>
Nov 9 00:15:37.181912046 (228h35m8.877760799s) [upload=201217]: Done uploading segments...
Nov 9 00:15:37.181957073 (228h35m8.877805816s) [upload=201217]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4)
Duration: 5.964730473s
core
uplink
us-select-1
|
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)antiblip:
1 out of 15 is failed (threshold=6) [...........X...] duration:
603.119249ms remote:
"tcp:[2620:cb:2000::1]:443" target:
https://www.storj.io/dcs-satellites trace:
https://jaeger.dev.storj.io/trace/4f1ed0c21ac068b9
FAIL (2025-11-09 00:46:36 UTC)
orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:46250->[2620:cb:2000::1]:443: read: connection reset by peer
Duration: 603.143476ms
core
website
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.................X........X.................................] duration:
3.451763713s trace:
https://jaeger.dev.storj.io/trace/538482b67f052294
FAIL (2025-11-09 00:45:50 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.45178772s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)antiblip:
1 out of 15 is failed (threshold=6) [............X..] duration:
303.625348ms remote:
"tcp:[2620:cb:2000::1]:443" target:
https://www.storj.io/dcs-satellites trace:
https://jaeger.dev.storj.io/trace/4f1ed0c21ac068b9
FAIL (2025-11-09 00:45:49 UTC)
orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:46250->[2620:cb:2000::1]:443: read: connection reset by peer
Duration: 303.657058ms
core
website
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
6.097851355s trace:
https://jaeger.dev.storj.io/trace/15a102436a900d91
FAIL (2025-11-09 00:45:46 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 4)
======LOGS======
pload=201217]: Uploading piece...
Nov 9 00:15:07.882409862 (228h34m39.578258615s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.882737906 (228h34m39.578586659s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883235645 (228h34m39.579084398s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883270351 (228h34m39.579119104s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883583637 (228h34m39.57943239s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883903466 (228h34m39.579752219s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:09.669212422 (228h34m41.365061175s) [upload=201217]: Next returned nil segment
Nov 9 00:15:09.669230136 (228h34m41.365078889s) [upload=201217]: Waiting for error group managing segments...
Nov 9 00:15:09.979393119 (228h34m41.675241862s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:09.982783635 (228h34m41.678632388s) [noise=true , address=45.91.116.9:4263 , piece_id=Z7MGGEQ63C5QOKI7BAAQCCJPJPLBM2MG2MBROD7GL4RRNCFVJJOQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:10.004031905 (228h34m41.699880668s) [noise=true , address=45.91.116.6:4973 , piece_id=2R4VUB2FG7IIY3KSBYSZFU7VTBGNS662DQ2AFOH7KDHNYSBGNUBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:10.072857382 (228h34m41.768706145s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:36.939056286 (228h35m8.634905039s) [noise=true , address=145.79.227.28:4726 , piece_id=EGLGRHWIZETHAYGPUIMSPEGOB3HXC2YMC32EGJIXSW7TQ4KGM2GA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection
Nov 9 00:15:36.939107635 (228h35m8.634956398s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection
Nov 9 00:15:36.939340538 (228h35m8.635189292s) [noise=true , address=145.79.227.10:4401 , piece_id=FKLG63DNHBF43ZT3MDH2D6GEDURMW6XA335EZND2LSQITYI5TYHQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection
Nov 9 00:15:36.939371116 (228h35m8.635219879s) [noise=true , address=145.79.227.15:4876 , piece_id=5XA3I6VEOLUYSQK374SW52H5LNMHEZX2HVOC6L6FFMAKGJV4RXTQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection
Nov 9 00:15:36.939473351 (228h35m8.635322104s) [noise=true , address=145.79.227.21:4473 , piece_id=3UNC3F6P7ERWHHZQTVSCIP4SBO6XL72I5F4ID5SOCIKTRFEWM6XQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection
Nov 9 00:15:36.939551320 (228h35m8.635400073s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: expected piece hash; context deadline exceeded; EOF
Nov 9 00:15:36.939605372 (228h35m8.635454125s) [noise=true , address=145.79.227.8:4330 , piece_id=M7O6MM4DRNV7Y63AGYACAKZ6674HNG777XVUGEAZ7XVEDIBBYTVQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection
Nov 9 00:15:36.939692238 (228h35m8.635540991s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection
Nov 9 00:15:36.939816855 (228h35m8.635665608s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection
Nov 9 00:15:36.939964185 (228h35m8.635812938s) [noise=true , address=145.79.227.9:4365 , piece_id=GYEVA6LDJIQJJ6AVSTAXAVD2JDQ6CBQERQJJMBTW5OV4EZMVTOKA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection
Nov 9 00:15:36.940093301 (228h35m8.635942054s) [noise=true , address=145.79.227.13:4804 , piece_id=MTPRWDDQQ5LMOTNCPOGKDH5IA57BICDAWSNOQ3H6BEBSZJSDOICQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection
Nov 9 00:15:36.940217358 (228h35m8.636066111s) [noise=true , address=145.79.227.25:4618 , piece_id=J7WJIE3QGIMDMAPSGEDMOUN23GX356DCB3LGFN7XJUNYK6CCLTMA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection
Nov 9 00:15:36.940262043 (228h35m8.636110786s) [seg_pos={0 0}, upload=201217]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil>
Nov 9 00:15:37.181912046 (228h35m8.877760799s) [upload=201217]: Done uploading segments...
Nov 9 00:15:37.181957073 (228h35m8.877805816s) [upload=201217]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4)
Duration: 6.097885639s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [..................X........X................................] duration:
3.40906833s trace:
https://jaeger.dev.storj.io/trace/538482b67f052294
FAIL (2025-11-09 00:45:31 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.409098276s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................] duration:
5.893763303s trace:
https://jaeger.dev.storj.io/trace/15a102436a900d91
FAIL (2025-11-09 00:45:18 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 4)
======LOGS======
pload=201217]: Uploading piece...
Nov 9 00:15:07.882409862 (228h34m39.578258615s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.882737906 (228h34m39.578586659s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883235645 (228h34m39.579084398s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883270351 (228h34m39.579119104s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883583637 (228h34m39.57943239s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883903466 (228h34m39.579752219s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:09.669212422 (228h34m41.365061175s) [upload=201217]: Next returned nil segment
Nov 9 00:15:09.669230136 (228h34m41.365078889s) [upload=201217]: Waiting for error group managing segments...
Nov 9 00:15:09.979393119 (228h34m41.675241862s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:09.982783635 (228h34m41.678632388s) [noise=true , address=45.91.116.9:4263 , piece_id=Z7MGGEQ63C5QOKI7BAAQCCJPJPLBM2MG2MBROD7GL4RRNCFVJJOQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:10.004031905 (228h34m41.699880668s) [noise=true , address=45.91.116.6:4973 , piece_id=2R4VUB2FG7IIY3KSBYSZFU7VTBGNS662DQ2AFOH7KDHNYSBGNUBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:10.072857382 (228h34m41.768706145s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:36.939056286 (228h35m8.634905039s) [noise=true , address=145.79.227.28:4726 , piece_id=EGLGRHWIZETHAYGPUIMSPEGOB3HXC2YMC32EGJIXSW7TQ4KGM2GA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection
Nov 9 00:15:36.939107635 (228h35m8.634956398s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection
Nov 9 00:15:36.939340538 (228h35m8.635189292s) [noise=true , address=145.79.227.10:4401 , piece_id=FKLG63DNHBF43ZT3MDH2D6GEDURMW6XA335EZND2LSQITYI5TYHQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection
Nov 9 00:15:36.939371116 (228h35m8.635219879s) [noise=true , address=145.79.227.15:4876 , piece_id=5XA3I6VEOLUYSQK374SW52H5LNMHEZX2HVOC6L6FFMAKGJV4RXTQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection
Nov 9 00:15:36.939473351 (228h35m8.635322104s) [noise=true , address=145.79.227.21:4473 , piece_id=3UNC3F6P7ERWHHZQTVSCIP4SBO6XL72I5F4ID5SOCIKTRFEWM6XQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection
Nov 9 00:15:36.939551320 (228h35m8.635400073s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: expected piece hash; context deadline exceeded; EOF
Nov 9 00:15:36.939605372 (228h35m8.635454125s) [noise=true , address=145.79.227.8:4330 , piece_id=M7O6MM4DRNV7Y63AGYACAKZ6674HNG777XVUGEAZ7XVEDIBBYTVQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection
Nov 9 00:15:36.939692238 (228h35m8.635540991s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection
Nov 9 00:15:36.939816855 (228h35m8.635665608s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection
Nov 9 00:15:36.939964185 (228h35m8.635812938s) [noise=true , address=145.79.227.9:4365 , piece_id=GYEVA6LDJIQJJ6AVSTAXAVD2JDQ6CBQERQJJMBTW5OV4EZMVTOKA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection
Nov 9 00:15:36.940093301 (228h35m8.635942054s) [noise=true , address=145.79.227.13:4804 , piece_id=MTPRWDDQQ5LMOTNCPOGKDH5IA57BICDAWSNOQ3H6BEBSZJSDOICQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection
Nov 9 00:15:36.940217358 (228h35m8.636066111s) [noise=true , address=145.79.227.25:4618 , piece_id=J7WJIE3QGIMDMAPSGEDMOUN23GX356DCB3LGFN7XJUNYK6CCLTMA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection
Nov 9 00:15:36.940262043 (228h35m8.636110786s) [seg_pos={0 0}, upload=201217]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil>
Nov 9 00:15:37.181912046 (228h35m8.877760799s) [upload=201217]: Done uploading segments...
Nov 9 00:15:37.181957073 (228h35m8.877805816s) [upload=201217]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4)
Duration: 5.893795865s
core
uplink
us-select-1
|
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)antiblip:
1 out of 15 is failed (threshold=6) [.............X.] duration:
329.399646ms remote:
"tcp:[2620:cb:2000::1]:443" target:
https://www.storj.io/dcs-satellites trace:
https://jaeger.dev.storj.io/trace/4f1ed0c21ac068b9
FAIL (2025-11-09 00:45:16 UTC)
orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:46250->[2620:cb:2000::1]:443: read: connection reset by peer
Duration: 329.428472ms
core
website
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [...................X........X...............................] duration:
3.461986215s trace:
https://jaeger.dev.storj.io/trace/538482b67f052294
FAIL (2025-11-09 00:44:55 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.462012506s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
5.943986377s trace:
https://jaeger.dev.storj.io/trace/15a102436a900d91
FAIL (2025-11-09 00:44:40 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 4)
======LOGS======
pload=201217]: Uploading piece...
Nov 9 00:15:07.882409862 (228h34m39.578258615s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.882737906 (228h34m39.578586659s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883235645 (228h34m39.579084398s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883270351 (228h34m39.579119104s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883583637 (228h34m39.57943239s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883903466 (228h34m39.579752219s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:09.669212422 (228h34m41.365061175s) [upload=201217]: Next returned nil segment
Nov 9 00:15:09.669230136 (228h34m41.365078889s) [upload=201217]: Waiting for error group managing segments...
Nov 9 00:15:09.979393119 (228h34m41.675241862s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:09.982783635 (228h34m41.678632388s) [noise=true , address=45.91.116.9:4263 , piece_id=Z7MGGEQ63C5QOKI7BAAQCCJPJPLBM2MG2MBROD7GL4RRNCFVJJOQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:10.004031905 (228h34m41.699880668s) [noise=true , address=45.91.116.6:4973 , piece_id=2R4VUB2FG7IIY3KSBYSZFU7VTBGNS662DQ2AFOH7KDHNYSBGNUBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:10.072857382 (228h34m41.768706145s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:36.939056286 (228h35m8.634905039s) [noise=true , address=145.79.227.28:4726 , piece_id=EGLGRHWIZETHAYGPUIMSPEGOB3HXC2YMC32EGJIXSW7TQ4KGM2GA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection
Nov 9 00:15:36.939107635 (228h35m8.634956398s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection
Nov 9 00:15:36.939340538 (228h35m8.635189292s) [noise=true , address=145.79.227.10:4401 , piece_id=FKLG63DNHBF43ZT3MDH2D6GEDURMW6XA335EZND2LSQITYI5TYHQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection
Nov 9 00:15:36.939371116 (228h35m8.635219879s) [noise=true , address=145.79.227.15:4876 , piece_id=5XA3I6VEOLUYSQK374SW52H5LNMHEZX2HVOC6L6FFMAKGJV4RXTQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection
Nov 9 00:15:36.939473351 (228h35m8.635322104s) [noise=true , address=145.79.227.21:4473 , piece_id=3UNC3F6P7ERWHHZQTVSCIP4SBO6XL72I5F4ID5SOCIKTRFEWM6XQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection
Nov 9 00:15:36.939551320 (228h35m8.635400073s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: expected piece hash; context deadline exceeded; EOF
Nov 9 00:15:36.939605372 (228h35m8.635454125s) [noise=true , address=145.79.227.8:4330 , piece_id=M7O6MM4DRNV7Y63AGYACAKZ6674HNG777XVUGEAZ7XVEDIBBYTVQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection
Nov 9 00:15:36.939692238 (228h35m8.635540991s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection
Nov 9 00:15:36.939816855 (228h35m8.635665608s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection
Nov 9 00:15:36.939964185 (228h35m8.635812938s) [noise=true , address=145.79.227.9:4365 , piece_id=GYEVA6LDJIQJJ6AVSTAXAVD2JDQ6CBQERQJJMBTW5OV4EZMVTOKA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection
Nov 9 00:15:36.940093301 (228h35m8.635942054s) [noise=true , address=145.79.227.13:4804 , piece_id=MTPRWDDQQ5LMOTNCPOGKDH5IA57BICDAWSNOQ3H6BEBSZJSDOICQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection
Nov 9 00:15:36.940217358 (228h35m8.636066111s) [noise=true , address=145.79.227.25:4618 , piece_id=J7WJIE3QGIMDMAPSGEDMOUN23GX356DCB3LGFN7XJUNYK6CCLTMA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection
Nov 9 00:15:36.940262043 (228h35m8.636110786s) [seg_pos={0 0}, upload=201217]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil>
Nov 9 00:15:37.181912046 (228h35m8.877760799s) [upload=201217]: Done uploading segments...
Nov 9 00:15:37.181957073 (228h35m8.877805816s) [upload=201217]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4)
Duration: 5.94400878s
core
uplink
us-select-1
|
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)antiblip:
1 out of 15 is failed (threshold=6) [..............X] duration:
1.472692625s remote:
"tcp:[2620:cb:2000::1]:443" target:
https://www.storj.io/dcs-satellites trace:
https://jaeger.dev.storj.io/trace/4f1ed0c21ac068b9
FAIL (2025-11-09 00:44:39 UTC)
orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:46250->[2620:cb:2000::1]:443: read: connection reset by peer
Duration: 1.47272066s
core
website
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [....................X........X..............................] duration:
3.492903607s trace:
https://jaeger.dev.storj.io/trace/538482b67f052294
FAIL (2025-11-09 00:44:38 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.492924678s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
5.98993292s trace:
https://jaeger.dev.storj.io/trace/15a102436a900d91
FAIL (2025-11-09 00:44:23 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 4)
======LOGS======
pload=201217]: Uploading piece...
Nov 9 00:15:07.882409862 (228h34m39.578258615s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.882737906 (228h34m39.578586659s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883235645 (228h34m39.579084398s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883270351 (228h34m39.579119104s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883583637 (228h34m39.57943239s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883903466 (228h34m39.579752219s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:09.669212422 (228h34m41.365061175s) [upload=201217]: Next returned nil segment
Nov 9 00:15:09.669230136 (228h34m41.365078889s) [upload=201217]: Waiting for error group managing segments...
Nov 9 00:15:09.979393119 (228h34m41.675241862s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:09.982783635 (228h34m41.678632388s) [noise=true , address=45.91.116.9:4263 , piece_id=Z7MGGEQ63C5QOKI7BAAQCCJPJPLBM2MG2MBROD7GL4RRNCFVJJOQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:10.004031905 (228h34m41.699880668s) [noise=true , address=45.91.116.6:4973 , piece_id=2R4VUB2FG7IIY3KSBYSZFU7VTBGNS662DQ2AFOH7KDHNYSBGNUBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:10.072857382 (228h34m41.768706145s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:36.939056286 (228h35m8.634905039s) [noise=true , address=145.79.227.28:4726 , piece_id=EGLGRHWIZETHAYGPUIMSPEGOB3HXC2YMC32EGJIXSW7TQ4KGM2GA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection
Nov 9 00:15:36.939107635 (228h35m8.634956398s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection
Nov 9 00:15:36.939340538 (228h35m8.635189292s) [noise=true , address=145.79.227.10:4401 , piece_id=FKLG63DNHBF43ZT3MDH2D6GEDURMW6XA335EZND2LSQITYI5TYHQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection
Nov 9 00:15:36.939371116 (228h35m8.635219879s) [noise=true , address=145.79.227.15:4876 , piece_id=5XA3I6VEOLUYSQK374SW52H5LNMHEZX2HVOC6L6FFMAKGJV4RXTQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection
Nov 9 00:15:36.939473351 (228h35m8.635322104s) [noise=true , address=145.79.227.21:4473 , piece_id=3UNC3F6P7ERWHHZQTVSCIP4SBO6XL72I5F4ID5SOCIKTRFEWM6XQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection
Nov 9 00:15:36.939551320 (228h35m8.635400073s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: expected piece hash; context deadline exceeded; EOF
Nov 9 00:15:36.939605372 (228h35m8.635454125s) [noise=true , address=145.79.227.8:4330 , piece_id=M7O6MM4DRNV7Y63AGYACAKZ6674HNG777XVUGEAZ7XVEDIBBYTVQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection
Nov 9 00:15:36.939692238 (228h35m8.635540991s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection
Nov 9 00:15:36.939816855 (228h35m8.635665608s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection
Nov 9 00:15:36.939964185 (228h35m8.635812938s) [noise=true , address=145.79.227.9:4365 , piece_id=GYEVA6LDJIQJJ6AVSTAXAVD2JDQ6CBQERQJJMBTW5OV4EZMVTOKA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection
Nov 9 00:15:36.940093301 (228h35m8.635942054s) [noise=true , address=145.79.227.13:4804 , piece_id=MTPRWDDQQ5LMOTNCPOGKDH5IA57BICDAWSNOQ3H6BEBSZJSDOICQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection
Nov 9 00:15:36.940217358 (228h35m8.636066111s) [noise=true , address=145.79.227.25:4618 , piece_id=J7WJIE3QGIMDMAPSGEDMOUN23GX356DCB3LGFN7XJUNYK6CCLTMA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection
Nov 9 00:15:36.940262043 (228h35m8.636110786s) [seg_pos={0 0}, upload=201217]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil>
Nov 9 00:15:37.181912046 (228h35m8.877760799s) [upload=201217]: Done uploading segments...
Nov 9 00:15:37.181957073 (228h35m8.877805816s) [upload=201217]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4)
Duration: 5.989964582s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.....................X........X.............................] duration:
3.476757031s trace:
https://jaeger.dev.storj.io/trace/538482b67f052294
FAIL (2025-11-09 00:43:46 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.476789292s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
5.901093903s trace:
https://jaeger.dev.storj.io/trace/15a102436a900d91
FAIL (2025-11-09 00:43:31 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 4)
======LOGS======
pload=201217]: Uploading piece...
Nov 9 00:15:07.882409862 (228h34m39.578258615s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.882737906 (228h34m39.578586659s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883235645 (228h34m39.579084398s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883270351 (228h34m39.579119104s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883583637 (228h34m39.57943239s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883903466 (228h34m39.579752219s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:09.669212422 (228h34m41.365061175s) [upload=201217]: Next returned nil segment
Nov 9 00:15:09.669230136 (228h34m41.365078889s) [upload=201217]: Waiting for error group managing segments...
Nov 9 00:15:09.979393119 (228h34m41.675241862s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:09.982783635 (228h34m41.678632388s) [noise=true , address=45.91.116.9:4263 , piece_id=Z7MGGEQ63C5QOKI7BAAQCCJPJPLBM2MG2MBROD7GL4RRNCFVJJOQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:10.004031905 (228h34m41.699880668s) [noise=true , address=45.91.116.6:4973 , piece_id=2R4VUB2FG7IIY3KSBYSZFU7VTBGNS662DQ2AFOH7KDHNYSBGNUBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:10.072857382 (228h34m41.768706145s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:36.939056286 (228h35m8.634905039s) [noise=true , address=145.79.227.28:4726 , piece_id=EGLGRHWIZETHAYGPUIMSPEGOB3HXC2YMC32EGJIXSW7TQ4KGM2GA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection
Nov 9 00:15:36.939107635 (228h35m8.634956398s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection
Nov 9 00:15:36.939340538 (228h35m8.635189292s) [noise=true , address=145.79.227.10:4401 , piece_id=FKLG63DNHBF43ZT3MDH2D6GEDURMW6XA335EZND2LSQITYI5TYHQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection
Nov 9 00:15:36.939371116 (228h35m8.635219879s) [noise=true , address=145.79.227.15:4876 , piece_id=5XA3I6VEOLUYSQK374SW52H5LNMHEZX2HVOC6L6FFMAKGJV4RXTQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection
Nov 9 00:15:36.939473351 (228h35m8.635322104s) [noise=true , address=145.79.227.21:4473 , piece_id=3UNC3F6P7ERWHHZQTVSCIP4SBO6XL72I5F4ID5SOCIKTRFEWM6XQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection
Nov 9 00:15:36.939551320 (228h35m8.635400073s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: expected piece hash; context deadline exceeded; EOF
Nov 9 00:15:36.939605372 (228h35m8.635454125s) [noise=true , address=145.79.227.8:4330 , piece_id=M7O6MM4DRNV7Y63AGYACAKZ6674HNG777XVUGEAZ7XVEDIBBYTVQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection
Nov 9 00:15:36.939692238 (228h35m8.635540991s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection
Nov 9 00:15:36.939816855 (228h35m8.635665608s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection
Nov 9 00:15:36.939964185 (228h35m8.635812938s) [noise=true , address=145.79.227.9:4365 , piece_id=GYEVA6LDJIQJJ6AVSTAXAVD2JDQ6CBQERQJJMBTW5OV4EZMVTOKA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection
Nov 9 00:15:36.940093301 (228h35m8.635942054s) [noise=true , address=145.79.227.13:4804 , piece_id=MTPRWDDQQ5LMOTNCPOGKDH5IA57BICDAWSNOQ3H6BEBSZJSDOICQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection
Nov 9 00:15:36.940217358 (228h35m8.636066111s) [noise=true , address=145.79.227.25:4618 , piece_id=J7WJIE3QGIMDMAPSGEDMOUN23GX356DCB3LGFN7XJUNYK6CCLTMA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection
Nov 9 00:15:36.940262043 (228h35m8.636110786s) [seg_pos={0 0}, upload=201217]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil>
Nov 9 00:15:37.181912046 (228h35m8.877760799s) [upload=201217]: Done uploading segments...
Nov 9 00:15:37.181957073 (228h35m8.877805816s) [upload=201217]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4)
Duration: 5.901124661s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [......................X........X............................] duration:
4.390528428s trace:
https://jaeger.dev.storj.io/trace/538482b67f052294
FAIL (2025-11-09 00:43:11 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.390614613s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.......................X........X...........................] duration:
3.37642046s trace:
https://jaeger.dev.storj.io/trace/538482b67f052294
FAIL (2025-11-09 00:42:39 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.376465616s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
6.057472346s trace:
https://jaeger.dev.storj.io/trace/15a102436a900d91
FAIL (2025-11-09 00:42:26 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 4)
======LOGS======
pload=201217]: Uploading piece...
Nov 9 00:15:07.882409862 (228h34m39.578258615s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.882737906 (228h34m39.578586659s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883235645 (228h34m39.579084398s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883270351 (228h34m39.579119104s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883583637 (228h34m39.57943239s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883903466 (228h34m39.579752219s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:09.669212422 (228h34m41.365061175s) [upload=201217]: Next returned nil segment
Nov 9 00:15:09.669230136 (228h34m41.365078889s) [upload=201217]: Waiting for error group managing segments...
Nov 9 00:15:09.979393119 (228h34m41.675241862s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:09.982783635 (228h34m41.678632388s) [noise=true , address=45.91.116.9:4263 , piece_id=Z7MGGEQ63C5QOKI7BAAQCCJPJPLBM2MG2MBROD7GL4RRNCFVJJOQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:10.004031905 (228h34m41.699880668s) [noise=true , address=45.91.116.6:4973 , piece_id=2R4VUB2FG7IIY3KSBYSZFU7VTBGNS662DQ2AFOH7KDHNYSBGNUBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:10.072857382 (228h34m41.768706145s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:36.939056286 (228h35m8.634905039s) [noise=true , address=145.79.227.28:4726 , piece_id=EGLGRHWIZETHAYGPUIMSPEGOB3HXC2YMC32EGJIXSW7TQ4KGM2GA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection
Nov 9 00:15:36.939107635 (228h35m8.634956398s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection
Nov 9 00:15:36.939340538 (228h35m8.635189292s) [noise=true , address=145.79.227.10:4401 , piece_id=FKLG63DNHBF43ZT3MDH2D6GEDURMW6XA335EZND2LSQITYI5TYHQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection
Nov 9 00:15:36.939371116 (228h35m8.635219879s) [noise=true , address=145.79.227.15:4876 , piece_id=5XA3I6VEOLUYSQK374SW52H5LNMHEZX2HVOC6L6FFMAKGJV4RXTQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection
Nov 9 00:15:36.939473351 (228h35m8.635322104s) [noise=true , address=145.79.227.21:4473 , piece_id=3UNC3F6P7ERWHHZQTVSCIP4SBO6XL72I5F4ID5SOCIKTRFEWM6XQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection
Nov 9 00:15:36.939551320 (228h35m8.635400073s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: expected piece hash; context deadline exceeded; EOF
Nov 9 00:15:36.939605372 (228h35m8.635454125s) [noise=true , address=145.79.227.8:4330 , piece_id=M7O6MM4DRNV7Y63AGYACAKZ6674HNG777XVUGEAZ7XVEDIBBYTVQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection
Nov 9 00:15:36.939692238 (228h35m8.635540991s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection
Nov 9 00:15:36.939816855 (228h35m8.635665608s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection
Nov 9 00:15:36.939964185 (228h35m8.635812938s) [noise=true , address=145.79.227.9:4365 , piece_id=GYEVA6LDJIQJJ6AVSTAXAVD2JDQ6CBQERQJJMBTW5OV4EZMVTOKA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection
Nov 9 00:15:36.940093301 (228h35m8.635942054s) [noise=true , address=145.79.227.13:4804 , piece_id=MTPRWDDQQ5LMOTNCPOGKDH5IA57BICDAWSNOQ3H6BEBSZJSDOICQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection
Nov 9 00:15:36.940217358 (228h35m8.636066111s) [noise=true , address=145.79.227.25:4618 , piece_id=J7WJIE3QGIMDMAPSGEDMOUN23GX356DCB3LGFN7XJUNYK6CCLTMA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection
Nov 9 00:15:36.940262043 (228h35m8.636110786s) [seg_pos={0 0}, upload=201217]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil>
Nov 9 00:15:37.181912046 (228h35m8.877760799s) [upload=201217]: Done uploading segments...
Nov 9 00:15:37.181957073 (228h35m8.877805816s) [upload=201217]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4)
Duration: 6.057497534s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
6.034517686s trace:
https://jaeger.dev.storj.io/trace/15a102436a900d91
FAIL (2025-11-09 00:41:47 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 4)
======LOGS======
pload=201217]: Uploading piece...
Nov 9 00:15:07.882409862 (228h34m39.578258615s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.882737906 (228h34m39.578586659s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883235645 (228h34m39.579084398s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883270351 (228h34m39.579119104s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883583637 (228h34m39.57943239s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883903466 (228h34m39.579752219s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:09.669212422 (228h34m41.365061175s) [upload=201217]: Next returned nil segment
Nov 9 00:15:09.669230136 (228h34m41.365078889s) [upload=201217]: Waiting for error group managing segments...
Nov 9 00:15:09.979393119 (228h34m41.675241862s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:09.982783635 (228h34m41.678632388s) [noise=true , address=45.91.116.9:4263 , piece_id=Z7MGGEQ63C5QOKI7BAAQCCJPJPLBM2MG2MBROD7GL4RRNCFVJJOQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:10.004031905 (228h34m41.699880668s) [noise=true , address=45.91.116.6:4973 , piece_id=2R4VUB2FG7IIY3KSBYSZFU7VTBGNS662DQ2AFOH7KDHNYSBGNUBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:10.072857382 (228h34m41.768706145s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:36.939056286 (228h35m8.634905039s) [noise=true , address=145.79.227.28:4726 , piece_id=EGLGRHWIZETHAYGPUIMSPEGOB3HXC2YMC32EGJIXSW7TQ4KGM2GA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection
Nov 9 00:15:36.939107635 (228h35m8.634956398s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection
Nov 9 00:15:36.939340538 (228h35m8.635189292s) [noise=true , address=145.79.227.10:4401 , piece_id=FKLG63DNHBF43ZT3MDH2D6GEDURMW6XA335EZND2LSQITYI5TYHQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection
Nov 9 00:15:36.939371116 (228h35m8.635219879s) [noise=true , address=145.79.227.15:4876 , piece_id=5XA3I6VEOLUYSQK374SW52H5LNMHEZX2HVOC6L6FFMAKGJV4RXTQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection
Nov 9 00:15:36.939473351 (228h35m8.635322104s) [noise=true , address=145.79.227.21:4473 , piece_id=3UNC3F6P7ERWHHZQTVSCIP4SBO6XL72I5F4ID5SOCIKTRFEWM6XQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection
Nov 9 00:15:36.939551320 (228h35m8.635400073s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: expected piece hash; context deadline exceeded; EOF
Nov 9 00:15:36.939605372 (228h35m8.635454125s) [noise=true , address=145.79.227.8:4330 , piece_id=M7O6MM4DRNV7Y63AGYACAKZ6674HNG777XVUGEAZ7XVEDIBBYTVQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection
Nov 9 00:15:36.939692238 (228h35m8.635540991s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection
Nov 9 00:15:36.939816855 (228h35m8.635665608s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection
Nov 9 00:15:36.939964185 (228h35m8.635812938s) [noise=true , address=145.79.227.9:4365 , piece_id=GYEVA6LDJIQJJ6AVSTAXAVD2JDQ6CBQERQJJMBTW5OV4EZMVTOKA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection
Nov 9 00:15:36.940093301 (228h35m8.635942054s) [noise=true , address=145.79.227.13:4804 , piece_id=MTPRWDDQQ5LMOTNCPOGKDH5IA57BICDAWSNOQ3H6BEBSZJSDOICQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection
Nov 9 00:15:36.940217358 (228h35m8.636066111s) [noise=true , address=145.79.227.25:4618 , piece_id=J7WJIE3QGIMDMAPSGEDMOUN23GX356DCB3LGFN7XJUNYK6CCLTMA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection
Nov 9 00:15:36.940262043 (228h35m8.636110786s) [seg_pos={0 0}, upload=201217]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil>
Nov 9 00:15:37.181912046 (228h35m8.877760799s) [upload=201217]: Done uploading segments...
Nov 9 00:15:37.181957073 (228h35m8.877805816s) [upload=201217]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4)
Duration: 6.034547552s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [........................X........X..........................] duration:
3.458161832s trace:
https://jaeger.dev.storj.io/trace/538482b67f052294
FAIL (2025-11-09 00:41:47 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.458210616s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
5.841343327s trace:
https://jaeger.dev.storj.io/trace/15a102436a900d91
FAIL (2025-11-09 00:40:58 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 4)
======LOGS======
pload=201217]: Uploading piece...
Nov 9 00:15:07.882409862 (228h34m39.578258615s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.882737906 (228h34m39.578586659s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883235645 (228h34m39.579084398s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883270351 (228h34m39.579119104s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883583637 (228h34m39.57943239s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883903466 (228h34m39.579752219s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:09.669212422 (228h34m41.365061175s) [upload=201217]: Next returned nil segment
Nov 9 00:15:09.669230136 (228h34m41.365078889s) [upload=201217]: Waiting for error group managing segments...
Nov 9 00:15:09.979393119 (228h34m41.675241862s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:09.982783635 (228h34m41.678632388s) [noise=true , address=45.91.116.9:4263 , piece_id=Z7MGGEQ63C5QOKI7BAAQCCJPJPLBM2MG2MBROD7GL4RRNCFVJJOQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:10.004031905 (228h34m41.699880668s) [noise=true , address=45.91.116.6:4973 , piece_id=2R4VUB2FG7IIY3KSBYSZFU7VTBGNS662DQ2AFOH7KDHNYSBGNUBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:10.072857382 (228h34m41.768706145s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:36.939056286 (228h35m8.634905039s) [noise=true , address=145.79.227.28:4726 , piece_id=EGLGRHWIZETHAYGPUIMSPEGOB3HXC2YMC32EGJIXSW7TQ4KGM2GA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection
Nov 9 00:15:36.939107635 (228h35m8.634956398s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection
Nov 9 00:15:36.939340538 (228h35m8.635189292s) [noise=true , address=145.79.227.10:4401 , piece_id=FKLG63DNHBF43ZT3MDH2D6GEDURMW6XA335EZND2LSQITYI5TYHQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection
Nov 9 00:15:36.939371116 (228h35m8.635219879s) [noise=true , address=145.79.227.15:4876 , piece_id=5XA3I6VEOLUYSQK374SW52H5LNMHEZX2HVOC6L6FFMAKGJV4RXTQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection
Nov 9 00:15:36.939473351 (228h35m8.635322104s) [noise=true , address=145.79.227.21:4473 , piece_id=3UNC3F6P7ERWHHZQTVSCIP4SBO6XL72I5F4ID5SOCIKTRFEWM6XQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection
Nov 9 00:15:36.939551320 (228h35m8.635400073s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: expected piece hash; context deadline exceeded; EOF
Nov 9 00:15:36.939605372 (228h35m8.635454125s) [noise=true , address=145.79.227.8:4330 , piece_id=M7O6MM4DRNV7Y63AGYACAKZ6674HNG777XVUGEAZ7XVEDIBBYTVQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection
Nov 9 00:15:36.939692238 (228h35m8.635540991s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection
Nov 9 00:15:36.939816855 (228h35m8.635665608s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection
Nov 9 00:15:36.939964185 (228h35m8.635812938s) [noise=true , address=145.79.227.9:4365 , piece_id=GYEVA6LDJIQJJ6AVSTAXAVD2JDQ6CBQERQJJMBTW5OV4EZMVTOKA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection
Nov 9 00:15:36.940093301 (228h35m8.635942054s) [noise=true , address=145.79.227.13:4804 , piece_id=MTPRWDDQQ5LMOTNCPOGKDH5IA57BICDAWSNOQ3H6BEBSZJSDOICQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection
Nov 9 00:15:36.940217358 (228h35m8.636066111s) [noise=true , address=145.79.227.25:4618 , piece_id=J7WJIE3QGIMDMAPSGEDMOUN23GX356DCB3LGFN7XJUNYK6CCLTMA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection
Nov 9 00:15:36.940262043 (228h35m8.636110786s) [seg_pos={0 0}, upload=201217]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil>
Nov 9 00:15:37.181912046 (228h35m8.877760799s) [upload=201217]: Done uploading segments...
Nov 9 00:15:37.181957073 (228h35m8.877805816s) [upload=201217]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4)
Duration: 5.841375149s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.........................X........X.........................] duration:
3.473126659s trace:
https://jaeger.dev.storj.io/trace/538482b67f052294
FAIL (2025-11-09 00:40:41 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.47315292s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [..........................X........X........................] duration:
3.425686704s trace:
https://jaeger.dev.storj.io/trace/538482b67f052294
FAIL (2025-11-09 00:40:11 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.42572678s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................] duration:
6.111758008s trace:
https://jaeger.dev.storj.io/trace/15a102436a900d91
FAIL (2025-11-09 00:40:02 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 4)
======LOGS======
pload=201217]: Uploading piece...
Nov 9 00:15:07.882409862 (228h34m39.578258615s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.882737906 (228h34m39.578586659s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883235645 (228h34m39.579084398s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883270351 (228h34m39.579119104s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883583637 (228h34m39.57943239s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883903466 (228h34m39.579752219s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:09.669212422 (228h34m41.365061175s) [upload=201217]: Next returned nil segment
Nov 9 00:15:09.669230136 (228h34m41.365078889s) [upload=201217]: Waiting for error group managing segments...
Nov 9 00:15:09.979393119 (228h34m41.675241862s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:09.982783635 (228h34m41.678632388s) [noise=true , address=45.91.116.9:4263 , piece_id=Z7MGGEQ63C5QOKI7BAAQCCJPJPLBM2MG2MBROD7GL4RRNCFVJJOQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:10.004031905 (228h34m41.699880668s) [noise=true , address=45.91.116.6:4973 , piece_id=2R4VUB2FG7IIY3KSBYSZFU7VTBGNS662DQ2AFOH7KDHNYSBGNUBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:10.072857382 (228h34m41.768706145s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:36.939056286 (228h35m8.634905039s) [noise=true , address=145.79.227.28:4726 , piece_id=EGLGRHWIZETHAYGPUIMSPEGOB3HXC2YMC32EGJIXSW7TQ4KGM2GA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection
Nov 9 00:15:36.939107635 (228h35m8.634956398s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection
Nov 9 00:15:36.939340538 (228h35m8.635189292s) [noise=true , address=145.79.227.10:4401 , piece_id=FKLG63DNHBF43ZT3MDH2D6GEDURMW6XA335EZND2LSQITYI5TYHQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection
Nov 9 00:15:36.939371116 (228h35m8.635219879s) [noise=true , address=145.79.227.15:4876 , piece_id=5XA3I6VEOLUYSQK374SW52H5LNMHEZX2HVOC6L6FFMAKGJV4RXTQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection
Nov 9 00:15:36.939473351 (228h35m8.635322104s) [noise=true , address=145.79.227.21:4473 , piece_id=3UNC3F6P7ERWHHZQTVSCIP4SBO6XL72I5F4ID5SOCIKTRFEWM6XQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection
Nov 9 00:15:36.939551320 (228h35m8.635400073s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: expected piece hash; context deadline exceeded; EOF
Nov 9 00:15:36.939605372 (228h35m8.635454125s) [noise=true , address=145.79.227.8:4330 , piece_id=M7O6MM4DRNV7Y63AGYACAKZ6674HNG777XVUGEAZ7XVEDIBBYTVQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection
Nov 9 00:15:36.939692238 (228h35m8.635540991s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection
Nov 9 00:15:36.939816855 (228h35m8.635665608s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection
Nov 9 00:15:36.939964185 (228h35m8.635812938s) [noise=true , address=145.79.227.9:4365 , piece_id=GYEVA6LDJIQJJ6AVSTAXAVD2JDQ6CBQERQJJMBTW5OV4EZMVTOKA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection
Nov 9 00:15:36.940093301 (228h35m8.635942054s) [noise=true , address=145.79.227.13:4804 , piece_id=MTPRWDDQQ5LMOTNCPOGKDH5IA57BICDAWSNOQ3H6BEBSZJSDOICQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection
Nov 9 00:15:36.940217358 (228h35m8.636066111s) [noise=true , address=145.79.227.25:4618 , piece_id=J7WJIE3QGIMDMAPSGEDMOUN23GX356DCB3LGFN7XJUNYK6CCLTMA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection
Nov 9 00:15:36.940262043 (228h35m8.636110786s) [seg_pos={0 0}, upload=201217]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil>
Nov 9 00:15:37.181912046 (228h35m8.877760799s) [upload=201217]: Done uploading segments...
Nov 9 00:15:37.181957073 (228h35m8.877805816s) [upload=201217]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4)
Duration: 6.111830475s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [...........................X........X.......................] duration:
3.41951983s trace:
https://jaeger.dev.storj.io/trace/538482b67f052294
FAIL (2025-11-09 00:39:15 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.419576859s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................] duration:
6.08201527s trace:
https://jaeger.dev.storj.io/trace/15a102436a900d91
FAIL (2025-11-09 00:38:50 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 4)
======LOGS======
pload=201217]: Uploading piece...
Nov 9 00:15:07.882409862 (228h34m39.578258615s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.882737906 (228h34m39.578586659s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883235645 (228h34m39.579084398s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883270351 (228h34m39.579119104s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883583637 (228h34m39.57943239s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883903466 (228h34m39.579752219s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:09.669212422 (228h34m41.365061175s) [upload=201217]: Next returned nil segment
Nov 9 00:15:09.669230136 (228h34m41.365078889s) [upload=201217]: Waiting for error group managing segments...
Nov 9 00:15:09.979393119 (228h34m41.675241862s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:09.982783635 (228h34m41.678632388s) [noise=true , address=45.91.116.9:4263 , piece_id=Z7MGGEQ63C5QOKI7BAAQCCJPJPLBM2MG2MBROD7GL4RRNCFVJJOQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:10.004031905 (228h34m41.699880668s) [noise=true , address=45.91.116.6:4973 , piece_id=2R4VUB2FG7IIY3KSBYSZFU7VTBGNS662DQ2AFOH7KDHNYSBGNUBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:10.072857382 (228h34m41.768706145s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:36.939056286 (228h35m8.634905039s) [noise=true , address=145.79.227.28:4726 , piece_id=EGLGRHWIZETHAYGPUIMSPEGOB3HXC2YMC32EGJIXSW7TQ4KGM2GA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection
Nov 9 00:15:36.939107635 (228h35m8.634956398s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection
Nov 9 00:15:36.939340538 (228h35m8.635189292s) [noise=true , address=145.79.227.10:4401 , piece_id=FKLG63DNHBF43ZT3MDH2D6GEDURMW6XA335EZND2LSQITYI5TYHQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection
Nov 9 00:15:36.939371116 (228h35m8.635219879s) [noise=true , address=145.79.227.15:4876 , piece_id=5XA3I6VEOLUYSQK374SW52H5LNMHEZX2HVOC6L6FFMAKGJV4RXTQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection
Nov 9 00:15:36.939473351 (228h35m8.635322104s) [noise=true , address=145.79.227.21:4473 , piece_id=3UNC3F6P7ERWHHZQTVSCIP4SBO6XL72I5F4ID5SOCIKTRFEWM6XQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection
Nov 9 00:15:36.939551320 (228h35m8.635400073s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: expected piece hash; context deadline exceeded; EOF
Nov 9 00:15:36.939605372 (228h35m8.635454125s) [noise=true , address=145.79.227.8:4330 , piece_id=M7O6MM4DRNV7Y63AGYACAKZ6674HNG777XVUGEAZ7XVEDIBBYTVQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection
Nov 9 00:15:36.939692238 (228h35m8.635540991s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection
Nov 9 00:15:36.939816855 (228h35m8.635665608s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection
Nov 9 00:15:36.939964185 (228h35m8.635812938s) [noise=true , address=145.79.227.9:4365 , piece_id=GYEVA6LDJIQJJ6AVSTAXAVD2JDQ6CBQERQJJMBTW5OV4EZMVTOKA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection
Nov 9 00:15:36.940093301 (228h35m8.635942054s) [noise=true , address=145.79.227.13:4804 , piece_id=MTPRWDDQQ5LMOTNCPOGKDH5IA57BICDAWSNOQ3H6BEBSZJSDOICQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection
Nov 9 00:15:36.940217358 (228h35m8.636066111s) [noise=true , address=145.79.227.25:4618 , piece_id=J7WJIE3QGIMDMAPSGEDMOUN23GX356DCB3LGFN7XJUNYK6CCLTMA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection
Nov 9 00:15:36.940262043 (228h35m8.636110786s) [seg_pos={0 0}, upload=201217]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil>
Nov 9 00:15:37.181912046 (228h35m8.877760799s) [upload=201217]: Done uploading segments...
Nov 9 00:15:37.181957073 (228h35m8.877805816s) [upload=201217]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4)
Duration: 6.082033805s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [............................X........X......................] duration:
3.535275201s trace:
https://jaeger.dev.storj.io/trace/538482b67f052294
FAIL (2025-11-09 00:38:10 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.535309406s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................] duration:
5.881893835s trace:
https://jaeger.dev.storj.io/trace/15a102436a900d91
FAIL (2025-11-09 00:37:53 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 4)
======LOGS======
pload=201217]: Uploading piece...
Nov 9 00:15:07.882409862 (228h34m39.578258615s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.882737906 (228h34m39.578586659s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883235645 (228h34m39.579084398s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883270351 (228h34m39.579119104s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883583637 (228h34m39.57943239s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883903466 (228h34m39.579752219s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:09.669212422 (228h34m41.365061175s) [upload=201217]: Next returned nil segment
Nov 9 00:15:09.669230136 (228h34m41.365078889s) [upload=201217]: Waiting for error group managing segments...
Nov 9 00:15:09.979393119 (228h34m41.675241862s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:09.982783635 (228h34m41.678632388s) [noise=true , address=45.91.116.9:4263 , piece_id=Z7MGGEQ63C5QOKI7BAAQCCJPJPLBM2MG2MBROD7GL4RRNCFVJJOQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:10.004031905 (228h34m41.699880668s) [noise=true , address=45.91.116.6:4973 , piece_id=2R4VUB2FG7IIY3KSBYSZFU7VTBGNS662DQ2AFOH7KDHNYSBGNUBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:10.072857382 (228h34m41.768706145s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:36.939056286 (228h35m8.634905039s) [noise=true , address=145.79.227.28:4726 , piece_id=EGLGRHWIZETHAYGPUIMSPEGOB3HXC2YMC32EGJIXSW7TQ4KGM2GA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection
Nov 9 00:15:36.939107635 (228h35m8.634956398s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection
Nov 9 00:15:36.939340538 (228h35m8.635189292s) [noise=true , address=145.79.227.10:4401 , piece_id=FKLG63DNHBF43ZT3MDH2D6GEDURMW6XA335EZND2LSQITYI5TYHQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection
Nov 9 00:15:36.939371116 (228h35m8.635219879s) [noise=true , address=145.79.227.15:4876 , piece_id=5XA3I6VEOLUYSQK374SW52H5LNMHEZX2HVOC6L6FFMAKGJV4RXTQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection
Nov 9 00:15:36.939473351 (228h35m8.635322104s) [noise=true , address=145.79.227.21:4473 , piece_id=3UNC3F6P7ERWHHZQTVSCIP4SBO6XL72I5F4ID5SOCIKTRFEWM6XQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection
Nov 9 00:15:36.939551320 (228h35m8.635400073s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: expected piece hash; context deadline exceeded; EOF
Nov 9 00:15:36.939605372 (228h35m8.635454125s) [noise=true , address=145.79.227.8:4330 , piece_id=M7O6MM4DRNV7Y63AGYACAKZ6674HNG777XVUGEAZ7XVEDIBBYTVQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection
Nov 9 00:15:36.939692238 (228h35m8.635540991s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection
Nov 9 00:15:36.939816855 (228h35m8.635665608s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection
Nov 9 00:15:36.939964185 (228h35m8.635812938s) [noise=true , address=145.79.227.9:4365 , piece_id=GYEVA6LDJIQJJ6AVSTAXAVD2JDQ6CBQERQJJMBTW5OV4EZMVTOKA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection
Nov 9 00:15:36.940093301 (228h35m8.635942054s) [noise=true , address=145.79.227.13:4804 , piece_id=MTPRWDDQQ5LMOTNCPOGKDH5IA57BICDAWSNOQ3H6BEBSZJSDOICQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection
Nov 9 00:15:36.940217358 (228h35m8.636066111s) [noise=true , address=145.79.227.25:4618 , piece_id=J7WJIE3QGIMDMAPSGEDMOUN23GX356DCB3LGFN7XJUNYK6CCLTMA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection
Nov 9 00:15:36.940262043 (228h35m8.636110786s) [seg_pos={0 0}, upload=201217]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil>
Nov 9 00:15:37.181912046 (228h35m8.877760799s) [upload=201217]: Done uploading segments...
Nov 9 00:15:37.181957073 (228h35m8.877805816s) [upload=201217]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4)
Duration: 5.881923411s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.............................X........X.....................] duration:
3.396486782s trace:
https://jaeger.dev.storj.io/trace/538482b67f052294
FAIL (2025-11-09 00:37:48 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.396518121s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [..............................X........X....................] duration:
3.465332657s trace:
https://jaeger.dev.storj.io/trace/538482b67f052294
FAIL (2025-11-09 00:37:28 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.465363526s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
6.126417189s trace:
https://jaeger.dev.storj.io/trace/15a102436a900d91
FAIL (2025-11-09 00:37:10 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 4)
======LOGS======
pload=201217]: Uploading piece...
Nov 9 00:15:07.882409862 (228h34m39.578258615s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.882737906 (228h34m39.578586659s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883235645 (228h34m39.579084398s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883270351 (228h34m39.579119104s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883583637 (228h34m39.57943239s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883903466 (228h34m39.579752219s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:09.669212422 (228h34m41.365061175s) [upload=201217]: Next returned nil segment
Nov 9 00:15:09.669230136 (228h34m41.365078889s) [upload=201217]: Waiting for error group managing segments...
Nov 9 00:15:09.979393119 (228h34m41.675241862s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:09.982783635 (228h34m41.678632388s) [noise=true , address=45.91.116.9:4263 , piece_id=Z7MGGEQ63C5QOKI7BAAQCCJPJPLBM2MG2MBROD7GL4RRNCFVJJOQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:10.004031905 (228h34m41.699880668s) [noise=true , address=45.91.116.6:4973 , piece_id=2R4VUB2FG7IIY3KSBYSZFU7VTBGNS662DQ2AFOH7KDHNYSBGNUBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:10.072857382 (228h34m41.768706145s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:36.939056286 (228h35m8.634905039s) [noise=true , address=145.79.227.28:4726 , piece_id=EGLGRHWIZETHAYGPUIMSPEGOB3HXC2YMC32EGJIXSW7TQ4KGM2GA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection
Nov 9 00:15:36.939107635 (228h35m8.634956398s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection
Nov 9 00:15:36.939340538 (228h35m8.635189292s) [noise=true , address=145.79.227.10:4401 , piece_id=FKLG63DNHBF43ZT3MDH2D6GEDURMW6XA335EZND2LSQITYI5TYHQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection
Nov 9 00:15:36.939371116 (228h35m8.635219879s) [noise=true , address=145.79.227.15:4876 , piece_id=5XA3I6VEOLUYSQK374SW52H5LNMHEZX2HVOC6L6FFMAKGJV4RXTQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection
Nov 9 00:15:36.939473351 (228h35m8.635322104s) [noise=true , address=145.79.227.21:4473 , piece_id=3UNC3F6P7ERWHHZQTVSCIP4SBO6XL72I5F4ID5SOCIKTRFEWM6XQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection
Nov 9 00:15:36.939551320 (228h35m8.635400073s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: expected piece hash; context deadline exceeded; EOF
Nov 9 00:15:36.939605372 (228h35m8.635454125s) [noise=true , address=145.79.227.8:4330 , piece_id=M7O6MM4DRNV7Y63AGYACAKZ6674HNG777XVUGEAZ7XVEDIBBYTVQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection
Nov 9 00:15:36.939692238 (228h35m8.635540991s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection
Nov 9 00:15:36.939816855 (228h35m8.635665608s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection
Nov 9 00:15:36.939964185 (228h35m8.635812938s) [noise=true , address=145.79.227.9:4365 , piece_id=GYEVA6LDJIQJJ6AVSTAXAVD2JDQ6CBQERQJJMBTW5OV4EZMVTOKA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection
Nov 9 00:15:36.940093301 (228h35m8.635942054s) [noise=true , address=145.79.227.13:4804 , piece_id=MTPRWDDQQ5LMOTNCPOGKDH5IA57BICDAWSNOQ3H6BEBSZJSDOICQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection
Nov 9 00:15:36.940217358 (228h35m8.636066111s) [noise=true , address=145.79.227.25:4618 , piece_id=J7WJIE3QGIMDMAPSGEDMOUN23GX356DCB3LGFN7XJUNYK6CCLTMA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection
Nov 9 00:15:36.940262043 (228h35m8.636110786s) [seg_pos={0 0}, upload=201217]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil>
Nov 9 00:15:37.181912046 (228h35m8.877760799s) [upload=201217]: Done uploading segments...
Nov 9 00:15:37.181957073 (228h35m8.877805816s) [upload=201217]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4)
Duration: 6.126450644s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [...............................X........X...................] duration:
3.469204689s trace:
https://jaeger.dev.storj.io/trace/538482b67f052294
FAIL (2025-11-09 00:36:34 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.469232493s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
5.936721043s trace:
https://jaeger.dev.storj.io/trace/15a102436a900d91
FAIL (2025-11-09 00:36:07 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 4)
======LOGS======
pload=201217]: Uploading piece...
Nov 9 00:15:07.882409862 (228h34m39.578258615s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.882737906 (228h34m39.578586659s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883235645 (228h34m39.579084398s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883270351 (228h34m39.579119104s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883583637 (228h34m39.57943239s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883903466 (228h34m39.579752219s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:09.669212422 (228h34m41.365061175s) [upload=201217]: Next returned nil segment
Nov 9 00:15:09.669230136 (228h34m41.365078889s) [upload=201217]: Waiting for error group managing segments...
Nov 9 00:15:09.979393119 (228h34m41.675241862s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:09.982783635 (228h34m41.678632388s) [noise=true , address=45.91.116.9:4263 , piece_id=Z7MGGEQ63C5QOKI7BAAQCCJPJPLBM2MG2MBROD7GL4RRNCFVJJOQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:10.004031905 (228h34m41.699880668s) [noise=true , address=45.91.116.6:4973 , piece_id=2R4VUB2FG7IIY3KSBYSZFU7VTBGNS662DQ2AFOH7KDHNYSBGNUBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:10.072857382 (228h34m41.768706145s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:36.939056286 (228h35m8.634905039s) [noise=true , address=145.79.227.28:4726 , piece_id=EGLGRHWIZETHAYGPUIMSPEGOB3HXC2YMC32EGJIXSW7TQ4KGM2GA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection
Nov 9 00:15:36.939107635 (228h35m8.634956398s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection
Nov 9 00:15:36.939340538 (228h35m8.635189292s) [noise=true , address=145.79.227.10:4401 , piece_id=FKLG63DNHBF43ZT3MDH2D6GEDURMW6XA335EZND2LSQITYI5TYHQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection
Nov 9 00:15:36.939371116 (228h35m8.635219879s) [noise=true , address=145.79.227.15:4876 , piece_id=5XA3I6VEOLUYSQK374SW52H5LNMHEZX2HVOC6L6FFMAKGJV4RXTQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection
Nov 9 00:15:36.939473351 (228h35m8.635322104s) [noise=true , address=145.79.227.21:4473 , piece_id=3UNC3F6P7ERWHHZQTVSCIP4SBO6XL72I5F4ID5SOCIKTRFEWM6XQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection
Nov 9 00:15:36.939551320 (228h35m8.635400073s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: expected piece hash; context deadline exceeded; EOF
Nov 9 00:15:36.939605372 (228h35m8.635454125s) [noise=true , address=145.79.227.8:4330 , piece_id=M7O6MM4DRNV7Y63AGYACAKZ6674HNG777XVUGEAZ7XVEDIBBYTVQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection
Nov 9 00:15:36.939692238 (228h35m8.635540991s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection
Nov 9 00:15:36.939816855 (228h35m8.635665608s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection
Nov 9 00:15:36.939964185 (228h35m8.635812938s) [noise=true , address=145.79.227.9:4365 , piece_id=GYEVA6LDJIQJJ6AVSTAXAVD2JDQ6CBQERQJJMBTW5OV4EZMVTOKA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection
Nov 9 00:15:36.940093301 (228h35m8.635942054s) [noise=true , address=145.79.227.13:4804 , piece_id=MTPRWDDQQ5LMOTNCPOGKDH5IA57BICDAWSNOQ3H6BEBSZJSDOICQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection
Nov 9 00:15:36.940217358 (228h35m8.636066111s) [noise=true , address=145.79.227.25:4618 , piece_id=J7WJIE3QGIMDMAPSGEDMOUN23GX356DCB3LGFN7XJUNYK6CCLTMA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection
Nov 9 00:15:36.940262043 (228h35m8.636110786s) [seg_pos={0 0}, upload=201217]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil>
Nov 9 00:15:37.181912046 (228h35m8.877760799s) [upload=201217]: Done uploading segments...
Nov 9 00:15:37.181957073 (228h35m8.877805816s) [upload=201217]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4)
Duration: 5.93675024s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [................................X........X..................] duration:
3.393370411s trace:
https://jaeger.dev.storj.io/trace/538482b67f052294
FAIL (2025-11-09 00:35:59 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.393397603s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.................................X........X.................] duration:
3.538125741s trace:
https://jaeger.dev.storj.io/trace/538482b67f052294
FAIL (2025-11-09 00:35:37 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.53816806s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
5.953646285s trace:
https://jaeger.dev.storj.io/trace/15a102436a900d91
FAIL (2025-11-09 00:35:15 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 4)
======LOGS======
pload=201217]: Uploading piece...
Nov 9 00:15:07.882409862 (228h34m39.578258615s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.882737906 (228h34m39.578586659s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883235645 (228h34m39.579084398s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883270351 (228h34m39.579119104s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883583637 (228h34m39.57943239s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883903466 (228h34m39.579752219s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:09.669212422 (228h34m41.365061175s) [upload=201217]: Next returned nil segment
Nov 9 00:15:09.669230136 (228h34m41.365078889s) [upload=201217]: Waiting for error group managing segments...
Nov 9 00:15:09.979393119 (228h34m41.675241862s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:09.982783635 (228h34m41.678632388s) [noise=true , address=45.91.116.9:4263 , piece_id=Z7MGGEQ63C5QOKI7BAAQCCJPJPLBM2MG2MBROD7GL4RRNCFVJJOQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:10.004031905 (228h34m41.699880668s) [noise=true , address=45.91.116.6:4973 , piece_id=2R4VUB2FG7IIY3KSBYSZFU7VTBGNS662DQ2AFOH7KDHNYSBGNUBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:10.072857382 (228h34m41.768706145s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:36.939056286 (228h35m8.634905039s) [noise=true , address=145.79.227.28:4726 , piece_id=EGLGRHWIZETHAYGPUIMSPEGOB3HXC2YMC32EGJIXSW7TQ4KGM2GA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection
Nov 9 00:15:36.939107635 (228h35m8.634956398s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection
Nov 9 00:15:36.939340538 (228h35m8.635189292s) [noise=true , address=145.79.227.10:4401 , piece_id=FKLG63DNHBF43ZT3MDH2D6GEDURMW6XA335EZND2LSQITYI5TYHQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection
Nov 9 00:15:36.939371116 (228h35m8.635219879s) [noise=true , address=145.79.227.15:4876 , piece_id=5XA3I6VEOLUYSQK374SW52H5LNMHEZX2HVOC6L6FFMAKGJV4RXTQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection
Nov 9 00:15:36.939473351 (228h35m8.635322104s) [noise=true , address=145.79.227.21:4473 , piece_id=3UNC3F6P7ERWHHZQTVSCIP4SBO6XL72I5F4ID5SOCIKTRFEWM6XQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection
Nov 9 00:15:36.939551320 (228h35m8.635400073s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: expected piece hash; context deadline exceeded; EOF
Nov 9 00:15:36.939605372 (228h35m8.635454125s) [noise=true , address=145.79.227.8:4330 , piece_id=M7O6MM4DRNV7Y63AGYACAKZ6674HNG777XVUGEAZ7XVEDIBBYTVQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection
Nov 9 00:15:36.939692238 (228h35m8.635540991s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection
Nov 9 00:15:36.939816855 (228h35m8.635665608s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection
Nov 9 00:15:36.939964185 (228h35m8.635812938s) [noise=true , address=145.79.227.9:4365 , piece_id=GYEVA6LDJIQJJ6AVSTAXAVD2JDQ6CBQERQJJMBTW5OV4EZMVTOKA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection
Nov 9 00:15:36.940093301 (228h35m8.635942054s) [noise=true , address=145.79.227.13:4804 , piece_id=MTPRWDDQQ5LMOTNCPOGKDH5IA57BICDAWSNOQ3H6BEBSZJSDOICQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection
Nov 9 00:15:36.940217358 (228h35m8.636066111s) [noise=true , address=145.79.227.25:4618 , piece_id=J7WJIE3QGIMDMAPSGEDMOUN23GX356DCB3LGFN7XJUNYK6CCLTMA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection
Nov 9 00:15:36.940262043 (228h35m8.636110786s) [seg_pos={0 0}, upload=201217]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil>
Nov 9 00:15:37.181912046 (228h35m8.877760799s) [upload=201217]: Done uploading segments...
Nov 9 00:15:37.181957073 (228h35m8.877805816s) [upload=201217]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4)
Duration: 5.953678918s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [..................................X........X................] duration:
3.458201364s trace:
https://jaeger.dev.storj.io/trace/538482b67f052294
FAIL (2025-11-09 00:34:35 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.45833573s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [...................................X........X...............] duration:
3.459568789s trace:
https://jaeger.dev.storj.io/trace/538482b67f052294
FAIL (2025-11-09 00:34:19 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.459614776s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
6.197215817s trace:
https://jaeger.dev.storj.io/trace/15a102436a900d91
FAIL (2025-11-09 00:34:07 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 4)
======LOGS======
pload=201217]: Uploading piece...
Nov 9 00:15:07.882409862 (228h34m39.578258615s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.882737906 (228h34m39.578586659s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883235645 (228h34m39.579084398s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883270351 (228h34m39.579119104s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883583637 (228h34m39.57943239s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883903466 (228h34m39.579752219s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:09.669212422 (228h34m41.365061175s) [upload=201217]: Next returned nil segment
Nov 9 00:15:09.669230136 (228h34m41.365078889s) [upload=201217]: Waiting for error group managing segments...
Nov 9 00:15:09.979393119 (228h34m41.675241862s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:09.982783635 (228h34m41.678632388s) [noise=true , address=45.91.116.9:4263 , piece_id=Z7MGGEQ63C5QOKI7BAAQCCJPJPLBM2MG2MBROD7GL4RRNCFVJJOQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:10.004031905 (228h34m41.699880668s) [noise=true , address=45.91.116.6:4973 , piece_id=2R4VUB2FG7IIY3KSBYSZFU7VTBGNS662DQ2AFOH7KDHNYSBGNUBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:10.072857382 (228h34m41.768706145s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:36.939056286 (228h35m8.634905039s) [noise=true , address=145.79.227.28:4726 , piece_id=EGLGRHWIZETHAYGPUIMSPEGOB3HXC2YMC32EGJIXSW7TQ4KGM2GA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection
Nov 9 00:15:36.939107635 (228h35m8.634956398s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection
Nov 9 00:15:36.939340538 (228h35m8.635189292s) [noise=true , address=145.79.227.10:4401 , piece_id=FKLG63DNHBF43ZT3MDH2D6GEDURMW6XA335EZND2LSQITYI5TYHQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection
Nov 9 00:15:36.939371116 (228h35m8.635219879s) [noise=true , address=145.79.227.15:4876 , piece_id=5XA3I6VEOLUYSQK374SW52H5LNMHEZX2HVOC6L6FFMAKGJV4RXTQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection
Nov 9 00:15:36.939473351 (228h35m8.635322104s) [noise=true , address=145.79.227.21:4473 , piece_id=3UNC3F6P7ERWHHZQTVSCIP4SBO6XL72I5F4ID5SOCIKTRFEWM6XQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection
Nov 9 00:15:36.939551320 (228h35m8.635400073s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: expected piece hash; context deadline exceeded; EOF
Nov 9 00:15:36.939605372 (228h35m8.635454125s) [noise=true , address=145.79.227.8:4330 , piece_id=M7O6MM4DRNV7Y63AGYACAKZ6674HNG777XVUGEAZ7XVEDIBBYTVQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection
Nov 9 00:15:36.939692238 (228h35m8.635540991s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection
Nov 9 00:15:36.939816855 (228h35m8.635665608s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection
Nov 9 00:15:36.939964185 (228h35m8.635812938s) [noise=true , address=145.79.227.9:4365 , piece_id=GYEVA6LDJIQJJ6AVSTAXAVD2JDQ6CBQERQJJMBTW5OV4EZMVTOKA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection
Nov 9 00:15:36.940093301 (228h35m8.635942054s) [noise=true , address=145.79.227.13:4804 , piece_id=MTPRWDDQQ5LMOTNCPOGKDH5IA57BICDAWSNOQ3H6BEBSZJSDOICQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection
Nov 9 00:15:36.940217358 (228h35m8.636066111s) [noise=true , address=145.79.227.25:4618 , piece_id=J7WJIE3QGIMDMAPSGEDMOUN23GX356DCB3LGFN7XJUNYK6CCLTMA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection
Nov 9 00:15:36.940262043 (228h35m8.636110786s) [seg_pos={0 0}, upload=201217]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil>
Nov 9 00:15:37.181912046 (228h35m8.877760799s) [upload=201217]: Done uploading segments...
Nov 9 00:15:37.181957073 (228h35m8.877805816s) [upload=201217]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4)
Duration: 6.19724322s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
6.024563135s trace:
https://jaeger.dev.storj.io/trace/15a102436a900d91
FAIL (2025-11-09 00:33:50 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 4)
======LOGS======
pload=201217]: Uploading piece...
Nov 9 00:15:07.882409862 (228h34m39.578258615s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.882737906 (228h34m39.578586659s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883235645 (228h34m39.579084398s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883270351 (228h34m39.579119104s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883583637 (228h34m39.57943239s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883903466 (228h34m39.579752219s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:09.669212422 (228h34m41.365061175s) [upload=201217]: Next returned nil segment
Nov 9 00:15:09.669230136 (228h34m41.365078889s) [upload=201217]: Waiting for error group managing segments...
Nov 9 00:15:09.979393119 (228h34m41.675241862s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:09.982783635 (228h34m41.678632388s) [noise=true , address=45.91.116.9:4263 , piece_id=Z7MGGEQ63C5QOKI7BAAQCCJPJPLBM2MG2MBROD7GL4RRNCFVJJOQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:10.004031905 (228h34m41.699880668s) [noise=true , address=45.91.116.6:4973 , piece_id=2R4VUB2FG7IIY3KSBYSZFU7VTBGNS662DQ2AFOH7KDHNYSBGNUBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:10.072857382 (228h34m41.768706145s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:36.939056286 (228h35m8.634905039s) [noise=true , address=145.79.227.28:4726 , piece_id=EGLGRHWIZETHAYGPUIMSPEGOB3HXC2YMC32EGJIXSW7TQ4KGM2GA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection
Nov 9 00:15:36.939107635 (228h35m8.634956398s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection
Nov 9 00:15:36.939340538 (228h35m8.635189292s) [noise=true , address=145.79.227.10:4401 , piece_id=FKLG63DNHBF43ZT3MDH2D6GEDURMW6XA335EZND2LSQITYI5TYHQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection
Nov 9 00:15:36.939371116 (228h35m8.635219879s) [noise=true , address=145.79.227.15:4876 , piece_id=5XA3I6VEOLUYSQK374SW52H5LNMHEZX2HVOC6L6FFMAKGJV4RXTQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection
Nov 9 00:15:36.939473351 (228h35m8.635322104s) [noise=true , address=145.79.227.21:4473 , piece_id=3UNC3F6P7ERWHHZQTVSCIP4SBO6XL72I5F4ID5SOCIKTRFEWM6XQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection
Nov 9 00:15:36.939551320 (228h35m8.635400073s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: expected piece hash; context deadline exceeded; EOF
Nov 9 00:15:36.939605372 (228h35m8.635454125s) [noise=true , address=145.79.227.8:4330 , piece_id=M7O6MM4DRNV7Y63AGYACAKZ6674HNG777XVUGEAZ7XVEDIBBYTVQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection
Nov 9 00:15:36.939692238 (228h35m8.635540991s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection
Nov 9 00:15:36.939816855 (228h35m8.635665608s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection
Nov 9 00:15:36.939964185 (228h35m8.635812938s) [noise=true , address=145.79.227.9:4365 , piece_id=GYEVA6LDJIQJJ6AVSTAXAVD2JDQ6CBQERQJJMBTW5OV4EZMVTOKA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection
Nov 9 00:15:36.940093301 (228h35m8.635942054s) [noise=true , address=145.79.227.13:4804 , piece_id=MTPRWDDQQ5LMOTNCPOGKDH5IA57BICDAWSNOQ3H6BEBSZJSDOICQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection
Nov 9 00:15:36.940217358 (228h35m8.636066111s) [noise=true , address=145.79.227.25:4618 , piece_id=J7WJIE3QGIMDMAPSGEDMOUN23GX356DCB3LGFN7XJUNYK6CCLTMA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection
Nov 9 00:15:36.940262043 (228h35m8.636110786s) [seg_pos={0 0}, upload=201217]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil>
Nov 9 00:15:37.181912046 (228h35m8.877760799s) [upload=201217]: Done uploading segments...
Nov 9 00:15:37.181957073 (228h35m8.877805816s) [upload=201217]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4)
Duration: 6.024589736s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [....................................X........X..............] duration:
3.390560093s trace:
https://jaeger.dev.storj.io/trace/538482b67f052294
FAIL (2025-11-09 00:33:48 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.390661887s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.....................................X........X.............] duration:
3.494287286s trace:
https://jaeger.dev.storj.io/trace/538482b67f052294
FAIL (2025-11-09 00:33:33 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.494312264s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
5.887312558s trace:
https://jaeger.dev.storj.io/trace/15a102436a900d91
FAIL (2025-11-09 00:32:46 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 4)
======LOGS======
pload=201217]: Uploading piece...
Nov 9 00:15:07.882409862 (228h34m39.578258615s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.882737906 (228h34m39.578586659s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883235645 (228h34m39.579084398s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883270351 (228h34m39.579119104s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883583637 (228h34m39.57943239s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883903466 (228h34m39.579752219s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:09.669212422 (228h34m41.365061175s) [upload=201217]: Next returned nil segment
Nov 9 00:15:09.669230136 (228h34m41.365078889s) [upload=201217]: Waiting for error group managing segments...
Nov 9 00:15:09.979393119 (228h34m41.675241862s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:09.982783635 (228h34m41.678632388s) [noise=true , address=45.91.116.9:4263 , piece_id=Z7MGGEQ63C5QOKI7BAAQCCJPJPLBM2MG2MBROD7GL4RRNCFVJJOQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:10.004031905 (228h34m41.699880668s) [noise=true , address=45.91.116.6:4973 , piece_id=2R4VUB2FG7IIY3KSBYSZFU7VTBGNS662DQ2AFOH7KDHNYSBGNUBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:10.072857382 (228h34m41.768706145s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:36.939056286 (228h35m8.634905039s) [noise=true , address=145.79.227.28:4726 , piece_id=EGLGRHWIZETHAYGPUIMSPEGOB3HXC2YMC32EGJIXSW7TQ4KGM2GA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection
Nov 9 00:15:36.939107635 (228h35m8.634956398s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection
Nov 9 00:15:36.939340538 (228h35m8.635189292s) [noise=true , address=145.79.227.10:4401 , piece_id=FKLG63DNHBF43ZT3MDH2D6GEDURMW6XA335EZND2LSQITYI5TYHQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection
Nov 9 00:15:36.939371116 (228h35m8.635219879s) [noise=true , address=145.79.227.15:4876 , piece_id=5XA3I6VEOLUYSQK374SW52H5LNMHEZX2HVOC6L6FFMAKGJV4RXTQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection
Nov 9 00:15:36.939473351 (228h35m8.635322104s) [noise=true , address=145.79.227.21:4473 , piece_id=3UNC3F6P7ERWHHZQTVSCIP4SBO6XL72I5F4ID5SOCIKTRFEWM6XQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection
Nov 9 00:15:36.939551320 (228h35m8.635400073s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: expected piece hash; context deadline exceeded; EOF
Nov 9 00:15:36.939605372 (228h35m8.635454125s) [noise=true , address=145.79.227.8:4330 , piece_id=M7O6MM4DRNV7Y63AGYACAKZ6674HNG777XVUGEAZ7XVEDIBBYTVQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection
Nov 9 00:15:36.939692238 (228h35m8.635540991s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection
Nov 9 00:15:36.939816855 (228h35m8.635665608s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection
Nov 9 00:15:36.939964185 (228h35m8.635812938s) [noise=true , address=145.79.227.9:4365 , piece_id=GYEVA6LDJIQJJ6AVSTAXAVD2JDQ6CBQERQJJMBTW5OV4EZMVTOKA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection
Nov 9 00:15:36.940093301 (228h35m8.635942054s) [noise=true , address=145.79.227.13:4804 , piece_id=MTPRWDDQQ5LMOTNCPOGKDH5IA57BICDAWSNOQ3H6BEBSZJSDOICQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection
Nov 9 00:15:36.940217358 (228h35m8.636066111s) [noise=true , address=145.79.227.25:4618 , piece_id=J7WJIE3QGIMDMAPSGEDMOUN23GX356DCB3LGFN7XJUNYK6CCLTMA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection
Nov 9 00:15:36.940262043 (228h35m8.636110786s) [seg_pos={0 0}, upload=201217]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil>
Nov 9 00:15:37.181912046 (228h35m8.877760799s) [upload=201217]: Done uploading segments...
Nov 9 00:15:37.181957073 (228h35m8.877805816s) [upload=201217]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4)
Duration: 5.887339169s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [......................................X........X............] duration:
3.446184793s trace:
https://jaeger.dev.storj.io/trace/538482b67f052294
FAIL (2025-11-09 00:32:40 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.446275206s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.......................................X........X...........] duration:
3.45789532s trace:
https://jaeger.dev.storj.io/trace/538482b67f052294
FAIL (2025-11-09 00:32:05 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.457920447s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
5.898934508s trace:
https://jaeger.dev.storj.io/trace/15a102436a900d91
FAIL (2025-11-09 00:31:35 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 4)
======LOGS======
pload=201217]: Uploading piece...
Nov 9 00:15:07.882409862 (228h34m39.578258615s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.882737906 (228h34m39.578586659s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883235645 (228h34m39.579084398s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883270351 (228h34m39.579119104s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883583637 (228h34m39.57943239s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883903466 (228h34m39.579752219s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:09.669212422 (228h34m41.365061175s) [upload=201217]: Next returned nil segment
Nov 9 00:15:09.669230136 (228h34m41.365078889s) [upload=201217]: Waiting for error group managing segments...
Nov 9 00:15:09.979393119 (228h34m41.675241862s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:09.982783635 (228h34m41.678632388s) [noise=true , address=45.91.116.9:4263 , piece_id=Z7MGGEQ63C5QOKI7BAAQCCJPJPLBM2MG2MBROD7GL4RRNCFVJJOQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:10.004031905 (228h34m41.699880668s) [noise=true , address=45.91.116.6:4973 , piece_id=2R4VUB2FG7IIY3KSBYSZFU7VTBGNS662DQ2AFOH7KDHNYSBGNUBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:10.072857382 (228h34m41.768706145s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:36.939056286 (228h35m8.634905039s) [noise=true , address=145.79.227.28:4726 , piece_id=EGLGRHWIZETHAYGPUIMSPEGOB3HXC2YMC32EGJIXSW7TQ4KGM2GA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection
Nov 9 00:15:36.939107635 (228h35m8.634956398s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection
Nov 9 00:15:36.939340538 (228h35m8.635189292s) [noise=true , address=145.79.227.10:4401 , piece_id=FKLG63DNHBF43ZT3MDH2D6GEDURMW6XA335EZND2LSQITYI5TYHQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection
Nov 9 00:15:36.939371116 (228h35m8.635219879s) [noise=true , address=145.79.227.15:4876 , piece_id=5XA3I6VEOLUYSQK374SW52H5LNMHEZX2HVOC6L6FFMAKGJV4RXTQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection
Nov 9 00:15:36.939473351 (228h35m8.635322104s) [noise=true , address=145.79.227.21:4473 , piece_id=3UNC3F6P7ERWHHZQTVSCIP4SBO6XL72I5F4ID5SOCIKTRFEWM6XQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection
Nov 9 00:15:36.939551320 (228h35m8.635400073s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: expected piece hash; context deadline exceeded; EOF
Nov 9 00:15:36.939605372 (228h35m8.635454125s) [noise=true , address=145.79.227.8:4330 , piece_id=M7O6MM4DRNV7Y63AGYACAKZ6674HNG777XVUGEAZ7XVEDIBBYTVQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection
Nov 9 00:15:36.939692238 (228h35m8.635540991s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection
Nov 9 00:15:36.939816855 (228h35m8.635665608s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection
Nov 9 00:15:36.939964185 (228h35m8.635812938s) [noise=true , address=145.79.227.9:4365 , piece_id=GYEVA6LDJIQJJ6AVSTAXAVD2JDQ6CBQERQJJMBTW5OV4EZMVTOKA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection
Nov 9 00:15:36.940093301 (228h35m8.635942054s) [noise=true , address=145.79.227.13:4804 , piece_id=MTPRWDDQQ5LMOTNCPOGKDH5IA57BICDAWSNOQ3H6BEBSZJSDOICQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection
Nov 9 00:15:36.940217358 (228h35m8.636066111s) [noise=true , address=145.79.227.25:4618 , piece_id=J7WJIE3QGIMDMAPSGEDMOUN23GX356DCB3LGFN7XJUNYK6CCLTMA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection
Nov 9 00:15:36.940262043 (228h35m8.636110786s) [seg_pos={0 0}, upload=201217]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil>
Nov 9 00:15:37.181912046 (228h35m8.877760799s) [upload=201217]: Done uploading segments...
Nov 9 00:15:37.181957073 (228h35m8.877805816s) [upload=201217]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4)
Duration: 5.898960045s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [........................................X........X..........] duration:
3.420879215s trace:
https://jaeger.dev.storj.io/trace/538482b67f052294
FAIL (2025-11-09 00:31:11 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.420910875s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
11.986952301s trace:
https://jaeger.dev.storj.io/trace/15a102436a900d91
FAIL (2025-11-09 00:30:45 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 4)
======LOGS======
pload=201217]: Uploading piece...
Nov 9 00:15:07.882409862 (228h34m39.578258615s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.882737906 (228h34m39.578586659s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883235645 (228h34m39.579084398s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883270351 (228h34m39.579119104s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883583637 (228h34m39.57943239s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:07.883903466 (228h34m39.579752219s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Uploading piece...
Nov 9 00:15:09.669212422 (228h34m41.365061175s) [upload=201217]: Next returned nil segment
Nov 9 00:15:09.669230136 (228h34m41.365078889s) [upload=201217]: Waiting for error group managing segments...
Nov 9 00:15:09.979393119 (228h34m41.675241862s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:09.982783635 (228h34m41.678632388s) [noise=true , address=45.91.116.9:4263 , piece_id=Z7MGGEQ63C5QOKI7BAAQCCJPJPLBM2MG2MBROD7GL4RRNCFVJJOQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:10.004031905 (228h34m41.699880668s) [noise=true , address=45.91.116.6:4973 , piece_id=2R4VUB2FG7IIY3KSBYSZFU7VTBGNS662DQ2AFOH7KDHNYSBGNUBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:10.072857382 (228h34m41.768706145s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil>
Nov 9 00:15:36.939056286 (228h35m8.634905039s) [noise=true , address=145.79.227.28:4726 , piece_id=EGLGRHWIZETHAYGPUIMSPEGOB3HXC2YMC32EGJIXSW7TQ4KGM2GA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection
Nov 9 00:15:36.939107635 (228h35m8.634956398s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection
Nov 9 00:15:36.939340538 (228h35m8.635189292s) [noise=true , address=145.79.227.10:4401 , piece_id=FKLG63DNHBF43ZT3MDH2D6GEDURMW6XA335EZND2LSQITYI5TYHQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection
Nov 9 00:15:36.939371116 (228h35m8.635219879s) [noise=true , address=145.79.227.15:4876 , piece_id=5XA3I6VEOLUYSQK374SW52H5LNMHEZX2HVOC6L6FFMAKGJV4RXTQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection
Nov 9 00:15:36.939473351 (228h35m8.635322104s) [noise=true , address=145.79.227.21:4473 , piece_id=3UNC3F6P7ERWHHZQTVSCIP4SBO6XL72I5F4ID5SOCIKTRFEWM6XQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection
Nov 9 00:15:36.939551320 (228h35m8.635400073s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: expected piece hash; context deadline exceeded; EOF
Nov 9 00:15:36.939605372 (228h35m8.635454125s) [noise=true , address=145.79.227.8:4330 , piece_id=M7O6MM4DRNV7Y63AGYACAKZ6674HNG777XVUGEAZ7XVEDIBBYTVQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection
Nov 9 00:15:36.939692238 (228h35m8.635540991s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection
Nov 9 00:15:36.939816855 (228h35m8.635665608s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection
Nov 9 00:15:36.939964185 (228h35m8.635812938s) [noise=true , address=145.79.227.9:4365 , piece_id=GYEVA6LDJIQJJ6AVSTAXAVD2JDQ6CBQERQJJMBTW5OV4EZMVTOKA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection
Nov 9 00:15:36.940093301 (228h35m8.635942054s) [noise=true , address=145.79.227.13:4804 , piece_id=MTPRWDDQQ5LMOTNCPOGKDH5IA57BICDAWSNOQ3H6BEBSZJSDOICQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection
Nov 9 00:15:36.940217358 (228h35m8.636066111s) [noise=true , address=145.79.227.25:4618 , piece_id=J7WJIE3QGIMDMAPSGEDMOUN23GX356DCB3LGFN7XJUNYK6CCLTMA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection
Nov 9 00:15:36.940262043 (228h35m8.636110786s) [seg_pos={0 0}, upload=201217]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil>
Nov 9 00:15:37.181912046 (228h35m8.877760799s) [upload=201217]: Done uploading segments...
Nov 9 00:15:37.181957073 (228h35m8.877805816s) [upload=201217]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4)
Duration: 11.986984302s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.........................................X........X.........] duration:
3.490110494s trace:
https://jaeger.dev.storj.io/trace/538482b67f052294
FAIL (2025-11-09 00:30:36 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.490143307s
core
uplink
us-select-1
us1-placement-20
|