Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......................................X........X............] duration: 3.446184793s trace: https://jaeger.dev.storj.io/trace/538482b67f052294 FAIL (2025-11-09 00:32:40 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.446275206s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......................................X........X...........] duration: 3.45789532s trace: https://jaeger.dev.storj.io/trace/538482b67f052294 FAIL (2025-11-09 00:32:05 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.457920447s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.898934508s trace: https://jaeger.dev.storj.io/trace/15a102436a900d91 FAIL (2025-11-09 00:31:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== pload=201217]: Uploading piece... Nov 9 00:15:07.882409862 (228h34m39.578258615s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.882737906 (228h34m39.578586659s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883235645 (228h34m39.579084398s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883270351 (228h34m39.579119104s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883583637 (228h34m39.57943239s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883903466 (228h34m39.579752219s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:09.669212422 (228h34m41.365061175s) [upload=201217]: Next returned nil segment Nov 9 00:15:09.669230136 (228h34m41.365078889s) [upload=201217]: Waiting for error group managing segments... Nov 9 00:15:09.979393119 (228h34m41.675241862s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:09.982783635 (228h34m41.678632388s) [noise=true , address=45.91.116.9:4263 , piece_id=Z7MGGEQ63C5QOKI7BAAQCCJPJPLBM2MG2MBROD7GL4RRNCFVJJOQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:10.004031905 (228h34m41.699880668s) [noise=true , address=45.91.116.6:4973 , piece_id=2R4VUB2FG7IIY3KSBYSZFU7VTBGNS662DQ2AFOH7KDHNYSBGNUBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:10.072857382 (228h34m41.768706145s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:36.939056286 (228h35m8.634905039s) [noise=true , address=145.79.227.28:4726 , piece_id=EGLGRHWIZETHAYGPUIMSPEGOB3HXC2YMC32EGJIXSW7TQ4KGM2GA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection Nov 9 00:15:36.939107635 (228h35m8.634956398s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection Nov 9 00:15:36.939340538 (228h35m8.635189292s) [noise=true , address=145.79.227.10:4401 , piece_id=FKLG63DNHBF43ZT3MDH2D6GEDURMW6XA335EZND2LSQITYI5TYHQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection Nov 9 00:15:36.939371116 (228h35m8.635219879s) [noise=true , address=145.79.227.15:4876 , piece_id=5XA3I6VEOLUYSQK374SW52H5LNMHEZX2HVOC6L6FFMAKGJV4RXTQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection Nov 9 00:15:36.939473351 (228h35m8.635322104s) [noise=true , address=145.79.227.21:4473 , piece_id=3UNC3F6P7ERWHHZQTVSCIP4SBO6XL72I5F4ID5SOCIKTRFEWM6XQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection Nov 9 00:15:36.939551320 (228h35m8.635400073s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 00:15:36.939605372 (228h35m8.635454125s) [noise=true , address=145.79.227.8:4330 , piece_id=M7O6MM4DRNV7Y63AGYACAKZ6674HNG777XVUGEAZ7XVEDIBBYTVQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection Nov 9 00:15:36.939692238 (228h35m8.635540991s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection Nov 9 00:15:36.939816855 (228h35m8.635665608s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection Nov 9 00:15:36.939964185 (228h35m8.635812938s) [noise=true , address=145.79.227.9:4365 , piece_id=GYEVA6LDJIQJJ6AVSTAXAVD2JDQ6CBQERQJJMBTW5OV4EZMVTOKA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection Nov 9 00:15:36.940093301 (228h35m8.635942054s) [noise=true , address=145.79.227.13:4804 , piece_id=MTPRWDDQQ5LMOTNCPOGKDH5IA57BICDAWSNOQ3H6BEBSZJSDOICQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection Nov 9 00:15:36.940217358 (228h35m8.636066111s) [noise=true , address=145.79.227.25:4618 , piece_id=J7WJIE3QGIMDMAPSGEDMOUN23GX356DCB3LGFN7XJUNYK6CCLTMA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection Nov 9 00:15:36.940262043 (228h35m8.636110786s) [seg_pos={0 0}, upload=201217]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 9 00:15:37.181912046 (228h35m8.877760799s) [upload=201217]: Done uploading segments... Nov 9 00:15:37.181957073 (228h35m8.877805816s) [upload=201217]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.898960045s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........................................X........X..........] duration: 3.420879215s trace: https://jaeger.dev.storj.io/trace/538482b67f052294 FAIL (2025-11-09 00:31:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.420910875s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 11.986952301s trace: https://jaeger.dev.storj.io/trace/15a102436a900d91 FAIL (2025-11-09 00:30:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== pload=201217]: Uploading piece... Nov 9 00:15:07.882409862 (228h34m39.578258615s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.882737906 (228h34m39.578586659s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883235645 (228h34m39.579084398s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883270351 (228h34m39.579119104s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883583637 (228h34m39.57943239s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883903466 (228h34m39.579752219s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:09.669212422 (228h34m41.365061175s) [upload=201217]: Next returned nil segment Nov 9 00:15:09.669230136 (228h34m41.365078889s) [upload=201217]: Waiting for error group managing segments... Nov 9 00:15:09.979393119 (228h34m41.675241862s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:09.982783635 (228h34m41.678632388s) [noise=true , address=45.91.116.9:4263 , piece_id=Z7MGGEQ63C5QOKI7BAAQCCJPJPLBM2MG2MBROD7GL4RRNCFVJJOQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:10.004031905 (228h34m41.699880668s) [noise=true , address=45.91.116.6:4973 , piece_id=2R4VUB2FG7IIY3KSBYSZFU7VTBGNS662DQ2AFOH7KDHNYSBGNUBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:10.072857382 (228h34m41.768706145s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:36.939056286 (228h35m8.634905039s) [noise=true , address=145.79.227.28:4726 , piece_id=EGLGRHWIZETHAYGPUIMSPEGOB3HXC2YMC32EGJIXSW7TQ4KGM2GA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection Nov 9 00:15:36.939107635 (228h35m8.634956398s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection Nov 9 00:15:36.939340538 (228h35m8.635189292s) [noise=true , address=145.79.227.10:4401 , piece_id=FKLG63DNHBF43ZT3MDH2D6GEDURMW6XA335EZND2LSQITYI5TYHQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection Nov 9 00:15:36.939371116 (228h35m8.635219879s) [noise=true , address=145.79.227.15:4876 , piece_id=5XA3I6VEOLUYSQK374SW52H5LNMHEZX2HVOC6L6FFMAKGJV4RXTQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection Nov 9 00:15:36.939473351 (228h35m8.635322104s) [noise=true , address=145.79.227.21:4473 , piece_id=3UNC3F6P7ERWHHZQTVSCIP4SBO6XL72I5F4ID5SOCIKTRFEWM6XQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection Nov 9 00:15:36.939551320 (228h35m8.635400073s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 00:15:36.939605372 (228h35m8.635454125s) [noise=true , address=145.79.227.8:4330 , piece_id=M7O6MM4DRNV7Y63AGYACAKZ6674HNG777XVUGEAZ7XVEDIBBYTVQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection Nov 9 00:15:36.939692238 (228h35m8.635540991s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection Nov 9 00:15:36.939816855 (228h35m8.635665608s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection Nov 9 00:15:36.939964185 (228h35m8.635812938s) [noise=true , address=145.79.227.9:4365 , piece_id=GYEVA6LDJIQJJ6AVSTAXAVD2JDQ6CBQERQJJMBTW5OV4EZMVTOKA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection Nov 9 00:15:36.940093301 (228h35m8.635942054s) [noise=true , address=145.79.227.13:4804 , piece_id=MTPRWDDQQ5LMOTNCPOGKDH5IA57BICDAWSNOQ3H6BEBSZJSDOICQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection Nov 9 00:15:36.940217358 (228h35m8.636066111s) [noise=true , address=145.79.227.25:4618 , piece_id=J7WJIE3QGIMDMAPSGEDMOUN23GX356DCB3LGFN7XJUNYK6CCLTMA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection Nov 9 00:15:36.940262043 (228h35m8.636110786s) [seg_pos={0 0}, upload=201217]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 9 00:15:37.181912046 (228h35m8.877760799s) [upload=201217]: Done uploading segments... Nov 9 00:15:37.181957073 (228h35m8.877805816s) [upload=201217]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 11.986984302s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........................................X........X.........] duration: 3.490110494s trace: https://jaeger.dev.storj.io/trace/538482b67f052294 FAIL (2025-11-09 00:30:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.490143307s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 12.928064436s trace: https://jaeger.dev.storj.io/trace/15a102436a900d91 FAIL (2025-11-09 00:29:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== pload=201217]: Uploading piece... Nov 9 00:15:07.882409862 (228h34m39.578258615s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.882737906 (228h34m39.578586659s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883235645 (228h34m39.579084398s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883270351 (228h34m39.579119104s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883583637 (228h34m39.57943239s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883903466 (228h34m39.579752219s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:09.669212422 (228h34m41.365061175s) [upload=201217]: Next returned nil segment Nov 9 00:15:09.669230136 (228h34m41.365078889s) [upload=201217]: Waiting for error group managing segments... Nov 9 00:15:09.979393119 (228h34m41.675241862s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:09.982783635 (228h34m41.678632388s) [noise=true , address=45.91.116.9:4263 , piece_id=Z7MGGEQ63C5QOKI7BAAQCCJPJPLBM2MG2MBROD7GL4RRNCFVJJOQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:10.004031905 (228h34m41.699880668s) [noise=true , address=45.91.116.6:4973 , piece_id=2R4VUB2FG7IIY3KSBYSZFU7VTBGNS662DQ2AFOH7KDHNYSBGNUBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:10.072857382 (228h34m41.768706145s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:36.939056286 (228h35m8.634905039s) [noise=true , address=145.79.227.28:4726 , piece_id=EGLGRHWIZETHAYGPUIMSPEGOB3HXC2YMC32EGJIXSW7TQ4KGM2GA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection Nov 9 00:15:36.939107635 (228h35m8.634956398s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection Nov 9 00:15:36.939340538 (228h35m8.635189292s) [noise=true , address=145.79.227.10:4401 , piece_id=FKLG63DNHBF43ZT3MDH2D6GEDURMW6XA335EZND2LSQITYI5TYHQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection Nov 9 00:15:36.939371116 (228h35m8.635219879s) [noise=true , address=145.79.227.15:4876 , piece_id=5XA3I6VEOLUYSQK374SW52H5LNMHEZX2HVOC6L6FFMAKGJV4RXTQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection Nov 9 00:15:36.939473351 (228h35m8.635322104s) [noise=true , address=145.79.227.21:4473 , piece_id=3UNC3F6P7ERWHHZQTVSCIP4SBO6XL72I5F4ID5SOCIKTRFEWM6XQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection Nov 9 00:15:36.939551320 (228h35m8.635400073s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 00:15:36.939605372 (228h35m8.635454125s) [noise=true , address=145.79.227.8:4330 , piece_id=M7O6MM4DRNV7Y63AGYACAKZ6674HNG777XVUGEAZ7XVEDIBBYTVQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection Nov 9 00:15:36.939692238 (228h35m8.635540991s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection Nov 9 00:15:36.939816855 (228h35m8.635665608s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection Nov 9 00:15:36.939964185 (228h35m8.635812938s) [noise=true , address=145.79.227.9:4365 , piece_id=GYEVA6LDJIQJJ6AVSTAXAVD2JDQ6CBQERQJJMBTW5OV4EZMVTOKA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection Nov 9 00:15:36.940093301 (228h35m8.635942054s) [noise=true , address=145.79.227.13:4804 , piece_id=MTPRWDDQQ5LMOTNCPOGKDH5IA57BICDAWSNOQ3H6BEBSZJSDOICQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection Nov 9 00:15:36.940217358 (228h35m8.636066111s) [noise=true , address=145.79.227.25:4618 , piece_id=J7WJIE3QGIMDMAPSGEDMOUN23GX356DCB3LGFN7XJUNYK6CCLTMA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection Nov 9 00:15:36.940262043 (228h35m8.636110786s) [seg_pos={0 0}, upload=201217]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 9 00:15:37.181912046 (228h35m8.877760799s) [upload=201217]: Done uploading segments... Nov 9 00:15:37.181957073 (228h35m8.877805816s) [upload=201217]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 12.92809277s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........................................X........X........] duration: 3.415075685s trace: https://jaeger.dev.storj.io/trace/538482b67f052294 FAIL (2025-11-09 00:29:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.415125481s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.84077449s trace: https://jaeger.dev.storj.io/trace/15a102436a900d91 FAIL (2025-11-09 00:28:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== pload=201217]: Uploading piece... Nov 9 00:15:07.882409862 (228h34m39.578258615s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.882737906 (228h34m39.578586659s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883235645 (228h34m39.579084398s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883270351 (228h34m39.579119104s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883583637 (228h34m39.57943239s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883903466 (228h34m39.579752219s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:09.669212422 (228h34m41.365061175s) [upload=201217]: Next returned nil segment Nov 9 00:15:09.669230136 (228h34m41.365078889s) [upload=201217]: Waiting for error group managing segments... Nov 9 00:15:09.979393119 (228h34m41.675241862s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:09.982783635 (228h34m41.678632388s) [noise=true , address=45.91.116.9:4263 , piece_id=Z7MGGEQ63C5QOKI7BAAQCCJPJPLBM2MG2MBROD7GL4RRNCFVJJOQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:10.004031905 (228h34m41.699880668s) [noise=true , address=45.91.116.6:4973 , piece_id=2R4VUB2FG7IIY3KSBYSZFU7VTBGNS662DQ2AFOH7KDHNYSBGNUBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:10.072857382 (228h34m41.768706145s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:36.939056286 (228h35m8.634905039s) [noise=true , address=145.79.227.28:4726 , piece_id=EGLGRHWIZETHAYGPUIMSPEGOB3HXC2YMC32EGJIXSW7TQ4KGM2GA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection Nov 9 00:15:36.939107635 (228h35m8.634956398s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection Nov 9 00:15:36.939340538 (228h35m8.635189292s) [noise=true , address=145.79.227.10:4401 , piece_id=FKLG63DNHBF43ZT3MDH2D6GEDURMW6XA335EZND2LSQITYI5TYHQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection Nov 9 00:15:36.939371116 (228h35m8.635219879s) [noise=true , address=145.79.227.15:4876 , piece_id=5XA3I6VEOLUYSQK374SW52H5LNMHEZX2HVOC6L6FFMAKGJV4RXTQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection Nov 9 00:15:36.939473351 (228h35m8.635322104s) [noise=true , address=145.79.227.21:4473 , piece_id=3UNC3F6P7ERWHHZQTVSCIP4SBO6XL72I5F4ID5SOCIKTRFEWM6XQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection Nov 9 00:15:36.939551320 (228h35m8.635400073s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 00:15:36.939605372 (228h35m8.635454125s) [noise=true , address=145.79.227.8:4330 , piece_id=M7O6MM4DRNV7Y63AGYACAKZ6674HNG777XVUGEAZ7XVEDIBBYTVQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection Nov 9 00:15:36.939692238 (228h35m8.635540991s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection Nov 9 00:15:36.939816855 (228h35m8.635665608s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection Nov 9 00:15:36.939964185 (228h35m8.635812938s) [noise=true , address=145.79.227.9:4365 , piece_id=GYEVA6LDJIQJJ6AVSTAXAVD2JDQ6CBQERQJJMBTW5OV4EZMVTOKA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection Nov 9 00:15:36.940093301 (228h35m8.635942054s) [noise=true , address=145.79.227.13:4804 , piece_id=MTPRWDDQQ5LMOTNCPOGKDH5IA57BICDAWSNOQ3H6BEBSZJSDOICQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection Nov 9 00:15:36.940217358 (228h35m8.636066111s) [noise=true , address=145.79.227.25:4618 , piece_id=J7WJIE3QGIMDMAPSGEDMOUN23GX356DCB3LGFN7XJUNYK6CCLTMA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection Nov 9 00:15:36.940262043 (228h35m8.636110786s) [seg_pos={0 0}, upload=201217]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 9 00:15:37.181912046 (228h35m8.877760799s) [upload=201217]: Done uploading segments... Nov 9 00:15:37.181957073 (228h35m8.877805816s) [upload=201217]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.840805149s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........................................X........X.......] duration: 3.523098228s trace: https://jaeger.dev.storj.io/trace/538482b67f052294 FAIL (2025-11-09 00:28:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.523127654s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.972978019s trace: https://jaeger.dev.storj.io/trace/15a102436a900d91 FAIL (2025-11-09 00:27:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== pload=201217]: Uploading piece... Nov 9 00:15:07.882409862 (228h34m39.578258615s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.882737906 (228h34m39.578586659s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883235645 (228h34m39.579084398s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883270351 (228h34m39.579119104s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883583637 (228h34m39.57943239s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883903466 (228h34m39.579752219s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:09.669212422 (228h34m41.365061175s) [upload=201217]: Next returned nil segment Nov 9 00:15:09.669230136 (228h34m41.365078889s) [upload=201217]: Waiting for error group managing segments... Nov 9 00:15:09.979393119 (228h34m41.675241862s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:09.982783635 (228h34m41.678632388s) [noise=true , address=45.91.116.9:4263 , piece_id=Z7MGGEQ63C5QOKI7BAAQCCJPJPLBM2MG2MBROD7GL4RRNCFVJJOQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:10.004031905 (228h34m41.699880668s) [noise=true , address=45.91.116.6:4973 , piece_id=2R4VUB2FG7IIY3KSBYSZFU7VTBGNS662DQ2AFOH7KDHNYSBGNUBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:10.072857382 (228h34m41.768706145s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:36.939056286 (228h35m8.634905039s) [noise=true , address=145.79.227.28:4726 , piece_id=EGLGRHWIZETHAYGPUIMSPEGOB3HXC2YMC32EGJIXSW7TQ4KGM2GA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection Nov 9 00:15:36.939107635 (228h35m8.634956398s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection Nov 9 00:15:36.939340538 (228h35m8.635189292s) [noise=true , address=145.79.227.10:4401 , piece_id=FKLG63DNHBF43ZT3MDH2D6GEDURMW6XA335EZND2LSQITYI5TYHQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection Nov 9 00:15:36.939371116 (228h35m8.635219879s) [noise=true , address=145.79.227.15:4876 , piece_id=5XA3I6VEOLUYSQK374SW52H5LNMHEZX2HVOC6L6FFMAKGJV4RXTQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection Nov 9 00:15:36.939473351 (228h35m8.635322104s) [noise=true , address=145.79.227.21:4473 , piece_id=3UNC3F6P7ERWHHZQTVSCIP4SBO6XL72I5F4ID5SOCIKTRFEWM6XQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection Nov 9 00:15:36.939551320 (228h35m8.635400073s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 00:15:36.939605372 (228h35m8.635454125s) [noise=true , address=145.79.227.8:4330 , piece_id=M7O6MM4DRNV7Y63AGYACAKZ6674HNG777XVUGEAZ7XVEDIBBYTVQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection Nov 9 00:15:36.939692238 (228h35m8.635540991s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection Nov 9 00:15:36.939816855 (228h35m8.635665608s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection Nov 9 00:15:36.939964185 (228h35m8.635812938s) [noise=true , address=145.79.227.9:4365 , piece_id=GYEVA6LDJIQJJ6AVSTAXAVD2JDQ6CBQERQJJMBTW5OV4EZMVTOKA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection Nov 9 00:15:36.940093301 (228h35m8.635942054s) [noise=true , address=145.79.227.13:4804 , piece_id=MTPRWDDQQ5LMOTNCPOGKDH5IA57BICDAWSNOQ3H6BEBSZJSDOICQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection Nov 9 00:15:36.940217358 (228h35m8.636066111s) [noise=true , address=145.79.227.25:4618 , piece_id=J7WJIE3QGIMDMAPSGEDMOUN23GX356DCB3LGFN7XJUNYK6CCLTMA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection Nov 9 00:15:36.940262043 (228h35m8.636110786s) [seg_pos={0 0}, upload=201217]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 9 00:15:37.181912046 (228h35m8.877760799s) [upload=201217]: Done uploading segments... Nov 9 00:15:37.181957073 (228h35m8.877805816s) [upload=201217]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.972995583s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............................................X........X......] duration: 3.524192911s trace: https://jaeger.dev.storj.io/trace/538482b67f052294 FAIL (2025-11-09 00:27:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.524242355s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 6.02252839s trace: https://jaeger.dev.storj.io/trace/15a102436a900d91 FAIL (2025-11-09 00:27:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== pload=201217]: Uploading piece... Nov 9 00:15:07.882409862 (228h34m39.578258615s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.882737906 (228h34m39.578586659s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883235645 (228h34m39.579084398s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883270351 (228h34m39.579119104s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883583637 (228h34m39.57943239s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883903466 (228h34m39.579752219s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:09.669212422 (228h34m41.365061175s) [upload=201217]: Next returned nil segment Nov 9 00:15:09.669230136 (228h34m41.365078889s) [upload=201217]: Waiting for error group managing segments... Nov 9 00:15:09.979393119 (228h34m41.675241862s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:09.982783635 (228h34m41.678632388s) [noise=true , address=45.91.116.9:4263 , piece_id=Z7MGGEQ63C5QOKI7BAAQCCJPJPLBM2MG2MBROD7GL4RRNCFVJJOQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:10.004031905 (228h34m41.699880668s) [noise=true , address=45.91.116.6:4973 , piece_id=2R4VUB2FG7IIY3KSBYSZFU7VTBGNS662DQ2AFOH7KDHNYSBGNUBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:10.072857382 (228h34m41.768706145s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:36.939056286 (228h35m8.634905039s) [noise=true , address=145.79.227.28:4726 , piece_id=EGLGRHWIZETHAYGPUIMSPEGOB3HXC2YMC32EGJIXSW7TQ4KGM2GA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection Nov 9 00:15:36.939107635 (228h35m8.634956398s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection Nov 9 00:15:36.939340538 (228h35m8.635189292s) [noise=true , address=145.79.227.10:4401 , piece_id=FKLG63DNHBF43ZT3MDH2D6GEDURMW6XA335EZND2LSQITYI5TYHQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection Nov 9 00:15:36.939371116 (228h35m8.635219879s) [noise=true , address=145.79.227.15:4876 , piece_id=5XA3I6VEOLUYSQK374SW52H5LNMHEZX2HVOC6L6FFMAKGJV4RXTQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection Nov 9 00:15:36.939473351 (228h35m8.635322104s) [noise=true , address=145.79.227.21:4473 , piece_id=3UNC3F6P7ERWHHZQTVSCIP4SBO6XL72I5F4ID5SOCIKTRFEWM6XQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection Nov 9 00:15:36.939551320 (228h35m8.635400073s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 00:15:36.939605372 (228h35m8.635454125s) [noise=true , address=145.79.227.8:4330 , piece_id=M7O6MM4DRNV7Y63AGYACAKZ6674HNG777XVUGEAZ7XVEDIBBYTVQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection Nov 9 00:15:36.939692238 (228h35m8.635540991s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection Nov 9 00:15:36.939816855 (228h35m8.635665608s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection Nov 9 00:15:36.939964185 (228h35m8.635812938s) [noise=true , address=145.79.227.9:4365 , piece_id=GYEVA6LDJIQJJ6AVSTAXAVD2JDQ6CBQERQJJMBTW5OV4EZMVTOKA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection Nov 9 00:15:36.940093301 (228h35m8.635942054s) [noise=true , address=145.79.227.13:4804 , piece_id=MTPRWDDQQ5LMOTNCPOGKDH5IA57BICDAWSNOQ3H6BEBSZJSDOICQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection Nov 9 00:15:36.940217358 (228h35m8.636066111s) [noise=true , address=145.79.227.25:4618 , piece_id=J7WJIE3QGIMDMAPSGEDMOUN23GX356DCB3LGFN7XJUNYK6CCLTMA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection Nov 9 00:15:36.940262043 (228h35m8.636110786s) [seg_pos={0 0}, upload=201217]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 9 00:15:37.181912046 (228h35m8.877760799s) [upload=201217]: Done uploading segments... Nov 9 00:15:37.181957073 (228h35m8.877805816s) [upload=201217]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.022558989s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.............................................X........X.....] duration: 3.440859521s trace: https://jaeger.dev.storj.io/trace/538482b67f052294 FAIL (2025-11-09 00:26:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.440896413s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.941660874s trace: https://jaeger.dev.storj.io/trace/15a102436a900d91 FAIL (2025-11-09 00:26:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== pload=201217]: Uploading piece... Nov 9 00:15:07.882409862 (228h34m39.578258615s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.882737906 (228h34m39.578586659s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883235645 (228h34m39.579084398s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883270351 (228h34m39.579119104s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883583637 (228h34m39.57943239s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883903466 (228h34m39.579752219s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:09.669212422 (228h34m41.365061175s) [upload=201217]: Next returned nil segment Nov 9 00:15:09.669230136 (228h34m41.365078889s) [upload=201217]: Waiting for error group managing segments... Nov 9 00:15:09.979393119 (228h34m41.675241862s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:09.982783635 (228h34m41.678632388s) [noise=true , address=45.91.116.9:4263 , piece_id=Z7MGGEQ63C5QOKI7BAAQCCJPJPLBM2MG2MBROD7GL4RRNCFVJJOQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:10.004031905 (228h34m41.699880668s) [noise=true , address=45.91.116.6:4973 , piece_id=2R4VUB2FG7IIY3KSBYSZFU7VTBGNS662DQ2AFOH7KDHNYSBGNUBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:10.072857382 (228h34m41.768706145s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:36.939056286 (228h35m8.634905039s) [noise=true , address=145.79.227.28:4726 , piece_id=EGLGRHWIZETHAYGPUIMSPEGOB3HXC2YMC32EGJIXSW7TQ4KGM2GA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection Nov 9 00:15:36.939107635 (228h35m8.634956398s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection Nov 9 00:15:36.939340538 (228h35m8.635189292s) [noise=true , address=145.79.227.10:4401 , piece_id=FKLG63DNHBF43ZT3MDH2D6GEDURMW6XA335EZND2LSQITYI5TYHQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection Nov 9 00:15:36.939371116 (228h35m8.635219879s) [noise=true , address=145.79.227.15:4876 , piece_id=5XA3I6VEOLUYSQK374SW52H5LNMHEZX2HVOC6L6FFMAKGJV4RXTQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection Nov 9 00:15:36.939473351 (228h35m8.635322104s) [noise=true , address=145.79.227.21:4473 , piece_id=3UNC3F6P7ERWHHZQTVSCIP4SBO6XL72I5F4ID5SOCIKTRFEWM6XQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection Nov 9 00:15:36.939551320 (228h35m8.635400073s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 00:15:36.939605372 (228h35m8.635454125s) [noise=true , address=145.79.227.8:4330 , piece_id=M7O6MM4DRNV7Y63AGYACAKZ6674HNG777XVUGEAZ7XVEDIBBYTVQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection Nov 9 00:15:36.939692238 (228h35m8.635540991s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection Nov 9 00:15:36.939816855 (228h35m8.635665608s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection Nov 9 00:15:36.939964185 (228h35m8.635812938s) [noise=true , address=145.79.227.9:4365 , piece_id=GYEVA6LDJIQJJ6AVSTAXAVD2JDQ6CBQERQJJMBTW5OV4EZMVTOKA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection Nov 9 00:15:36.940093301 (228h35m8.635942054s) [noise=true , address=145.79.227.13:4804 , piece_id=MTPRWDDQQ5LMOTNCPOGKDH5IA57BICDAWSNOQ3H6BEBSZJSDOICQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection Nov 9 00:15:36.940217358 (228h35m8.636066111s) [noise=true , address=145.79.227.25:4618 , piece_id=J7WJIE3QGIMDMAPSGEDMOUN23GX356DCB3LGFN7XJUNYK6CCLTMA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection Nov 9 00:15:36.940262043 (228h35m8.636110786s) [seg_pos={0 0}, upload=201217]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 9 00:15:37.181912046 (228h35m8.877760799s) [upload=201217]: Done uploading segments... Nov 9 00:15:37.181957073 (228h35m8.877805816s) [upload=201217]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.941690831s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..............................................X........X....] duration: 3.498658382s trace: https://jaeger.dev.storj.io/trace/538482b67f052294 FAIL (2025-11-09 00:25:55 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.498683951s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [X..............] duration: 60.12006ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/170f05fb2ea3592a FAIL (2025-11-09 00:25:48 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:54288->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 60.148093ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 6.264504506s trace: https://jaeger.dev.storj.io/trace/15a102436a900d91 FAIL (2025-11-09 00:25:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== pload=201217]: Uploading piece... Nov 9 00:15:07.882409862 (228h34m39.578258615s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.882737906 (228h34m39.578586659s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883235645 (228h34m39.579084398s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883270351 (228h34m39.579119104s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883583637 (228h34m39.57943239s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883903466 (228h34m39.579752219s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:09.669212422 (228h34m41.365061175s) [upload=201217]: Next returned nil segment Nov 9 00:15:09.669230136 (228h34m41.365078889s) [upload=201217]: Waiting for error group managing segments... Nov 9 00:15:09.979393119 (228h34m41.675241862s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:09.982783635 (228h34m41.678632388s) [noise=true , address=45.91.116.9:4263 , piece_id=Z7MGGEQ63C5QOKI7BAAQCCJPJPLBM2MG2MBROD7GL4RRNCFVJJOQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:10.004031905 (228h34m41.699880668s) [noise=true , address=45.91.116.6:4973 , piece_id=2R4VUB2FG7IIY3KSBYSZFU7VTBGNS662DQ2AFOH7KDHNYSBGNUBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:10.072857382 (228h34m41.768706145s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:36.939056286 (228h35m8.634905039s) [noise=true , address=145.79.227.28:4726 , piece_id=EGLGRHWIZETHAYGPUIMSPEGOB3HXC2YMC32EGJIXSW7TQ4KGM2GA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection Nov 9 00:15:36.939107635 (228h35m8.634956398s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection Nov 9 00:15:36.939340538 (228h35m8.635189292s) [noise=true , address=145.79.227.10:4401 , piece_id=FKLG63DNHBF43ZT3MDH2D6GEDURMW6XA335EZND2LSQITYI5TYHQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection Nov 9 00:15:36.939371116 (228h35m8.635219879s) [noise=true , address=145.79.227.15:4876 , piece_id=5XA3I6VEOLUYSQK374SW52H5LNMHEZX2HVOC6L6FFMAKGJV4RXTQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection Nov 9 00:15:36.939473351 (228h35m8.635322104s) [noise=true , address=145.79.227.21:4473 , piece_id=3UNC3F6P7ERWHHZQTVSCIP4SBO6XL72I5F4ID5SOCIKTRFEWM6XQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection Nov 9 00:15:36.939551320 (228h35m8.635400073s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 00:15:36.939605372 (228h35m8.635454125s) [noise=true , address=145.79.227.8:4330 , piece_id=M7O6MM4DRNV7Y63AGYACAKZ6674HNG777XVUGEAZ7XVEDIBBYTVQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection Nov 9 00:15:36.939692238 (228h35m8.635540991s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection Nov 9 00:15:36.939816855 (228h35m8.635665608s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection Nov 9 00:15:36.939964185 (228h35m8.635812938s) [noise=true , address=145.79.227.9:4365 , piece_id=GYEVA6LDJIQJJ6AVSTAXAVD2JDQ6CBQERQJJMBTW5OV4EZMVTOKA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection Nov 9 00:15:36.940093301 (228h35m8.635942054s) [noise=true , address=145.79.227.13:4804 , piece_id=MTPRWDDQQ5LMOTNCPOGKDH5IA57BICDAWSNOQ3H6BEBSZJSDOICQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection Nov 9 00:15:36.940217358 (228h35m8.636066111s) [noise=true , address=145.79.227.25:4618 , piece_id=J7WJIE3QGIMDMAPSGEDMOUN23GX356DCB3LGFN7XJUNYK6CCLTMA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection Nov 9 00:15:36.940262043 (228h35m8.636110786s) [seg_pos={0 0}, upload=201217]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 9 00:15:37.181912046 (228h35m8.877760799s) [upload=201217]: Done uploading segments... Nov 9 00:15:37.181957073 (228h35m8.877805816s) [upload=201217]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.264566434s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...............................................X........X...] duration: 3.421989984s trace: https://jaeger.dev.storj.io/trace/538482b67f052294 FAIL (2025-11-09 00:24:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.422016424s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.X.............] duration: 64.262827ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/170f05fb2ea3592a FAIL (2025-11-09 00:24:38 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:54288->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 64.293877ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [................................................X........X..] duration: 3.466169586s trace: https://jaeger.dev.storj.io/trace/538482b67f052294 FAIL (2025-11-09 00:24:17 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.466203981s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 6.029991362s trace: https://jaeger.dev.storj.io/trace/15a102436a900d91 FAIL (2025-11-09 00:24:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== pload=201217]: Uploading piece... Nov 9 00:15:07.882409862 (228h34m39.578258615s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.882737906 (228h34m39.578586659s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883235645 (228h34m39.579084398s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883270351 (228h34m39.579119104s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883583637 (228h34m39.57943239s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883903466 (228h34m39.579752219s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:09.669212422 (228h34m41.365061175s) [upload=201217]: Next returned nil segment Nov 9 00:15:09.669230136 (228h34m41.365078889s) [upload=201217]: Waiting for error group managing segments... Nov 9 00:15:09.979393119 (228h34m41.675241862s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:09.982783635 (228h34m41.678632388s) [noise=true , address=45.91.116.9:4263 , piece_id=Z7MGGEQ63C5QOKI7BAAQCCJPJPLBM2MG2MBROD7GL4RRNCFVJJOQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:10.004031905 (228h34m41.699880668s) [noise=true , address=45.91.116.6:4973 , piece_id=2R4VUB2FG7IIY3KSBYSZFU7VTBGNS662DQ2AFOH7KDHNYSBGNUBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:10.072857382 (228h34m41.768706145s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:36.939056286 (228h35m8.634905039s) [noise=true , address=145.79.227.28:4726 , piece_id=EGLGRHWIZETHAYGPUIMSPEGOB3HXC2YMC32EGJIXSW7TQ4KGM2GA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection Nov 9 00:15:36.939107635 (228h35m8.634956398s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection Nov 9 00:15:36.939340538 (228h35m8.635189292s) [noise=true , address=145.79.227.10:4401 , piece_id=FKLG63DNHBF43ZT3MDH2D6GEDURMW6XA335EZND2LSQITYI5TYHQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection Nov 9 00:15:36.939371116 (228h35m8.635219879s) [noise=true , address=145.79.227.15:4876 , piece_id=5XA3I6VEOLUYSQK374SW52H5LNMHEZX2HVOC6L6FFMAKGJV4RXTQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection Nov 9 00:15:36.939473351 (228h35m8.635322104s) [noise=true , address=145.79.227.21:4473 , piece_id=3UNC3F6P7ERWHHZQTVSCIP4SBO6XL72I5F4ID5SOCIKTRFEWM6XQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection Nov 9 00:15:36.939551320 (228h35m8.635400073s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 00:15:36.939605372 (228h35m8.635454125s) [noise=true , address=145.79.227.8:4330 , piece_id=M7O6MM4DRNV7Y63AGYACAKZ6674HNG777XVUGEAZ7XVEDIBBYTVQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection Nov 9 00:15:36.939692238 (228h35m8.635540991s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection Nov 9 00:15:36.939816855 (228h35m8.635665608s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection Nov 9 00:15:36.939964185 (228h35m8.635812938s) [noise=true , address=145.79.227.9:4365 , piece_id=GYEVA6LDJIQJJ6AVSTAXAVD2JDQ6CBQERQJJMBTW5OV4EZMVTOKA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection Nov 9 00:15:36.940093301 (228h35m8.635942054s) [noise=true , address=145.79.227.13:4804 , piece_id=MTPRWDDQQ5LMOTNCPOGKDH5IA57BICDAWSNOQ3H6BEBSZJSDOICQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection Nov 9 00:15:36.940217358 (228h35m8.636066111s) [noise=true , address=145.79.227.25:4618 , piece_id=J7WJIE3QGIMDMAPSGEDMOUN23GX356DCB3LGFN7XJUNYK6CCLTMA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection Nov 9 00:15:36.940262043 (228h35m8.636110786s) [seg_pos={0 0}, upload=201217]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 9 00:15:37.181912046 (228h35m8.877760799s) [upload=201217]: Done uploading segments... Nov 9 00:15:37.181957073 (228h35m8.877805816s) [upload=201217]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.030272547s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [..X............] duration: 66.172103ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/170f05fb2ea3592a FAIL (2025-11-09 00:24:08 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:54288->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 66.20157ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [...X...........] duration: 54.029521ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/170f05fb2ea3592a FAIL (2025-11-09 00:23:51 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:54288->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 54.061623ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.89566987s trace: https://jaeger.dev.storj.io/trace/15a102436a900d91 FAIL (2025-11-09 00:23:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== pload=201217]: Uploading piece... Nov 9 00:15:07.882409862 (228h34m39.578258615s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.882737906 (228h34m39.578586659s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883235645 (228h34m39.579084398s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883270351 (228h34m39.579119104s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883583637 (228h34m39.57943239s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883903466 (228h34m39.579752219s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:09.669212422 (228h34m41.365061175s) [upload=201217]: Next returned nil segment Nov 9 00:15:09.669230136 (228h34m41.365078889s) [upload=201217]: Waiting for error group managing segments... Nov 9 00:15:09.979393119 (228h34m41.675241862s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:09.982783635 (228h34m41.678632388s) [noise=true , address=45.91.116.9:4263 , piece_id=Z7MGGEQ63C5QOKI7BAAQCCJPJPLBM2MG2MBROD7GL4RRNCFVJJOQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:10.004031905 (228h34m41.699880668s) [noise=true , address=45.91.116.6:4973 , piece_id=2R4VUB2FG7IIY3KSBYSZFU7VTBGNS662DQ2AFOH7KDHNYSBGNUBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:10.072857382 (228h34m41.768706145s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:36.939056286 (228h35m8.634905039s) [noise=true , address=145.79.227.28:4726 , piece_id=EGLGRHWIZETHAYGPUIMSPEGOB3HXC2YMC32EGJIXSW7TQ4KGM2GA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection Nov 9 00:15:36.939107635 (228h35m8.634956398s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection Nov 9 00:15:36.939340538 (228h35m8.635189292s) [noise=true , address=145.79.227.10:4401 , piece_id=FKLG63DNHBF43ZT3MDH2D6GEDURMW6XA335EZND2LSQITYI5TYHQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection Nov 9 00:15:36.939371116 (228h35m8.635219879s) [noise=true , address=145.79.227.15:4876 , piece_id=5XA3I6VEOLUYSQK374SW52H5LNMHEZX2HVOC6L6FFMAKGJV4RXTQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection Nov 9 00:15:36.939473351 (228h35m8.635322104s) [noise=true , address=145.79.227.21:4473 , piece_id=3UNC3F6P7ERWHHZQTVSCIP4SBO6XL72I5F4ID5SOCIKTRFEWM6XQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection Nov 9 00:15:36.939551320 (228h35m8.635400073s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 00:15:36.939605372 (228h35m8.635454125s) [noise=true , address=145.79.227.8:4330 , piece_id=M7O6MM4DRNV7Y63AGYACAKZ6674HNG777XVUGEAZ7XVEDIBBYTVQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection Nov 9 00:15:36.939692238 (228h35m8.635540991s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection Nov 9 00:15:36.939816855 (228h35m8.635665608s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection Nov 9 00:15:36.939964185 (228h35m8.635812938s) [noise=true , address=145.79.227.9:4365 , piece_id=GYEVA6LDJIQJJ6AVSTAXAVD2JDQ6CBQERQJJMBTW5OV4EZMVTOKA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection Nov 9 00:15:36.940093301 (228h35m8.635942054s) [noise=true , address=145.79.227.13:4804 , piece_id=MTPRWDDQQ5LMOTNCPOGKDH5IA57BICDAWSNOQ3H6BEBSZJSDOICQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection Nov 9 00:15:36.940217358 (228h35m8.636066111s) [noise=true , address=145.79.227.25:4618 , piece_id=J7WJIE3QGIMDMAPSGEDMOUN23GX356DCB3LGFN7XJUNYK6CCLTMA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection Nov 9 00:15:36.940262043 (228h35m8.636110786s) [seg_pos={0 0}, upload=201217]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 9 00:15:37.181912046 (228h35m8.877760799s) [upload=201217]: Done uploading segments... Nov 9 00:15:37.181957073 (228h35m8.877805816s) [upload=201217]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.895700689s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [....X..........] duration: 49.991363ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/170f05fb2ea3592a FAIL (2025-11-09 00:23:26 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:54288->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 50.053401ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.................................................X........X.] duration: 3.49532913s trace: https://jaeger.dev.storj.io/trace/538482b67f052294 FAIL (2025-11-09 00:23:08 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.495361201s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.....X.........] duration: 70.745172ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/170f05fb2ea3592a FAIL (2025-11-09 00:22:51 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:54288->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 70.767695ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.872256275s trace: https://jaeger.dev.storj.io/trace/15a102436a900d91 FAIL (2025-11-09 00:22:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== pload=201217]: Uploading piece... Nov 9 00:15:07.882409862 (228h34m39.578258615s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.882737906 (228h34m39.578586659s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883235645 (228h34m39.579084398s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883270351 (228h34m39.579119104s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883583637 (228h34m39.57943239s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883903466 (228h34m39.579752219s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:09.669212422 (228h34m41.365061175s) [upload=201217]: Next returned nil segment Nov 9 00:15:09.669230136 (228h34m41.365078889s) [upload=201217]: Waiting for error group managing segments... Nov 9 00:15:09.979393119 (228h34m41.675241862s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:09.982783635 (228h34m41.678632388s) [noise=true , address=45.91.116.9:4263 , piece_id=Z7MGGEQ63C5QOKI7BAAQCCJPJPLBM2MG2MBROD7GL4RRNCFVJJOQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:10.004031905 (228h34m41.699880668s) [noise=true , address=45.91.116.6:4973 , piece_id=2R4VUB2FG7IIY3KSBYSZFU7VTBGNS662DQ2AFOH7KDHNYSBGNUBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:10.072857382 (228h34m41.768706145s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:36.939056286 (228h35m8.634905039s) [noise=true , address=145.79.227.28:4726 , piece_id=EGLGRHWIZETHAYGPUIMSPEGOB3HXC2YMC32EGJIXSW7TQ4KGM2GA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection Nov 9 00:15:36.939107635 (228h35m8.634956398s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection Nov 9 00:15:36.939340538 (228h35m8.635189292s) [noise=true , address=145.79.227.10:4401 , piece_id=FKLG63DNHBF43ZT3MDH2D6GEDURMW6XA335EZND2LSQITYI5TYHQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection Nov 9 00:15:36.939371116 (228h35m8.635219879s) [noise=true , address=145.79.227.15:4876 , piece_id=5XA3I6VEOLUYSQK374SW52H5LNMHEZX2HVOC6L6FFMAKGJV4RXTQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection Nov 9 00:15:36.939473351 (228h35m8.635322104s) [noise=true , address=145.79.227.21:4473 , piece_id=3UNC3F6P7ERWHHZQTVSCIP4SBO6XL72I5F4ID5SOCIKTRFEWM6XQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection Nov 9 00:15:36.939551320 (228h35m8.635400073s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 00:15:36.939605372 (228h35m8.635454125s) [noise=true , address=145.79.227.8:4330 , piece_id=M7O6MM4DRNV7Y63AGYACAKZ6674HNG777XVUGEAZ7XVEDIBBYTVQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection Nov 9 00:15:36.939692238 (228h35m8.635540991s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection Nov 9 00:15:36.939816855 (228h35m8.635665608s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection Nov 9 00:15:36.939964185 (228h35m8.635812938s) [noise=true , address=145.79.227.9:4365 , piece_id=GYEVA6LDJIQJJ6AVSTAXAVD2JDQ6CBQERQJJMBTW5OV4EZMVTOKA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection Nov 9 00:15:36.940093301 (228h35m8.635942054s) [noise=true , address=145.79.227.13:4804 , piece_id=MTPRWDDQQ5LMOTNCPOGKDH5IA57BICDAWSNOQ3H6BEBSZJSDOICQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection Nov 9 00:15:36.940217358 (228h35m8.636066111s) [noise=true , address=145.79.227.25:4618 , piece_id=J7WJIE3QGIMDMAPSGEDMOUN23GX356DCB3LGFN7XJUNYK6CCLTMA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection Nov 9 00:15:36.940262043 (228h35m8.636110786s) [seg_pos={0 0}, upload=201217]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 9 00:15:37.181912046 (228h35m8.877760799s) [upload=201217]: Done uploading segments... Nov 9 00:15:37.181957073 (228h35m8.877805816s) [upload=201217]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.872305691s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..................................................X........X] duration: 5.001857932s trace: https://jaeger.dev.storj.io/trace/538482b67f052294 FAIL (2025-11-09 00:22:18 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001884331s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................................X........] duration: 3.440992355s trace: https://jaeger.dev.storj.io/trace/774ceef135bba751 FAIL (2025-11-09 00:21:57 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.441016321s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [......X........] duration: 44.390668ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/170f05fb2ea3592a FAIL (2025-11-09 00:21:54 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:54288->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 44.427958ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.904958346s trace: https://jaeger.dev.storj.io/trace/15a102436a900d91 FAIL (2025-11-09 00:21:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== pload=201217]: Uploading piece... Nov 9 00:15:07.882409862 (228h34m39.578258615s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.882737906 (228h34m39.578586659s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883235645 (228h34m39.579084398s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883270351 (228h34m39.579119104s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883583637 (228h34m39.57943239s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883903466 (228h34m39.579752219s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:09.669212422 (228h34m41.365061175s) [upload=201217]: Next returned nil segment Nov 9 00:15:09.669230136 (228h34m41.365078889s) [upload=201217]: Waiting for error group managing segments... Nov 9 00:15:09.979393119 (228h34m41.675241862s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:09.982783635 (228h34m41.678632388s) [noise=true , address=45.91.116.9:4263 , piece_id=Z7MGGEQ63C5QOKI7BAAQCCJPJPLBM2MG2MBROD7GL4RRNCFVJJOQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:10.004031905 (228h34m41.699880668s) [noise=true , address=45.91.116.6:4973 , piece_id=2R4VUB2FG7IIY3KSBYSZFU7VTBGNS662DQ2AFOH7KDHNYSBGNUBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:10.072857382 (228h34m41.768706145s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:36.939056286 (228h35m8.634905039s) [noise=true , address=145.79.227.28:4726 , piece_id=EGLGRHWIZETHAYGPUIMSPEGOB3HXC2YMC32EGJIXSW7TQ4KGM2GA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection Nov 9 00:15:36.939107635 (228h35m8.634956398s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection Nov 9 00:15:36.939340538 (228h35m8.635189292s) [noise=true , address=145.79.227.10:4401 , piece_id=FKLG63DNHBF43ZT3MDH2D6GEDURMW6XA335EZND2LSQITYI5TYHQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection Nov 9 00:15:36.939371116 (228h35m8.635219879s) [noise=true , address=145.79.227.15:4876 , piece_id=5XA3I6VEOLUYSQK374SW52H5LNMHEZX2HVOC6L6FFMAKGJV4RXTQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection Nov 9 00:15:36.939473351 (228h35m8.635322104s) [noise=true , address=145.79.227.21:4473 , piece_id=3UNC3F6P7ERWHHZQTVSCIP4SBO6XL72I5F4ID5SOCIKTRFEWM6XQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection Nov 9 00:15:36.939551320 (228h35m8.635400073s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 00:15:36.939605372 (228h35m8.635454125s) [noise=true , address=145.79.227.8:4330 , piece_id=M7O6MM4DRNV7Y63AGYACAKZ6674HNG777XVUGEAZ7XVEDIBBYTVQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection Nov 9 00:15:36.939692238 (228h35m8.635540991s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection Nov 9 00:15:36.939816855 (228h35m8.635665608s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection Nov 9 00:15:36.939964185 (228h35m8.635812938s) [noise=true , address=145.79.227.9:4365 , piece_id=GYEVA6LDJIQJJ6AVSTAXAVD2JDQ6CBQERQJJMBTW5OV4EZMVTOKA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection Nov 9 00:15:36.940093301 (228h35m8.635942054s) [noise=true , address=145.79.227.13:4804 , piece_id=MTPRWDDQQ5LMOTNCPOGKDH5IA57BICDAWSNOQ3H6BEBSZJSDOICQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection Nov 9 00:15:36.940217358 (228h35m8.636066111s) [noise=true , address=145.79.227.25:4618 , piece_id=J7WJIE3QGIMDMAPSGEDMOUN23GX356DCB3LGFN7XJUNYK6CCLTMA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection Nov 9 00:15:36.940262043 (228h35m8.636110786s) [seg_pos={0 0}, upload=201217]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 9 00:15:37.181912046 (228h35m8.877760799s) [upload=201217]: Done uploading segments... Nov 9 00:15:37.181957073 (228h35m8.877805816s) [upload=201217]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.904989296s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.915549457s trace: https://jaeger.dev.storj.io/trace/15a102436a900d91 FAIL (2025-11-09 00:21:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== pload=201217]: Uploading piece... Nov 9 00:15:07.882409862 (228h34m39.578258615s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.882737906 (228h34m39.578586659s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883235645 (228h34m39.579084398s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883270351 (228h34m39.579119104s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883583637 (228h34m39.57943239s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883903466 (228h34m39.579752219s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:09.669212422 (228h34m41.365061175s) [upload=201217]: Next returned nil segment Nov 9 00:15:09.669230136 (228h34m41.365078889s) [upload=201217]: Waiting for error group managing segments... Nov 9 00:15:09.979393119 (228h34m41.675241862s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:09.982783635 (228h34m41.678632388s) [noise=true , address=45.91.116.9:4263 , piece_id=Z7MGGEQ63C5QOKI7BAAQCCJPJPLBM2MG2MBROD7GL4RRNCFVJJOQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:10.004031905 (228h34m41.699880668s) [noise=true , address=45.91.116.6:4973 , piece_id=2R4VUB2FG7IIY3KSBYSZFU7VTBGNS662DQ2AFOH7KDHNYSBGNUBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:10.072857382 (228h34m41.768706145s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:36.939056286 (228h35m8.634905039s) [noise=true , address=145.79.227.28:4726 , piece_id=EGLGRHWIZETHAYGPUIMSPEGOB3HXC2YMC32EGJIXSW7TQ4KGM2GA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection Nov 9 00:15:36.939107635 (228h35m8.634956398s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection Nov 9 00:15:36.939340538 (228h35m8.635189292s) [noise=true , address=145.79.227.10:4401 , piece_id=FKLG63DNHBF43ZT3MDH2D6GEDURMW6XA335EZND2LSQITYI5TYHQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection Nov 9 00:15:36.939371116 (228h35m8.635219879s) [noise=true , address=145.79.227.15:4876 , piece_id=5XA3I6VEOLUYSQK374SW52H5LNMHEZX2HVOC6L6FFMAKGJV4RXTQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection Nov 9 00:15:36.939473351 (228h35m8.635322104s) [noise=true , address=145.79.227.21:4473 , piece_id=3UNC3F6P7ERWHHZQTVSCIP4SBO6XL72I5F4ID5SOCIKTRFEWM6XQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection Nov 9 00:15:36.939551320 (228h35m8.635400073s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 00:15:36.939605372 (228h35m8.635454125s) [noise=true , address=145.79.227.8:4330 , piece_id=M7O6MM4DRNV7Y63AGYACAKZ6674HNG777XVUGEAZ7XVEDIBBYTVQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection Nov 9 00:15:36.939692238 (228h35m8.635540991s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection Nov 9 00:15:36.939816855 (228h35m8.635665608s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection Nov 9 00:15:36.939964185 (228h35m8.635812938s) [noise=true , address=145.79.227.9:4365 , piece_id=GYEVA6LDJIQJJ6AVSTAXAVD2JDQ6CBQERQJJMBTW5OV4EZMVTOKA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection Nov 9 00:15:36.940093301 (228h35m8.635942054s) [noise=true , address=145.79.227.13:4804 , piece_id=MTPRWDDQQ5LMOTNCPOGKDH5IA57BICDAWSNOQ3H6BEBSZJSDOICQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection Nov 9 00:15:36.940217358 (228h35m8.636066111s) [noise=true , address=145.79.227.25:4618 , piece_id=J7WJIE3QGIMDMAPSGEDMOUN23GX356DCB3LGFN7XJUNYK6CCLTMA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection Nov 9 00:15:36.940262043 (228h35m8.636110786s) [seg_pos={0 0}, upload=201217]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 9 00:15:37.181912046 (228h35m8.877760799s) [upload=201217]: Done uploading segments... Nov 9 00:15:37.181957073 (228h35m8.877805816s) [upload=201217]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.915573262s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 3.451470387s trace: https://jaeger.dev.storj.io/trace/774ceef135bba751 FAIL (2025-11-09 00:20:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.451518158s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.......X.......] duration: 52.854295ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/170f05fb2ea3592a FAIL (2025-11-09 00:20:41 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:54288->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 52.988749ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................................X......] duration: 3.459331676s trace: https://jaeger.dev.storj.io/trace/774ceef135bba751 FAIL (2025-11-09 00:19:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.459402351s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.92115127s trace: https://jaeger.dev.storj.io/trace/15a102436a900d91 FAIL (2025-11-09 00:19:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== pload=201217]: Uploading piece... Nov 9 00:15:07.882409862 (228h34m39.578258615s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.882737906 (228h34m39.578586659s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883235645 (228h34m39.579084398s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883270351 (228h34m39.579119104s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883583637 (228h34m39.57943239s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883903466 (228h34m39.579752219s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:09.669212422 (228h34m41.365061175s) [upload=201217]: Next returned nil segment Nov 9 00:15:09.669230136 (228h34m41.365078889s) [upload=201217]: Waiting for error group managing segments... Nov 9 00:15:09.979393119 (228h34m41.675241862s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:09.982783635 (228h34m41.678632388s) [noise=true , address=45.91.116.9:4263 , piece_id=Z7MGGEQ63C5QOKI7BAAQCCJPJPLBM2MG2MBROD7GL4RRNCFVJJOQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:10.004031905 (228h34m41.699880668s) [noise=true , address=45.91.116.6:4973 , piece_id=2R4VUB2FG7IIY3KSBYSZFU7VTBGNS662DQ2AFOH7KDHNYSBGNUBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:10.072857382 (228h34m41.768706145s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:36.939056286 (228h35m8.634905039s) [noise=true , address=145.79.227.28:4726 , piece_id=EGLGRHWIZETHAYGPUIMSPEGOB3HXC2YMC32EGJIXSW7TQ4KGM2GA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection Nov 9 00:15:36.939107635 (228h35m8.634956398s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection Nov 9 00:15:36.939340538 (228h35m8.635189292s) [noise=true , address=145.79.227.10:4401 , piece_id=FKLG63DNHBF43ZT3MDH2D6GEDURMW6XA335EZND2LSQITYI5TYHQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection Nov 9 00:15:36.939371116 (228h35m8.635219879s) [noise=true , address=145.79.227.15:4876 , piece_id=5XA3I6VEOLUYSQK374SW52H5LNMHEZX2HVOC6L6FFMAKGJV4RXTQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection Nov 9 00:15:36.939473351 (228h35m8.635322104s) [noise=true , address=145.79.227.21:4473 , piece_id=3UNC3F6P7ERWHHZQTVSCIP4SBO6XL72I5F4ID5SOCIKTRFEWM6XQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection Nov 9 00:15:36.939551320 (228h35m8.635400073s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 00:15:36.939605372 (228h35m8.635454125s) [noise=true , address=145.79.227.8:4330 , piece_id=M7O6MM4DRNV7Y63AGYACAKZ6674HNG777XVUGEAZ7XVEDIBBYTVQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection Nov 9 00:15:36.939692238 (228h35m8.635540991s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection Nov 9 00:15:36.939816855 (228h35m8.635665608s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection Nov 9 00:15:36.939964185 (228h35m8.635812938s) [noise=true , address=145.79.227.9:4365 , piece_id=GYEVA6LDJIQJJ6AVSTAXAVD2JDQ6CBQERQJJMBTW5OV4EZMVTOKA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection Nov 9 00:15:36.940093301 (228h35m8.635942054s) [noise=true , address=145.79.227.13:4804 , piece_id=MTPRWDDQQ5LMOTNCPOGKDH5IA57BICDAWSNOQ3H6BEBSZJSDOICQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection Nov 9 00:15:36.940217358 (228h35m8.636066111s) [noise=true , address=145.79.227.25:4618 , piece_id=J7WJIE3QGIMDMAPSGEDMOUN23GX356DCB3LGFN7XJUNYK6CCLTMA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection Nov 9 00:15:36.940262043 (228h35m8.636110786s) [seg_pos={0 0}, upload=201217]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 9 00:15:37.181912046 (228h35m8.877760799s) [upload=201217]: Done uploading segments... Nov 9 00:15:37.181957073 (228h35m8.877805816s) [upload=201217]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.921184795s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [........X......] duration: 348.29244ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/170f05fb2ea3592a FAIL (2025-11-09 00:19:28 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:54288->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 348.322558ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................................X.....] duration: 3.507873408s trace: https://jaeger.dev.storj.io/trace/774ceef135bba751 FAIL (2025-11-09 00:19:17 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.507903446s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 11.723304649s trace: https://jaeger.dev.storj.io/trace/15a102436a900d91 FAIL (2025-11-09 00:19:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== pload=201217]: Uploading piece... Nov 9 00:15:07.882409862 (228h34m39.578258615s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.882737906 (228h34m39.578586659s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883235645 (228h34m39.579084398s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883270351 (228h34m39.579119104s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883583637 (228h34m39.57943239s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883903466 (228h34m39.579752219s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:09.669212422 (228h34m41.365061175s) [upload=201217]: Next returned nil segment Nov 9 00:15:09.669230136 (228h34m41.365078889s) [upload=201217]: Waiting for error group managing segments... Nov 9 00:15:09.979393119 (228h34m41.675241862s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:09.982783635 (228h34m41.678632388s) [noise=true , address=45.91.116.9:4263 , piece_id=Z7MGGEQ63C5QOKI7BAAQCCJPJPLBM2MG2MBROD7GL4RRNCFVJJOQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:10.004031905 (228h34m41.699880668s) [noise=true , address=45.91.116.6:4973 , piece_id=2R4VUB2FG7IIY3KSBYSZFU7VTBGNS662DQ2AFOH7KDHNYSBGNUBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:10.072857382 (228h34m41.768706145s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:36.939056286 (228h35m8.634905039s) [noise=true , address=145.79.227.28:4726 , piece_id=EGLGRHWIZETHAYGPUIMSPEGOB3HXC2YMC32EGJIXSW7TQ4KGM2GA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection Nov 9 00:15:36.939107635 (228h35m8.634956398s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection Nov 9 00:15:36.939340538 (228h35m8.635189292s) [noise=true , address=145.79.227.10:4401 , piece_id=FKLG63DNHBF43ZT3MDH2D6GEDURMW6XA335EZND2LSQITYI5TYHQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection Nov 9 00:15:36.939371116 (228h35m8.635219879s) [noise=true , address=145.79.227.15:4876 , piece_id=5XA3I6VEOLUYSQK374SW52H5LNMHEZX2HVOC6L6FFMAKGJV4RXTQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection Nov 9 00:15:36.939473351 (228h35m8.635322104s) [noise=true , address=145.79.227.21:4473 , piece_id=3UNC3F6P7ERWHHZQTVSCIP4SBO6XL72I5F4ID5SOCIKTRFEWM6XQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection Nov 9 00:15:36.939551320 (228h35m8.635400073s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 00:15:36.939605372 (228h35m8.635454125s) [noise=true , address=145.79.227.8:4330 , piece_id=M7O6MM4DRNV7Y63AGYACAKZ6674HNG777XVUGEAZ7XVEDIBBYTVQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection Nov 9 00:15:36.939692238 (228h35m8.635540991s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection Nov 9 00:15:36.939816855 (228h35m8.635665608s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection Nov 9 00:15:36.939964185 (228h35m8.635812938s) [noise=true , address=145.79.227.9:4365 , piece_id=GYEVA6LDJIQJJ6AVSTAXAVD2JDQ6CBQERQJJMBTW5OV4EZMVTOKA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection Nov 9 00:15:36.940093301 (228h35m8.635942054s) [noise=true , address=145.79.227.13:4804 , piece_id=MTPRWDDQQ5LMOTNCPOGKDH5IA57BICDAWSNOQ3H6BEBSZJSDOICQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection Nov 9 00:15:36.940217358 (228h35m8.636066111s) [noise=true , address=145.79.227.25:4618 , piece_id=J7WJIE3QGIMDMAPSGEDMOUN23GX356DCB3LGFN7XJUNYK6CCLTMA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection Nov 9 00:15:36.940262043 (228h35m8.636110786s) [seg_pos={0 0}, upload=201217]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 9 00:15:37.181912046 (228h35m8.877760799s) [upload=201217]: Done uploading segments... Nov 9 00:15:37.181957073 (228h35m8.877805816s) [upload=201217]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 11.723420089s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.........X.....] duration: 352.525782ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/170f05fb2ea3592a FAIL (2025-11-09 00:18:42 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:54288->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 352.55156ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................................X....] duration: 3.492868958s trace: https://jaeger.dev.storj.io/trace/774ceef135bba751 FAIL (2025-11-09 00:18:29 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.492898343s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [..........X....] duration: 55.68219ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/170f05fb2ea3592a FAIL (2025-11-09 00:18:25 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:54288->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 55.712366ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [...........X...] duration: 61.221727ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/170f05fb2ea3592a FAIL (2025-11-09 00:18:02 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:54288->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 61.255351ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 6.004896334s trace: https://jaeger.dev.storj.io/trace/15a102436a900d91 FAIL (2025-11-09 00:17:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== pload=201217]: Uploading piece... Nov 9 00:15:07.882409862 (228h34m39.578258615s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.882737906 (228h34m39.578586659s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883235645 (228h34m39.579084398s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883270351 (228h34m39.579119104s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883583637 (228h34m39.57943239s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883903466 (228h34m39.579752219s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:09.669212422 (228h34m41.365061175s) [upload=201217]: Next returned nil segment Nov 9 00:15:09.669230136 (228h34m41.365078889s) [upload=201217]: Waiting for error group managing segments... Nov 9 00:15:09.979393119 (228h34m41.675241862s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:09.982783635 (228h34m41.678632388s) [noise=true , address=45.91.116.9:4263 , piece_id=Z7MGGEQ63C5QOKI7BAAQCCJPJPLBM2MG2MBROD7GL4RRNCFVJJOQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:10.004031905 (228h34m41.699880668s) [noise=true , address=45.91.116.6:4973 , piece_id=2R4VUB2FG7IIY3KSBYSZFU7VTBGNS662DQ2AFOH7KDHNYSBGNUBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:10.072857382 (228h34m41.768706145s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:36.939056286 (228h35m8.634905039s) [noise=true , address=145.79.227.28:4726 , piece_id=EGLGRHWIZETHAYGPUIMSPEGOB3HXC2YMC32EGJIXSW7TQ4KGM2GA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection Nov 9 00:15:36.939107635 (228h35m8.634956398s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection Nov 9 00:15:36.939340538 (228h35m8.635189292s) [noise=true , address=145.79.227.10:4401 , piece_id=FKLG63DNHBF43ZT3MDH2D6GEDURMW6XA335EZND2LSQITYI5TYHQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection Nov 9 00:15:36.939371116 (228h35m8.635219879s) [noise=true , address=145.79.227.15:4876 , piece_id=5XA3I6VEOLUYSQK374SW52H5LNMHEZX2HVOC6L6FFMAKGJV4RXTQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection Nov 9 00:15:36.939473351 (228h35m8.635322104s) [noise=true , address=145.79.227.21:4473 , piece_id=3UNC3F6P7ERWHHZQTVSCIP4SBO6XL72I5F4ID5SOCIKTRFEWM6XQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection Nov 9 00:15:36.939551320 (228h35m8.635400073s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 00:15:36.939605372 (228h35m8.635454125s) [noise=true , address=145.79.227.8:4330 , piece_id=M7O6MM4DRNV7Y63AGYACAKZ6674HNG777XVUGEAZ7XVEDIBBYTVQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection Nov 9 00:15:36.939692238 (228h35m8.635540991s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection Nov 9 00:15:36.939816855 (228h35m8.635665608s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection Nov 9 00:15:36.939964185 (228h35m8.635812938s) [noise=true , address=145.79.227.9:4365 , piece_id=GYEVA6LDJIQJJ6AVSTAXAVD2JDQ6CBQERQJJMBTW5OV4EZMVTOKA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection Nov 9 00:15:36.940093301 (228h35m8.635942054s) [noise=true , address=145.79.227.13:4804 , piece_id=MTPRWDDQQ5LMOTNCPOGKDH5IA57BICDAWSNOQ3H6BEBSZJSDOICQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection Nov 9 00:15:36.940217358 (228h35m8.636066111s) [noise=true , address=145.79.227.25:4618 , piece_id=J7WJIE3QGIMDMAPSGEDMOUN23GX356DCB3LGFN7XJUNYK6CCLTMA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection Nov 9 00:15:36.940262043 (228h35m8.636110786s) [seg_pos={0 0}, upload=201217]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 9 00:15:37.181912046 (228h35m8.877760799s) [upload=201217]: Done uploading segments... Nov 9 00:15:37.181957073 (228h35m8.877805816s) [upload=201217]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.004923676s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 6.156630852s trace: https://jaeger.dev.storj.io/trace/15a102436a900d91 FAIL (2025-11-09 00:17:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== pload=201217]: Uploading piece... Nov 9 00:15:07.882409862 (228h34m39.578258615s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.882737906 (228h34m39.578586659s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883235645 (228h34m39.579084398s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883270351 (228h34m39.579119104s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883583637 (228h34m39.57943239s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883903466 (228h34m39.579752219s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:09.669212422 (228h34m41.365061175s) [upload=201217]: Next returned nil segment Nov 9 00:15:09.669230136 (228h34m41.365078889s) [upload=201217]: Waiting for error group managing segments... Nov 9 00:15:09.979393119 (228h34m41.675241862s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:09.982783635 (228h34m41.678632388s) [noise=true , address=45.91.116.9:4263 , piece_id=Z7MGGEQ63C5QOKI7BAAQCCJPJPLBM2MG2MBROD7GL4RRNCFVJJOQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:10.004031905 (228h34m41.699880668s) [noise=true , address=45.91.116.6:4973 , piece_id=2R4VUB2FG7IIY3KSBYSZFU7VTBGNS662DQ2AFOH7KDHNYSBGNUBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:10.072857382 (228h34m41.768706145s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:36.939056286 (228h35m8.634905039s) [noise=true , address=145.79.227.28:4726 , piece_id=EGLGRHWIZETHAYGPUIMSPEGOB3HXC2YMC32EGJIXSW7TQ4KGM2GA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection Nov 9 00:15:36.939107635 (228h35m8.634956398s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection Nov 9 00:15:36.939340538 (228h35m8.635189292s) [noise=true , address=145.79.227.10:4401 , piece_id=FKLG63DNHBF43ZT3MDH2D6GEDURMW6XA335EZND2LSQITYI5TYHQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection Nov 9 00:15:36.939371116 (228h35m8.635219879s) [noise=true , address=145.79.227.15:4876 , piece_id=5XA3I6VEOLUYSQK374SW52H5LNMHEZX2HVOC6L6FFMAKGJV4RXTQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection Nov 9 00:15:36.939473351 (228h35m8.635322104s) [noise=true , address=145.79.227.21:4473 , piece_id=3UNC3F6P7ERWHHZQTVSCIP4SBO6XL72I5F4ID5SOCIKTRFEWM6XQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection Nov 9 00:15:36.939551320 (228h35m8.635400073s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 00:15:36.939605372 (228h35m8.635454125s) [noise=true , address=145.79.227.8:4330 , piece_id=M7O6MM4DRNV7Y63AGYACAKZ6674HNG777XVUGEAZ7XVEDIBBYTVQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection Nov 9 00:15:36.939692238 (228h35m8.635540991s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection Nov 9 00:15:36.939816855 (228h35m8.635665608s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection Nov 9 00:15:36.939964185 (228h35m8.635812938s) [noise=true , address=145.79.227.9:4365 , piece_id=GYEVA6LDJIQJJ6AVSTAXAVD2JDQ6CBQERQJJMBTW5OV4EZMVTOKA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection Nov 9 00:15:36.940093301 (228h35m8.635942054s) [noise=true , address=145.79.227.13:4804 , piece_id=MTPRWDDQQ5LMOTNCPOGKDH5IA57BICDAWSNOQ3H6BEBSZJSDOICQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection Nov 9 00:15:36.940217358 (228h35m8.636066111s) [noise=true , address=145.79.227.25:4618 , piece_id=J7WJIE3QGIMDMAPSGEDMOUN23GX356DCB3LGFN7XJUNYK6CCLTMA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection Nov 9 00:15:36.940262043 (228h35m8.636110786s) [seg_pos={0 0}, upload=201217]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 9 00:15:37.181912046 (228h35m8.877760799s) [upload=201217]: Done uploading segments... Nov 9 00:15:37.181957073 (228h35m8.877805816s) [upload=201217]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.156649838s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................................X...] duration: 3.529503681s trace: https://jaeger.dev.storj.io/trace/774ceef135bba751 FAIL (2025-11-09 00:17:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.529533737s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [............X..] duration: 60.227163ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/170f05fb2ea3592a FAIL (2025-11-09 00:17:14 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:54288->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 60.255006ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.711733431s trace: https://jaeger.dev.storj.io/trace/15a102436a900d91 FAIL (2025-11-09 00:16:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== pload=201217]: Uploading piece... Nov 9 00:15:07.882409862 (228h34m39.578258615s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.882737906 (228h34m39.578586659s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883235645 (228h34m39.579084398s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883270351 (228h34m39.579119104s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883583637 (228h34m39.57943239s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:07.883903466 (228h34m39.579752219s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Uploading piece... Nov 9 00:15:09.669212422 (228h34m41.365061175s) [upload=201217]: Next returned nil segment Nov 9 00:15:09.669230136 (228h34m41.365078889s) [upload=201217]: Waiting for error group managing segments... Nov 9 00:15:09.979393119 (228h34m41.675241862s) [noise=true , address=45.91.116.10:4082 , piece_id=3VLUPZU4BJ6Q7BR5CGDFTGVHWR5UTYOHC35ZWILT3OTL4V2ETZGQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:09.982783635 (228h34m41.678632388s) [noise=true , address=45.91.116.9:4263 , piece_id=Z7MGGEQ63C5QOKI7BAAQCCJPJPLBM2MG2MBROD7GL4RRNCFVJJOQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:10.004031905 (228h34m41.699880668s) [noise=true , address=45.91.116.6:4973 , piece_id=2R4VUB2FG7IIY3KSBYSZFU7VTBGNS662DQ2AFOH7KDHNYSBGNUBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:10.072857382 (228h34m41.768706145s) [noise=true , address=77.90.31.10:4520 , piece_id=QFF5PLOA5UXWFJCDC3FYFVN4YJQX3GGVVUH3ACCNJ4LB5FT3BPAQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: <nil> Nov 9 00:15:36.939056286 (228h35m8.634905039s) [noise=true , address=145.79.227.28:4726 , piece_id=EGLGRHWIZETHAYGPUIMSPEGOB3HXC2YMC32EGJIXSW7TQ4KGM2GA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, address:145.79.227.28:4726): protocol: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43478->145.79.227.28:4726: use of closed network connection Nov 9 00:15:36.939107635 (228h35m8.634956398s) [noise=true , address=145.79.227.27:4690 , piece_id=ZQ2KUEEUTKDO44XPN5YBKH6BUTLWA7TSCZRJ3QLLGTT4DWYHGP6Q, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44312->145.79.227.27:4690: use of closed network connection Nov 9 00:15:36.939340538 (228h35m8.635189292s) [noise=true , address=145.79.227.10:4401 , piece_id=FKLG63DNHBF43ZT3MDH2D6GEDURMW6XA335EZND2LSQITYI5TYHQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40318->145.79.227.10:4401: use of closed network connection Nov 9 00:15:36.939371116 (228h35m8.635219879s) [noise=true , address=145.79.227.15:4876 , piece_id=5XA3I6VEOLUYSQK374SW52H5LNMHEZX2HVOC6L6FFMAKGJV4RXTQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54874->145.79.227.15:4876: use of closed network connection Nov 9 00:15:36.939473351 (228h35m8.635322104s) [noise=true , address=145.79.227.21:4473 , piece_id=3UNC3F6P7ERWHHZQTVSCIP4SBO6XL72I5F4ID5SOCIKTRFEWM6XQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40864->145.79.227.21:4473: use of closed network connection Nov 9 00:15:36.939551320 (228h35m8.635400073s) [noise=true , address=145.79.227.22:4509 , piece_id=QWLYFFMJSIQLRC5OWIOTWLIYUOZY34VJDQTMQJOXTCQ4DWYFCBVA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 00:15:36.939605372 (228h35m8.635454125s) [noise=true , address=145.79.227.8:4330 , piece_id=M7O6MM4DRNV7Y63AGYACAKZ6674HNG777XVUGEAZ7XVEDIBBYTVQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33426->145.79.227.8:4330: use of closed network connection Nov 9 00:15:36.939692238 (228h35m8.635540991s) [noise=true , address=145.79.227.8:4331 , piece_id=RDWGYWFEQCZIPKTSAG235PUJJDIP4BCDNJTX237RLFPTE2C45IBA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36256->145.79.227.8:4331: use of closed network connection Nov 9 00:15:36.939816855 (228h35m8.635665608s) [noise=true , address=145.79.227.5:4222 , piece_id=F26B25SIZ7TBDEKYDU34P477KIHHJUIHSYKRHGHOKKJVSQNNC3FA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44504->145.79.227.5:4222: use of closed network connection Nov 9 00:15:36.939964185 (228h35m8.635812938s) [noise=true , address=145.79.227.9:4365 , piece_id=GYEVA6LDJIQJJ6AVSTAXAVD2JDQ6CBQERQJJMBTW5OV4EZMVTOKA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45216->145.79.227.9:4365: use of closed network connection Nov 9 00:15:36.940093301 (228h35m8.635942054s) [noise=true , address=145.79.227.13:4804 , piece_id=MTPRWDDQQ5LMOTNCPOGKDH5IA57BICDAWSNOQ3H6BEBSZJSDOICQ, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40310->145.79.227.13:4804: use of closed network connection Nov 9 00:15:36.940217358 (228h35m8.636066111s) [noise=true , address=145.79.227.25:4618 , piece_id=J7WJIE3QGIMDMAPSGEDMOUN23GX356DCB3LGFN7XJUNYK6CCLTMA, seg_pos={0 0}, upload=201217]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52014->145.79.227.25:4618: use of closed network connection Nov 9 00:15:36.940262043 (228h35m8.636110786s) [seg_pos={0 0}, upload=201217]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 9 00:15:37.181912046 (228h35m8.877760799s) [upload=201217]: Done uploading segments... Nov 9 00:15:37.181957073 (228h35m8.877805816s) [upload=201217]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.711762838s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)