Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
6.031206454s trace:
https://jaeger.dev.storj.io/trace/66a9c91bc4a6ccf0
FAIL (2025-12-18 18:03:39 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:41370->45.38.24.2:443: read: connection reset by peer
Duration: 6.031233396s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
6.085976139s trace:
https://jaeger.dev.storj.io/trace/66a9c91bc4a6ccf0
FAIL (2025-12-18 18:03:18 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:41370->45.38.24.2:443: read: connection reset by peer
Duration: 6.086003491s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [...........................................X.XXX............] duration:
5.997804926s trace:
https://jaeger.dev.storj.io/trace/55e899dcc96ebc44
FAIL (2025-12-18 18:03:08 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: UDNDAZHIL3CV4LIC3RUFACBHCTQHF6PL6EGDJLWM4FSMYUXC4SQA) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: JYBWCPDYGATJSR3H2ISPX2KB4ZKBVIETIOEGRNSZZLQDE7IPAJHQ) context deadline exceeded
======LOGS======
Dec 18 17:53:46.907446718 (314h24m46.778529753s) [upload=275696]: Starting upload
Dec 18 17:53:46.907512523 (314h24m46.778595578s) [upload=275696]: Done starting upload
Dec 18 17:53:46.907634715 (314h24m46.77871775s) [upload=275696]: Uploading segments...
Dec 18 17:53:46.907721671 (314h24m46.778804716s) [upload=275696]: Got next segment. Inline: false
Dec 18 17:53:46.907766126 (314h24m46.778849161s) [upload=275696]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 18 17:53:46.907771726 (314h24m46.778854761s) [upload=275696]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 18 17:53:47.154892897 (314h24m47.025975943s) [seg_pos={0 0}, upload=275696]: Begin upload segment...
Dec 18 17:53:47.154946931 (314h24m47.026029976s) [seg_pos={0 0}, upload=275696]: Done begin upload segment.
Dec 18 17:53:47.154978721 (314h24m47.026061756s) [noise=true , address=145.79.227.42:4122 , piece_id=A245U2W7Y4XGA5JNGKPWBOM2HDQTOIZPMYF6GEZXZGJGIPCM4JPQ, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.155041712 (314h24m47.026124747s) [noise=true , address=77.90.31.21:4880 , piece_id=EDFGJA5GIV2HY2KV55BDGMRZGMV3QDJ6OM5BOODGX4RJZBAAPAIA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.155513359 (314h24m47.026596394s) [noise=true , address=145.79.227.52:4446 , piece_id=KOGZIRIVHIRYRX6B4HPS27EZKTPK4AK45HM644HLBOA5QUJLDB6A, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.155877683 (314h24m47.026960718s) [noise=true , address=145.79.227.37:4566 , piece_id=TUE5OHICYNTQQ26VILHTZXFMTIGWVUQS7JMYJEARRYVUT6CMCJPA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.156232409 (314h24m47.027315454s) [noise=true , address=145.79.227.14:4839 , piece_id=PW76MJZTJNUDTLCJ4GGQPLSEAJENKJYQOU5T62TAGCFTGQBNGFTA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.156821220 (314h24m47.027904265s) [noise=true , address=145.79.227.27:4691 , piece_id=4ROXPEVMSBJE7EF5MHM6RBEQKOPVGHPVV5RCCNGX6R2CKWWLIV7Q, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.157223125 (314h24m47.02830616s) [noise=true , address=45.91.116.10:4081 , piece_id=JYBWCPDYGATJSR3H2ISPX2KB4ZKBVIETIOEGRNSZZLQDE7IPAJHQ, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.157618027 (314h24m47.028701062s) [noise=true , address=145.79.227.49:4339 , piece_id=CGSAGFA7UHNQ53CLLYZ2NMD57JK25P2YVPCMKW27W7HIE2OWI3UA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.158280779 (314h24m47.029363814s) [noise=true , address=45.91.116.11:4120 , piece_id=UDNDAZHIL3CV4LIC3RUFACBHCTQHF6PL6EGDJLWM4FSMYUXC4SQA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.159585022 (314h24m47.030668067s) [noise=true , address=145.79.227.41:4087 , piece_id=NKF2BL3HHWERKL5DPZ5SH73LS7XXT325TFWS3BSSXHETFMLL6VCQ, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.161483899 (314h24m47.032566944s) [noise=true , address=145.79.227.51:4411 , piece_id=RIEZSDOKZCRKA7KX775JPXEZVRZWIGOEAG62LSBRWPEVP4AQLUTA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.161534034 (314h24m47.032617079s) [noise=true , address=145.79.227.35:4481 , piece_id=ONU6UGBZKGYZN2KSAS57OCCKTFMDT5HMDCVJU5EIDYTENVRVOI2Q, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.163305847 (314h24m47.034388893s) [noise=true , address=145.79.227.45:4229 , piece_id=RDOEHHL2QNKQWZIRPWLOY6HW47OI4ZDXIPJHTR76BVQ3EFHZMKLA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.165089595 (314h24m47.03617264s) [noise=true , address=145.79.227.46:4601 , piece_id=3QTBNM4CT6QE74FQCOBDICM5UYXW32REQ25YEKQWXAVDC46RQBCA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.166839817 (314h24m47.037922862s) [noise=true , address=145.79.227.44:4195 , piece_id=3FIM4BHP2C62BSJXHU7GVYQ3DOC5YL3KOGDHNRMRXM3MJ6MWOQMQ, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.168326769 (314h24m47.039409814s) [noise=true , address=145.79.227.31:4798 , piece_id=W2VFZPXKGVORU5JCQIIP2IR5SCSEMWYRWJOKHFHBI3V7WCSMLBWA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:48.203795023 (314h24m48.074878059s) [noise=true , address=145.79.227.44:4195 , piece_id=3FIM4BHP2C62BSJXHU7GVYQ3DOC5YL3KOGDHNRMRXM3MJ6MWOQMQ, seg_pos={0 0}, upload=275696]: Done uploading piece. err: ecclient: upload failed (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, address:145.79.227.44:4195): protocol: storage node overloaded, request limit: 2000
Dec 18 17:53:48.207081231 (314h24m48.078164276s) [noise=true , address=145.79.227.31:4798 , piece_id=W2VFZPXKGVORU5JCQIIP2IR5SCSEMWYRWJOKHFHBI3V7WCSMLBWA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: storage node overloaded, request limit: 2000
Dec 18 17:53:48.863464901 (314h24m48.734547956s) [upload=275696]: Next returned nil segment
Dec 18 17:53:48.863489558 (314h24m48.734572623s) [upload=275696]: Waiting for error group managing segments...
Dec 18 17:53:49.149217626 (314h24m49.020300661s) [noise=true , address=145.79.227.51:4411 , piece_id=RIEZSDOKZCRKA7KX775JPXEZVRZWIGOEAG62LSBRWPEVP4AQLUTA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.151240638 (314h24m49.022323683s) [noise=true , address=145.79.227.52:4446 , piece_id=KOGZIRIVHIRYRX6B4HPS27EZKTPK4AK45HM644HLBOA5QUJLDB6A, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.159687424 (314h24m49.030770459s) [noise=true , address=145.79.227.46:4601 , piece_id=3QTBNM4CT6QE74FQCOBDICM5UYXW32REQ25YEKQWXAVDC46RQBCA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.163440961 (314h24m49.034524006s) [noise=true , address=145.79.227.37:4566 , piece_id=TUE5OHICYNTQQ26VILHTZXFMTIGWVUQS7JMYJEARRYVUT6CMCJPA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.166507440 (314h24m49.037590475s) [noise=true , address=145.79.227.49:4339 , piece_id=CGSAGFA7UHNQ53CLLYZ2NMD57JK25P2YVPCMKW27W7HIE2OWI3UA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.183896243 (314h24m49.054979288s) [noise=true , address=145.79.227.41:4087 , piece_id=NKF2BL3HHWERKL5DPZ5SH73LS7XXT325TFWS3BSSXHETFMLL6VCQ, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.219629698 (314h24m49.090712733s) [noise=true , address=145.79.227.45:4229 , piece_id=RDOEHHL2QNKQWZIRPWLOY6HW47OI4ZDXIPJHTR76BVQ3EFHZMKLA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.229654178 (314h24m49.100737213s) [noise=true , address=145.79.227.35:4481 , piece_id=ONU6UGBZKGYZN2KSAS57OCCKTFMDT5HMDCVJU5EIDYTENVRVOI2Q, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.348528404 (314h24m49.219611439s) [noise=true , address=145.79.227.42:4122 , piece_id=A245U2W7Y4XGA5JNGKPWBOM2HDQTOIZPMYF6GEZXZGJGIPCM4JPQ, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.361664419 (314h24m49.232747454s) [noise=true , address=45.91.116.11:4120 , piece_id=UDNDAZHIL3CV4LIC3RUFACBHCTQHF6PL6EGDJLWM4FSMYUXC4SQA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.524862349 (314h24m49.395945394s) [noise=true , address=45.91.116.10:4081 , piece_id=JYBWCPDYGATJSR3H2ISPX2KB4ZKBVIETIOEGRNSZZLQDE7IPAJHQ, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.698660224 (314h24m49.569743279s) [noise=true , address=145.79.227.14:4839 , piece_id=PW76MJZTJNUDTLCJ4GGQPLSEAJENKJYQOU5T62TAGCFTGQBNGFTA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:54:13.109258041 (314h25m12.980341086s) [noise=true , address=145.79.227.27:4691 , piece_id=4ROXPEVMSBJE7EF5MHM6RBEQKOPVGHPVV5RCCNGX6R2CKWWLIV7Q, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:54:13.109281937 (314h25m12.980364982s) [seg_pos={0 0}, upload=275696]: Segment reached optimal threshold of 13 pieces.
Dec 18 17:54:13.109482640 (314h25m12.980565705s) [noise=true , address=77.90.31.21:4880 , piece_id=EDFGJA5GIV2HY2KV55BDGMRZGMV3QDJ6OM5BOODGX4RJZBAAPAIA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF
Dec 18 17:54:13.109493139 (314h25m12.980576174s) [noise=true , address=77.90.31.21:4880 , piece_id=EDFGJA5GIV2HY2KV55BDGMRZGMV3QDJ6OM5BOODGX4RJZBAAPAIA, seg_pos={0 0}, upload=275696]: Long tail cancellation due to optimal threshold
Dec 18 17:54:13.109515152 (314h25m12.980598187s) [seg_pos={0 0}, upload=275696]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 18 17:54:13.109598921 (314h25m12.980681966s) [upload=275696]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 18 17:54:13.109606365 (314h25m12.9806894s) [upload=275696]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 18 17:54:13.397754392 (314h25m13.268837437s) [upload=275696]: Done uploading segments...
Dec 18 17:54:13.397776934 (314h25m13.268859969s) [upload=275696]: Upload finished. err: <nil>
Duration: 5.997834212s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
6.091935426s trace:
https://jaeger.dev.storj.io/trace/66a9c91bc4a6ccf0
FAIL (2025-12-18 18:03:01 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:41370->45.38.24.2:443: read: connection reset by peer
Duration: 6.091965252s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [............................................X.XXX...........] duration:
5.831108817s trace:
https://jaeger.dev.storj.io/trace/55e899dcc96ebc44
FAIL (2025-12-18 18:02:30 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: UDNDAZHIL3CV4LIC3RUFACBHCTQHF6PL6EGDJLWM4FSMYUXC4SQA) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: JYBWCPDYGATJSR3H2ISPX2KB4ZKBVIETIOEGRNSZZLQDE7IPAJHQ) context deadline exceeded
======LOGS======
Dec 18 17:53:46.907446718 (314h24m46.778529753s) [upload=275696]: Starting upload
Dec 18 17:53:46.907512523 (314h24m46.778595578s) [upload=275696]: Done starting upload
Dec 18 17:53:46.907634715 (314h24m46.77871775s) [upload=275696]: Uploading segments...
Dec 18 17:53:46.907721671 (314h24m46.778804716s) [upload=275696]: Got next segment. Inline: false
Dec 18 17:53:46.907766126 (314h24m46.778849161s) [upload=275696]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 18 17:53:46.907771726 (314h24m46.778854761s) [upload=275696]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 18 17:53:47.154892897 (314h24m47.025975943s) [seg_pos={0 0}, upload=275696]: Begin upload segment...
Dec 18 17:53:47.154946931 (314h24m47.026029976s) [seg_pos={0 0}, upload=275696]: Done begin upload segment.
Dec 18 17:53:47.154978721 (314h24m47.026061756s) [noise=true , address=145.79.227.42:4122 , piece_id=A245U2W7Y4XGA5JNGKPWBOM2HDQTOIZPMYF6GEZXZGJGIPCM4JPQ, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.155041712 (314h24m47.026124747s) [noise=true , address=77.90.31.21:4880 , piece_id=EDFGJA5GIV2HY2KV55BDGMRZGMV3QDJ6OM5BOODGX4RJZBAAPAIA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.155513359 (314h24m47.026596394s) [noise=true , address=145.79.227.52:4446 , piece_id=KOGZIRIVHIRYRX6B4HPS27EZKTPK4AK45HM644HLBOA5QUJLDB6A, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.155877683 (314h24m47.026960718s) [noise=true , address=145.79.227.37:4566 , piece_id=TUE5OHICYNTQQ26VILHTZXFMTIGWVUQS7JMYJEARRYVUT6CMCJPA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.156232409 (314h24m47.027315454s) [noise=true , address=145.79.227.14:4839 , piece_id=PW76MJZTJNUDTLCJ4GGQPLSEAJENKJYQOU5T62TAGCFTGQBNGFTA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.156821220 (314h24m47.027904265s) [noise=true , address=145.79.227.27:4691 , piece_id=4ROXPEVMSBJE7EF5MHM6RBEQKOPVGHPVV5RCCNGX6R2CKWWLIV7Q, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.157223125 (314h24m47.02830616s) [noise=true , address=45.91.116.10:4081 , piece_id=JYBWCPDYGATJSR3H2ISPX2KB4ZKBVIETIOEGRNSZZLQDE7IPAJHQ, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.157618027 (314h24m47.028701062s) [noise=true , address=145.79.227.49:4339 , piece_id=CGSAGFA7UHNQ53CLLYZ2NMD57JK25P2YVPCMKW27W7HIE2OWI3UA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.158280779 (314h24m47.029363814s) [noise=true , address=45.91.116.11:4120 , piece_id=UDNDAZHIL3CV4LIC3RUFACBHCTQHF6PL6EGDJLWM4FSMYUXC4SQA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.159585022 (314h24m47.030668067s) [noise=true , address=145.79.227.41:4087 , piece_id=NKF2BL3HHWERKL5DPZ5SH73LS7XXT325TFWS3BSSXHETFMLL6VCQ, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.161483899 (314h24m47.032566944s) [noise=true , address=145.79.227.51:4411 , piece_id=RIEZSDOKZCRKA7KX775JPXEZVRZWIGOEAG62LSBRWPEVP4AQLUTA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.161534034 (314h24m47.032617079s) [noise=true , address=145.79.227.35:4481 , piece_id=ONU6UGBZKGYZN2KSAS57OCCKTFMDT5HMDCVJU5EIDYTENVRVOI2Q, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.163305847 (314h24m47.034388893s) [noise=true , address=145.79.227.45:4229 , piece_id=RDOEHHL2QNKQWZIRPWLOY6HW47OI4ZDXIPJHTR76BVQ3EFHZMKLA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.165089595 (314h24m47.03617264s) [noise=true , address=145.79.227.46:4601 , piece_id=3QTBNM4CT6QE74FQCOBDICM5UYXW32REQ25YEKQWXAVDC46RQBCA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.166839817 (314h24m47.037922862s) [noise=true , address=145.79.227.44:4195 , piece_id=3FIM4BHP2C62BSJXHU7GVYQ3DOC5YL3KOGDHNRMRXM3MJ6MWOQMQ, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.168326769 (314h24m47.039409814s) [noise=true , address=145.79.227.31:4798 , piece_id=W2VFZPXKGVORU5JCQIIP2IR5SCSEMWYRWJOKHFHBI3V7WCSMLBWA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:48.203795023 (314h24m48.074878059s) [noise=true , address=145.79.227.44:4195 , piece_id=3FIM4BHP2C62BSJXHU7GVYQ3DOC5YL3KOGDHNRMRXM3MJ6MWOQMQ, seg_pos={0 0}, upload=275696]: Done uploading piece. err: ecclient: upload failed (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, address:145.79.227.44:4195): protocol: storage node overloaded, request limit: 2000
Dec 18 17:53:48.207081231 (314h24m48.078164276s) [noise=true , address=145.79.227.31:4798 , piece_id=W2VFZPXKGVORU5JCQIIP2IR5SCSEMWYRWJOKHFHBI3V7WCSMLBWA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: storage node overloaded, request limit: 2000
Dec 18 17:53:48.863464901 (314h24m48.734547956s) [upload=275696]: Next returned nil segment
Dec 18 17:53:48.863489558 (314h24m48.734572623s) [upload=275696]: Waiting for error group managing segments...
Dec 18 17:53:49.149217626 (314h24m49.020300661s) [noise=true , address=145.79.227.51:4411 , piece_id=RIEZSDOKZCRKA7KX775JPXEZVRZWIGOEAG62LSBRWPEVP4AQLUTA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.151240638 (314h24m49.022323683s) [noise=true , address=145.79.227.52:4446 , piece_id=KOGZIRIVHIRYRX6B4HPS27EZKTPK4AK45HM644HLBOA5QUJLDB6A, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.159687424 (314h24m49.030770459s) [noise=true , address=145.79.227.46:4601 , piece_id=3QTBNM4CT6QE74FQCOBDICM5UYXW32REQ25YEKQWXAVDC46RQBCA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.163440961 (314h24m49.034524006s) [noise=true , address=145.79.227.37:4566 , piece_id=TUE5OHICYNTQQ26VILHTZXFMTIGWVUQS7JMYJEARRYVUT6CMCJPA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.166507440 (314h24m49.037590475s) [noise=true , address=145.79.227.49:4339 , piece_id=CGSAGFA7UHNQ53CLLYZ2NMD57JK25P2YVPCMKW27W7HIE2OWI3UA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.183896243 (314h24m49.054979288s) [noise=true , address=145.79.227.41:4087 , piece_id=NKF2BL3HHWERKL5DPZ5SH73LS7XXT325TFWS3BSSXHETFMLL6VCQ, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.219629698 (314h24m49.090712733s) [noise=true , address=145.79.227.45:4229 , piece_id=RDOEHHL2QNKQWZIRPWLOY6HW47OI4ZDXIPJHTR76BVQ3EFHZMKLA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.229654178 (314h24m49.100737213s) [noise=true , address=145.79.227.35:4481 , piece_id=ONU6UGBZKGYZN2KSAS57OCCKTFMDT5HMDCVJU5EIDYTENVRVOI2Q, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.348528404 (314h24m49.219611439s) [noise=true , address=145.79.227.42:4122 , piece_id=A245U2W7Y4XGA5JNGKPWBOM2HDQTOIZPMYF6GEZXZGJGIPCM4JPQ, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.361664419 (314h24m49.232747454s) [noise=true , address=45.91.116.11:4120 , piece_id=UDNDAZHIL3CV4LIC3RUFACBHCTQHF6PL6EGDJLWM4FSMYUXC4SQA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.524862349 (314h24m49.395945394s) [noise=true , address=45.91.116.10:4081 , piece_id=JYBWCPDYGATJSR3H2ISPX2KB4ZKBVIETIOEGRNSZZLQDE7IPAJHQ, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.698660224 (314h24m49.569743279s) [noise=true , address=145.79.227.14:4839 , piece_id=PW76MJZTJNUDTLCJ4GGQPLSEAJENKJYQOU5T62TAGCFTGQBNGFTA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:54:13.109258041 (314h25m12.980341086s) [noise=true , address=145.79.227.27:4691 , piece_id=4ROXPEVMSBJE7EF5MHM6RBEQKOPVGHPVV5RCCNGX6R2CKWWLIV7Q, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:54:13.109281937 (314h25m12.980364982s) [seg_pos={0 0}, upload=275696]: Segment reached optimal threshold of 13 pieces.
Dec 18 17:54:13.109482640 (314h25m12.980565705s) [noise=true , address=77.90.31.21:4880 , piece_id=EDFGJA5GIV2HY2KV55BDGMRZGMV3QDJ6OM5BOODGX4RJZBAAPAIA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF
Dec 18 17:54:13.109493139 (314h25m12.980576174s) [noise=true , address=77.90.31.21:4880 , piece_id=EDFGJA5GIV2HY2KV55BDGMRZGMV3QDJ6OM5BOODGX4RJZBAAPAIA, seg_pos={0 0}, upload=275696]: Long tail cancellation due to optimal threshold
Dec 18 17:54:13.109515152 (314h25m12.980598187s) [seg_pos={0 0}, upload=275696]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 18 17:54:13.109598921 (314h25m12.980681966s) [upload=275696]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 18 17:54:13.109606365 (314h25m12.9806894s) [upload=275696]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 18 17:54:13.397754392 (314h25m13.268837437s) [upload=275696]: Done uploading segments...
Dec 18 17:54:13.397776934 (314h25m13.268859969s) [upload=275696]: Upload finished. err: <nil>
Duration: 5.831138684s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.............................................X.XXX..........] duration:
5.767773566s trace:
https://jaeger.dev.storj.io/trace/55e899dcc96ebc44
FAIL (2025-12-18 18:02:08 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: UDNDAZHIL3CV4LIC3RUFACBHCTQHF6PL6EGDJLWM4FSMYUXC4SQA) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: JYBWCPDYGATJSR3H2ISPX2KB4ZKBVIETIOEGRNSZZLQDE7IPAJHQ) context deadline exceeded
======LOGS======
Dec 18 17:53:46.907446718 (314h24m46.778529753s) [upload=275696]: Starting upload
Dec 18 17:53:46.907512523 (314h24m46.778595578s) [upload=275696]: Done starting upload
Dec 18 17:53:46.907634715 (314h24m46.77871775s) [upload=275696]: Uploading segments...
Dec 18 17:53:46.907721671 (314h24m46.778804716s) [upload=275696]: Got next segment. Inline: false
Dec 18 17:53:46.907766126 (314h24m46.778849161s) [upload=275696]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 18 17:53:46.907771726 (314h24m46.778854761s) [upload=275696]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 18 17:53:47.154892897 (314h24m47.025975943s) [seg_pos={0 0}, upload=275696]: Begin upload segment...
Dec 18 17:53:47.154946931 (314h24m47.026029976s) [seg_pos={0 0}, upload=275696]: Done begin upload segment.
Dec 18 17:53:47.154978721 (314h24m47.026061756s) [noise=true , address=145.79.227.42:4122 , piece_id=A245U2W7Y4XGA5JNGKPWBOM2HDQTOIZPMYF6GEZXZGJGIPCM4JPQ, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.155041712 (314h24m47.026124747s) [noise=true , address=77.90.31.21:4880 , piece_id=EDFGJA5GIV2HY2KV55BDGMRZGMV3QDJ6OM5BOODGX4RJZBAAPAIA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.155513359 (314h24m47.026596394s) [noise=true , address=145.79.227.52:4446 , piece_id=KOGZIRIVHIRYRX6B4HPS27EZKTPK4AK45HM644HLBOA5QUJLDB6A, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.155877683 (314h24m47.026960718s) [noise=true , address=145.79.227.37:4566 , piece_id=TUE5OHICYNTQQ26VILHTZXFMTIGWVUQS7JMYJEARRYVUT6CMCJPA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.156232409 (314h24m47.027315454s) [noise=true , address=145.79.227.14:4839 , piece_id=PW76MJZTJNUDTLCJ4GGQPLSEAJENKJYQOU5T62TAGCFTGQBNGFTA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.156821220 (314h24m47.027904265s) [noise=true , address=145.79.227.27:4691 , piece_id=4ROXPEVMSBJE7EF5MHM6RBEQKOPVGHPVV5RCCNGX6R2CKWWLIV7Q, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.157223125 (314h24m47.02830616s) [noise=true , address=45.91.116.10:4081 , piece_id=JYBWCPDYGATJSR3H2ISPX2KB4ZKBVIETIOEGRNSZZLQDE7IPAJHQ, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.157618027 (314h24m47.028701062s) [noise=true , address=145.79.227.49:4339 , piece_id=CGSAGFA7UHNQ53CLLYZ2NMD57JK25P2YVPCMKW27W7HIE2OWI3UA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.158280779 (314h24m47.029363814s) [noise=true , address=45.91.116.11:4120 , piece_id=UDNDAZHIL3CV4LIC3RUFACBHCTQHF6PL6EGDJLWM4FSMYUXC4SQA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.159585022 (314h24m47.030668067s) [noise=true , address=145.79.227.41:4087 , piece_id=NKF2BL3HHWERKL5DPZ5SH73LS7XXT325TFWS3BSSXHETFMLL6VCQ, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.161483899 (314h24m47.032566944s) [noise=true , address=145.79.227.51:4411 , piece_id=RIEZSDOKZCRKA7KX775JPXEZVRZWIGOEAG62LSBRWPEVP4AQLUTA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.161534034 (314h24m47.032617079s) [noise=true , address=145.79.227.35:4481 , piece_id=ONU6UGBZKGYZN2KSAS57OCCKTFMDT5HMDCVJU5EIDYTENVRVOI2Q, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.163305847 (314h24m47.034388893s) [noise=true , address=145.79.227.45:4229 , piece_id=RDOEHHL2QNKQWZIRPWLOY6HW47OI4ZDXIPJHTR76BVQ3EFHZMKLA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.165089595 (314h24m47.03617264s) [noise=true , address=145.79.227.46:4601 , piece_id=3QTBNM4CT6QE74FQCOBDICM5UYXW32REQ25YEKQWXAVDC46RQBCA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.166839817 (314h24m47.037922862s) [noise=true , address=145.79.227.44:4195 , piece_id=3FIM4BHP2C62BSJXHU7GVYQ3DOC5YL3KOGDHNRMRXM3MJ6MWOQMQ, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.168326769 (314h24m47.039409814s) [noise=true , address=145.79.227.31:4798 , piece_id=W2VFZPXKGVORU5JCQIIP2IR5SCSEMWYRWJOKHFHBI3V7WCSMLBWA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:48.203795023 (314h24m48.074878059s) [noise=true , address=145.79.227.44:4195 , piece_id=3FIM4BHP2C62BSJXHU7GVYQ3DOC5YL3KOGDHNRMRXM3MJ6MWOQMQ, seg_pos={0 0}, upload=275696]: Done uploading piece. err: ecclient: upload failed (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, address:145.79.227.44:4195): protocol: storage node overloaded, request limit: 2000
Dec 18 17:53:48.207081231 (314h24m48.078164276s) [noise=true , address=145.79.227.31:4798 , piece_id=W2VFZPXKGVORU5JCQIIP2IR5SCSEMWYRWJOKHFHBI3V7WCSMLBWA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: storage node overloaded, request limit: 2000
Dec 18 17:53:48.863464901 (314h24m48.734547956s) [upload=275696]: Next returned nil segment
Dec 18 17:53:48.863489558 (314h24m48.734572623s) [upload=275696]: Waiting for error group managing segments...
Dec 18 17:53:49.149217626 (314h24m49.020300661s) [noise=true , address=145.79.227.51:4411 , piece_id=RIEZSDOKZCRKA7KX775JPXEZVRZWIGOEAG62LSBRWPEVP4AQLUTA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.151240638 (314h24m49.022323683s) [noise=true , address=145.79.227.52:4446 , piece_id=KOGZIRIVHIRYRX6B4HPS27EZKTPK4AK45HM644HLBOA5QUJLDB6A, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.159687424 (314h24m49.030770459s) [noise=true , address=145.79.227.46:4601 , piece_id=3QTBNM4CT6QE74FQCOBDICM5UYXW32REQ25YEKQWXAVDC46RQBCA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.163440961 (314h24m49.034524006s) [noise=true , address=145.79.227.37:4566 , piece_id=TUE5OHICYNTQQ26VILHTZXFMTIGWVUQS7JMYJEARRYVUT6CMCJPA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.166507440 (314h24m49.037590475s) [noise=true , address=145.79.227.49:4339 , piece_id=CGSAGFA7UHNQ53CLLYZ2NMD57JK25P2YVPCMKW27W7HIE2OWI3UA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.183896243 (314h24m49.054979288s) [noise=true , address=145.79.227.41:4087 , piece_id=NKF2BL3HHWERKL5DPZ5SH73LS7XXT325TFWS3BSSXHETFMLL6VCQ, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.219629698 (314h24m49.090712733s) [noise=true , address=145.79.227.45:4229 , piece_id=RDOEHHL2QNKQWZIRPWLOY6HW47OI4ZDXIPJHTR76BVQ3EFHZMKLA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.229654178 (314h24m49.100737213s) [noise=true , address=145.79.227.35:4481 , piece_id=ONU6UGBZKGYZN2KSAS57OCCKTFMDT5HMDCVJU5EIDYTENVRVOI2Q, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.348528404 (314h24m49.219611439s) [noise=true , address=145.79.227.42:4122 , piece_id=A245U2W7Y4XGA5JNGKPWBOM2HDQTOIZPMYF6GEZXZGJGIPCM4JPQ, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.361664419 (314h24m49.232747454s) [noise=true , address=45.91.116.11:4120 , piece_id=UDNDAZHIL3CV4LIC3RUFACBHCTQHF6PL6EGDJLWM4FSMYUXC4SQA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.524862349 (314h24m49.395945394s) [noise=true , address=45.91.116.10:4081 , piece_id=JYBWCPDYGATJSR3H2ISPX2KB4ZKBVIETIOEGRNSZZLQDE7IPAJHQ, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.698660224 (314h24m49.569743279s) [noise=true , address=145.79.227.14:4839 , piece_id=PW76MJZTJNUDTLCJ4GGQPLSEAJENKJYQOU5T62TAGCFTGQBNGFTA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:54:13.109258041 (314h25m12.980341086s) [noise=true , address=145.79.227.27:4691 , piece_id=4ROXPEVMSBJE7EF5MHM6RBEQKOPVGHPVV5RCCNGX6R2CKWWLIV7Q, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:54:13.109281937 (314h25m12.980364982s) [seg_pos={0 0}, upload=275696]: Segment reached optimal threshold of 13 pieces.
Dec 18 17:54:13.109482640 (314h25m12.980565705s) [noise=true , address=77.90.31.21:4880 , piece_id=EDFGJA5GIV2HY2KV55BDGMRZGMV3QDJ6OM5BOODGX4RJZBAAPAIA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF
Dec 18 17:54:13.109493139 (314h25m12.980576174s) [noise=true , address=77.90.31.21:4880 , piece_id=EDFGJA5GIV2HY2KV55BDGMRZGMV3QDJ6OM5BOODGX4RJZBAAPAIA, seg_pos={0 0}, upload=275696]: Long tail cancellation due to optimal threshold
Dec 18 17:54:13.109515152 (314h25m12.980598187s) [seg_pos={0 0}, upload=275696]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 18 17:54:13.109598921 (314h25m12.980681966s) [upload=275696]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 18 17:54:13.109606365 (314h25m12.9806894s) [upload=275696]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 18 17:54:13.397754392 (314h25m13.268837437s) [upload=275696]: Done uploading segments...
Dec 18 17:54:13.397776934 (314h25m13.268859969s) [upload=275696]: Upload finished. err: <nil>
Duration: 5.767802792s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
5.9809084s trace:
https://jaeger.dev.storj.io/trace/66a9c91bc4a6ccf0
FAIL (2025-12-18 18:02:04 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:41370->45.38.24.2:443: read: connection reset by peer
Duration: 5.980942504s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
5.766835536s trace:
https://jaeger.dev.storj.io/trace/66a9c91bc4a6ccf0
FAIL (2025-12-18 18:01:31 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:41370->45.38.24.2:443: read: connection reset by peer
Duration: 5.766878618s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [..............................................X.XXX.........] duration:
6.485043705s trace:
https://jaeger.dev.storj.io/trace/55e899dcc96ebc44
FAIL (2025-12-18 18:00:58 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: UDNDAZHIL3CV4LIC3RUFACBHCTQHF6PL6EGDJLWM4FSMYUXC4SQA) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: JYBWCPDYGATJSR3H2ISPX2KB4ZKBVIETIOEGRNSZZLQDE7IPAJHQ) context deadline exceeded
======LOGS======
Dec 18 17:53:46.907446718 (314h24m46.778529753s) [upload=275696]: Starting upload
Dec 18 17:53:46.907512523 (314h24m46.778595578s) [upload=275696]: Done starting upload
Dec 18 17:53:46.907634715 (314h24m46.77871775s) [upload=275696]: Uploading segments...
Dec 18 17:53:46.907721671 (314h24m46.778804716s) [upload=275696]: Got next segment. Inline: false
Dec 18 17:53:46.907766126 (314h24m46.778849161s) [upload=275696]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 18 17:53:46.907771726 (314h24m46.778854761s) [upload=275696]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 18 17:53:47.154892897 (314h24m47.025975943s) [seg_pos={0 0}, upload=275696]: Begin upload segment...
Dec 18 17:53:47.154946931 (314h24m47.026029976s) [seg_pos={0 0}, upload=275696]: Done begin upload segment.
Dec 18 17:53:47.154978721 (314h24m47.026061756s) [noise=true , address=145.79.227.42:4122 , piece_id=A245U2W7Y4XGA5JNGKPWBOM2HDQTOIZPMYF6GEZXZGJGIPCM4JPQ, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.155041712 (314h24m47.026124747s) [noise=true , address=77.90.31.21:4880 , piece_id=EDFGJA5GIV2HY2KV55BDGMRZGMV3QDJ6OM5BOODGX4RJZBAAPAIA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.155513359 (314h24m47.026596394s) [noise=true , address=145.79.227.52:4446 , piece_id=KOGZIRIVHIRYRX6B4HPS27EZKTPK4AK45HM644HLBOA5QUJLDB6A, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.155877683 (314h24m47.026960718s) [noise=true , address=145.79.227.37:4566 , piece_id=TUE5OHICYNTQQ26VILHTZXFMTIGWVUQS7JMYJEARRYVUT6CMCJPA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.156232409 (314h24m47.027315454s) [noise=true , address=145.79.227.14:4839 , piece_id=PW76MJZTJNUDTLCJ4GGQPLSEAJENKJYQOU5T62TAGCFTGQBNGFTA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.156821220 (314h24m47.027904265s) [noise=true , address=145.79.227.27:4691 , piece_id=4ROXPEVMSBJE7EF5MHM6RBEQKOPVGHPVV5RCCNGX6R2CKWWLIV7Q, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.157223125 (314h24m47.02830616s) [noise=true , address=45.91.116.10:4081 , piece_id=JYBWCPDYGATJSR3H2ISPX2KB4ZKBVIETIOEGRNSZZLQDE7IPAJHQ, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.157618027 (314h24m47.028701062s) [noise=true , address=145.79.227.49:4339 , piece_id=CGSAGFA7UHNQ53CLLYZ2NMD57JK25P2YVPCMKW27W7HIE2OWI3UA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.158280779 (314h24m47.029363814s) [noise=true , address=45.91.116.11:4120 , piece_id=UDNDAZHIL3CV4LIC3RUFACBHCTQHF6PL6EGDJLWM4FSMYUXC4SQA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.159585022 (314h24m47.030668067s) [noise=true , address=145.79.227.41:4087 , piece_id=NKF2BL3HHWERKL5DPZ5SH73LS7XXT325TFWS3BSSXHETFMLL6VCQ, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.161483899 (314h24m47.032566944s) [noise=true , address=145.79.227.51:4411 , piece_id=RIEZSDOKZCRKA7KX775JPXEZVRZWIGOEAG62LSBRWPEVP4AQLUTA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.161534034 (314h24m47.032617079s) [noise=true , address=145.79.227.35:4481 , piece_id=ONU6UGBZKGYZN2KSAS57OCCKTFMDT5HMDCVJU5EIDYTENVRVOI2Q, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.163305847 (314h24m47.034388893s) [noise=true , address=145.79.227.45:4229 , piece_id=RDOEHHL2QNKQWZIRPWLOY6HW47OI4ZDXIPJHTR76BVQ3EFHZMKLA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.165089595 (314h24m47.03617264s) [noise=true , address=145.79.227.46:4601 , piece_id=3QTBNM4CT6QE74FQCOBDICM5UYXW32REQ25YEKQWXAVDC46RQBCA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.166839817 (314h24m47.037922862s) [noise=true , address=145.79.227.44:4195 , piece_id=3FIM4BHP2C62BSJXHU7GVYQ3DOC5YL3KOGDHNRMRXM3MJ6MWOQMQ, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.168326769 (314h24m47.039409814s) [noise=true , address=145.79.227.31:4798 , piece_id=W2VFZPXKGVORU5JCQIIP2IR5SCSEMWYRWJOKHFHBI3V7WCSMLBWA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:48.203795023 (314h24m48.074878059s) [noise=true , address=145.79.227.44:4195 , piece_id=3FIM4BHP2C62BSJXHU7GVYQ3DOC5YL3KOGDHNRMRXM3MJ6MWOQMQ, seg_pos={0 0}, upload=275696]: Done uploading piece. err: ecclient: upload failed (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, address:145.79.227.44:4195): protocol: storage node overloaded, request limit: 2000
Dec 18 17:53:48.207081231 (314h24m48.078164276s) [noise=true , address=145.79.227.31:4798 , piece_id=W2VFZPXKGVORU5JCQIIP2IR5SCSEMWYRWJOKHFHBI3V7WCSMLBWA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: storage node overloaded, request limit: 2000
Dec 18 17:53:48.863464901 (314h24m48.734547956s) [upload=275696]: Next returned nil segment
Dec 18 17:53:48.863489558 (314h24m48.734572623s) [upload=275696]: Waiting for error group managing segments...
Dec 18 17:53:49.149217626 (314h24m49.020300661s) [noise=true , address=145.79.227.51:4411 , piece_id=RIEZSDOKZCRKA7KX775JPXEZVRZWIGOEAG62LSBRWPEVP4AQLUTA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.151240638 (314h24m49.022323683s) [noise=true , address=145.79.227.52:4446 , piece_id=KOGZIRIVHIRYRX6B4HPS27EZKTPK4AK45HM644HLBOA5QUJLDB6A, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.159687424 (314h24m49.030770459s) [noise=true , address=145.79.227.46:4601 , piece_id=3QTBNM4CT6QE74FQCOBDICM5UYXW32REQ25YEKQWXAVDC46RQBCA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.163440961 (314h24m49.034524006s) [noise=true , address=145.79.227.37:4566 , piece_id=TUE5OHICYNTQQ26VILHTZXFMTIGWVUQS7JMYJEARRYVUT6CMCJPA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.166507440 (314h24m49.037590475s) [noise=true , address=145.79.227.49:4339 , piece_id=CGSAGFA7UHNQ53CLLYZ2NMD57JK25P2YVPCMKW27W7HIE2OWI3UA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.183896243 (314h24m49.054979288s) [noise=true , address=145.79.227.41:4087 , piece_id=NKF2BL3HHWERKL5DPZ5SH73LS7XXT325TFWS3BSSXHETFMLL6VCQ, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.219629698 (314h24m49.090712733s) [noise=true , address=145.79.227.45:4229 , piece_id=RDOEHHL2QNKQWZIRPWLOY6HW47OI4ZDXIPJHTR76BVQ3EFHZMKLA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.229654178 (314h24m49.100737213s) [noise=true , address=145.79.227.35:4481 , piece_id=ONU6UGBZKGYZN2KSAS57OCCKTFMDT5HMDCVJU5EIDYTENVRVOI2Q, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.348528404 (314h24m49.219611439s) [noise=true , address=145.79.227.42:4122 , piece_id=A245U2W7Y4XGA5JNGKPWBOM2HDQTOIZPMYF6GEZXZGJGIPCM4JPQ, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.361664419 (314h24m49.232747454s) [noise=true , address=45.91.116.11:4120 , piece_id=UDNDAZHIL3CV4LIC3RUFACBHCTQHF6PL6EGDJLWM4FSMYUXC4SQA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.524862349 (314h24m49.395945394s) [noise=true , address=45.91.116.10:4081 , piece_id=JYBWCPDYGATJSR3H2ISPX2KB4ZKBVIETIOEGRNSZZLQDE7IPAJHQ, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.698660224 (314h24m49.569743279s) [noise=true , address=145.79.227.14:4839 , piece_id=PW76MJZTJNUDTLCJ4GGQPLSEAJENKJYQOU5T62TAGCFTGQBNGFTA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:54:13.109258041 (314h25m12.980341086s) [noise=true , address=145.79.227.27:4691 , piece_id=4ROXPEVMSBJE7EF5MHM6RBEQKOPVGHPVV5RCCNGX6R2CKWWLIV7Q, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:54:13.109281937 (314h25m12.980364982s) [seg_pos={0 0}, upload=275696]: Segment reached optimal threshold of 13 pieces.
Dec 18 17:54:13.109482640 (314h25m12.980565705s) [noise=true , address=77.90.31.21:4880 , piece_id=EDFGJA5GIV2HY2KV55BDGMRZGMV3QDJ6OM5BOODGX4RJZBAAPAIA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF
Dec 18 17:54:13.109493139 (314h25m12.980576174s) [noise=true , address=77.90.31.21:4880 , piece_id=EDFGJA5GIV2HY2KV55BDGMRZGMV3QDJ6OM5BOODGX4RJZBAAPAIA, seg_pos={0 0}, upload=275696]: Long tail cancellation due to optimal threshold
Dec 18 17:54:13.109515152 (314h25m12.980598187s) [seg_pos={0 0}, upload=275696]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 18 17:54:13.109598921 (314h25m12.980681966s) [upload=275696]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 18 17:54:13.109606365 (314h25m12.9806894s) [upload=275696]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 18 17:54:13.397754392 (314h25m13.268837437s) [upload=275696]: Done uploading segments...
Dec 18 17:54:13.397776934 (314h25m13.268859969s) [upload=275696]: Upload finished. err: <nil>
Duration: 6.485073432s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
6.138268085s trace:
https://jaeger.dev.storj.io/trace/66a9c91bc4a6ccf0
FAIL (2025-12-18 18:00:30 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:41370->45.38.24.2:443: read: connection reset by peer
Duration: 6.138299284s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [...............................................X.XXX........] duration:
6.074453063s trace:
https://jaeger.dev.storj.io/trace/55e899dcc96ebc44
FAIL (2025-12-18 18:00:18 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: UDNDAZHIL3CV4LIC3RUFACBHCTQHF6PL6EGDJLWM4FSMYUXC4SQA) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: JYBWCPDYGATJSR3H2ISPX2KB4ZKBVIETIOEGRNSZZLQDE7IPAJHQ) context deadline exceeded
======LOGS======
Dec 18 17:53:46.907446718 (314h24m46.778529753s) [upload=275696]: Starting upload
Dec 18 17:53:46.907512523 (314h24m46.778595578s) [upload=275696]: Done starting upload
Dec 18 17:53:46.907634715 (314h24m46.77871775s) [upload=275696]: Uploading segments...
Dec 18 17:53:46.907721671 (314h24m46.778804716s) [upload=275696]: Got next segment. Inline: false
Dec 18 17:53:46.907766126 (314h24m46.778849161s) [upload=275696]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 18 17:53:46.907771726 (314h24m46.778854761s) [upload=275696]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 18 17:53:47.154892897 (314h24m47.025975943s) [seg_pos={0 0}, upload=275696]: Begin upload segment...
Dec 18 17:53:47.154946931 (314h24m47.026029976s) [seg_pos={0 0}, upload=275696]: Done begin upload segment.
Dec 18 17:53:47.154978721 (314h24m47.026061756s) [noise=true , address=145.79.227.42:4122 , piece_id=A245U2W7Y4XGA5JNGKPWBOM2HDQTOIZPMYF6GEZXZGJGIPCM4JPQ, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.155041712 (314h24m47.026124747s) [noise=true , address=77.90.31.21:4880 , piece_id=EDFGJA5GIV2HY2KV55BDGMRZGMV3QDJ6OM5BOODGX4RJZBAAPAIA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.155513359 (314h24m47.026596394s) [noise=true , address=145.79.227.52:4446 , piece_id=KOGZIRIVHIRYRX6B4HPS27EZKTPK4AK45HM644HLBOA5QUJLDB6A, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.155877683 (314h24m47.026960718s) [noise=true , address=145.79.227.37:4566 , piece_id=TUE5OHICYNTQQ26VILHTZXFMTIGWVUQS7JMYJEARRYVUT6CMCJPA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.156232409 (314h24m47.027315454s) [noise=true , address=145.79.227.14:4839 , piece_id=PW76MJZTJNUDTLCJ4GGQPLSEAJENKJYQOU5T62TAGCFTGQBNGFTA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.156821220 (314h24m47.027904265s) [noise=true , address=145.79.227.27:4691 , piece_id=4ROXPEVMSBJE7EF5MHM6RBEQKOPVGHPVV5RCCNGX6R2CKWWLIV7Q, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.157223125 (314h24m47.02830616s) [noise=true , address=45.91.116.10:4081 , piece_id=JYBWCPDYGATJSR3H2ISPX2KB4ZKBVIETIOEGRNSZZLQDE7IPAJHQ, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.157618027 (314h24m47.028701062s) [noise=true , address=145.79.227.49:4339 , piece_id=CGSAGFA7UHNQ53CLLYZ2NMD57JK25P2YVPCMKW27W7HIE2OWI3UA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.158280779 (314h24m47.029363814s) [noise=true , address=45.91.116.11:4120 , piece_id=UDNDAZHIL3CV4LIC3RUFACBHCTQHF6PL6EGDJLWM4FSMYUXC4SQA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.159585022 (314h24m47.030668067s) [noise=true , address=145.79.227.41:4087 , piece_id=NKF2BL3HHWERKL5DPZ5SH73LS7XXT325TFWS3BSSXHETFMLL6VCQ, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.161483899 (314h24m47.032566944s) [noise=true , address=145.79.227.51:4411 , piece_id=RIEZSDOKZCRKA7KX775JPXEZVRZWIGOEAG62LSBRWPEVP4AQLUTA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.161534034 (314h24m47.032617079s) [noise=true , address=145.79.227.35:4481 , piece_id=ONU6UGBZKGYZN2KSAS57OCCKTFMDT5HMDCVJU5EIDYTENVRVOI2Q, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.163305847 (314h24m47.034388893s) [noise=true , address=145.79.227.45:4229 , piece_id=RDOEHHL2QNKQWZIRPWLOY6HW47OI4ZDXIPJHTR76BVQ3EFHZMKLA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.165089595 (314h24m47.03617264s) [noise=true , address=145.79.227.46:4601 , piece_id=3QTBNM4CT6QE74FQCOBDICM5UYXW32REQ25YEKQWXAVDC46RQBCA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.166839817 (314h24m47.037922862s) [noise=true , address=145.79.227.44:4195 , piece_id=3FIM4BHP2C62BSJXHU7GVYQ3DOC5YL3KOGDHNRMRXM3MJ6MWOQMQ, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.168326769 (314h24m47.039409814s) [noise=true , address=145.79.227.31:4798 , piece_id=W2VFZPXKGVORU5JCQIIP2IR5SCSEMWYRWJOKHFHBI3V7WCSMLBWA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:48.203795023 (314h24m48.074878059s) [noise=true , address=145.79.227.44:4195 , piece_id=3FIM4BHP2C62BSJXHU7GVYQ3DOC5YL3KOGDHNRMRXM3MJ6MWOQMQ, seg_pos={0 0}, upload=275696]: Done uploading piece. err: ecclient: upload failed (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, address:145.79.227.44:4195): protocol: storage node overloaded, request limit: 2000
Dec 18 17:53:48.207081231 (314h24m48.078164276s) [noise=true , address=145.79.227.31:4798 , piece_id=W2VFZPXKGVORU5JCQIIP2IR5SCSEMWYRWJOKHFHBI3V7WCSMLBWA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: storage node overloaded, request limit: 2000
Dec 18 17:53:48.863464901 (314h24m48.734547956s) [upload=275696]: Next returned nil segment
Dec 18 17:53:48.863489558 (314h24m48.734572623s) [upload=275696]: Waiting for error group managing segments...
Dec 18 17:53:49.149217626 (314h24m49.020300661s) [noise=true , address=145.79.227.51:4411 , piece_id=RIEZSDOKZCRKA7KX775JPXEZVRZWIGOEAG62LSBRWPEVP4AQLUTA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.151240638 (314h24m49.022323683s) [noise=true , address=145.79.227.52:4446 , piece_id=KOGZIRIVHIRYRX6B4HPS27EZKTPK4AK45HM644HLBOA5QUJLDB6A, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.159687424 (314h24m49.030770459s) [noise=true , address=145.79.227.46:4601 , piece_id=3QTBNM4CT6QE74FQCOBDICM5UYXW32REQ25YEKQWXAVDC46RQBCA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.163440961 (314h24m49.034524006s) [noise=true , address=145.79.227.37:4566 , piece_id=TUE5OHICYNTQQ26VILHTZXFMTIGWVUQS7JMYJEARRYVUT6CMCJPA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.166507440 (314h24m49.037590475s) [noise=true , address=145.79.227.49:4339 , piece_id=CGSAGFA7UHNQ53CLLYZ2NMD57JK25P2YVPCMKW27W7HIE2OWI3UA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.183896243 (314h24m49.054979288s) [noise=true , address=145.79.227.41:4087 , piece_id=NKF2BL3HHWERKL5DPZ5SH73LS7XXT325TFWS3BSSXHETFMLL6VCQ, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.219629698 (314h24m49.090712733s) [noise=true , address=145.79.227.45:4229 , piece_id=RDOEHHL2QNKQWZIRPWLOY6HW47OI4ZDXIPJHTR76BVQ3EFHZMKLA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.229654178 (314h24m49.100737213s) [noise=true , address=145.79.227.35:4481 , piece_id=ONU6UGBZKGYZN2KSAS57OCCKTFMDT5HMDCVJU5EIDYTENVRVOI2Q, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.348528404 (314h24m49.219611439s) [noise=true , address=145.79.227.42:4122 , piece_id=A245U2W7Y4XGA5JNGKPWBOM2HDQTOIZPMYF6GEZXZGJGIPCM4JPQ, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.361664419 (314h24m49.232747454s) [noise=true , address=45.91.116.11:4120 , piece_id=UDNDAZHIL3CV4LIC3RUFACBHCTQHF6PL6EGDJLWM4FSMYUXC4SQA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.524862349 (314h24m49.395945394s) [noise=true , address=45.91.116.10:4081 , piece_id=JYBWCPDYGATJSR3H2ISPX2KB4ZKBVIETIOEGRNSZZLQDE7IPAJHQ, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.698660224 (314h24m49.569743279s) [noise=true , address=145.79.227.14:4839 , piece_id=PW76MJZTJNUDTLCJ4GGQPLSEAJENKJYQOU5T62TAGCFTGQBNGFTA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:54:13.109258041 (314h25m12.980341086s) [noise=true , address=145.79.227.27:4691 , piece_id=4ROXPEVMSBJE7EF5MHM6RBEQKOPVGHPVV5RCCNGX6R2CKWWLIV7Q, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:54:13.109281937 (314h25m12.980364982s) [seg_pos={0 0}, upload=275696]: Segment reached optimal threshold of 13 pieces.
Dec 18 17:54:13.109482640 (314h25m12.980565705s) [noise=true , address=77.90.31.21:4880 , piece_id=EDFGJA5GIV2HY2KV55BDGMRZGMV3QDJ6OM5BOODGX4RJZBAAPAIA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF
Dec 18 17:54:13.109493139 (314h25m12.980576174s) [noise=true , address=77.90.31.21:4880 , piece_id=EDFGJA5GIV2HY2KV55BDGMRZGMV3QDJ6OM5BOODGX4RJZBAAPAIA, seg_pos={0 0}, upload=275696]: Long tail cancellation due to optimal threshold
Dec 18 17:54:13.109515152 (314h25m12.980598187s) [seg_pos={0 0}, upload=275696]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 18 17:54:13.109598921 (314h25m12.980681966s) [upload=275696]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 18 17:54:13.109606365 (314h25m12.9806894s) [upload=275696]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 18 17:54:13.397754392 (314h25m13.268837437s) [upload=275696]: Done uploading segments...
Dec 18 17:54:13.397776934 (314h25m13.268859969s) [upload=275696]: Upload finished. err: <nil>
Duration: 6.074483201s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
5.923723956s trace:
https://jaeger.dev.storj.io/trace/66a9c91bc4a6ccf0
FAIL (2025-12-18 17:59:42 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:41370->45.38.24.2:443: read: connection reset by peer
Duration: 5.923741619s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [................................................X.XXX.......] duration:
5.892456239s trace:
https://jaeger.dev.storj.io/trace/55e899dcc96ebc44
FAIL (2025-12-18 17:59:06 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: UDNDAZHIL3CV4LIC3RUFACBHCTQHF6PL6EGDJLWM4FSMYUXC4SQA) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: JYBWCPDYGATJSR3H2ISPX2KB4ZKBVIETIOEGRNSZZLQDE7IPAJHQ) context deadline exceeded
======LOGS======
Dec 18 17:53:46.907446718 (314h24m46.778529753s) [upload=275696]: Starting upload
Dec 18 17:53:46.907512523 (314h24m46.778595578s) [upload=275696]: Done starting upload
Dec 18 17:53:46.907634715 (314h24m46.77871775s) [upload=275696]: Uploading segments...
Dec 18 17:53:46.907721671 (314h24m46.778804716s) [upload=275696]: Got next segment. Inline: false
Dec 18 17:53:46.907766126 (314h24m46.778849161s) [upload=275696]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 18 17:53:46.907771726 (314h24m46.778854761s) [upload=275696]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 18 17:53:47.154892897 (314h24m47.025975943s) [seg_pos={0 0}, upload=275696]: Begin upload segment...
Dec 18 17:53:47.154946931 (314h24m47.026029976s) [seg_pos={0 0}, upload=275696]: Done begin upload segment.
Dec 18 17:53:47.154978721 (314h24m47.026061756s) [noise=true , address=145.79.227.42:4122 , piece_id=A245U2W7Y4XGA5JNGKPWBOM2HDQTOIZPMYF6GEZXZGJGIPCM4JPQ, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.155041712 (314h24m47.026124747s) [noise=true , address=77.90.31.21:4880 , piece_id=EDFGJA5GIV2HY2KV55BDGMRZGMV3QDJ6OM5BOODGX4RJZBAAPAIA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.155513359 (314h24m47.026596394s) [noise=true , address=145.79.227.52:4446 , piece_id=KOGZIRIVHIRYRX6B4HPS27EZKTPK4AK45HM644HLBOA5QUJLDB6A, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.155877683 (314h24m47.026960718s) [noise=true , address=145.79.227.37:4566 , piece_id=TUE5OHICYNTQQ26VILHTZXFMTIGWVUQS7JMYJEARRYVUT6CMCJPA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.156232409 (314h24m47.027315454s) [noise=true , address=145.79.227.14:4839 , piece_id=PW76MJZTJNUDTLCJ4GGQPLSEAJENKJYQOU5T62TAGCFTGQBNGFTA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.156821220 (314h24m47.027904265s) [noise=true , address=145.79.227.27:4691 , piece_id=4ROXPEVMSBJE7EF5MHM6RBEQKOPVGHPVV5RCCNGX6R2CKWWLIV7Q, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.157223125 (314h24m47.02830616s) [noise=true , address=45.91.116.10:4081 , piece_id=JYBWCPDYGATJSR3H2ISPX2KB4ZKBVIETIOEGRNSZZLQDE7IPAJHQ, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.157618027 (314h24m47.028701062s) [noise=true , address=145.79.227.49:4339 , piece_id=CGSAGFA7UHNQ53CLLYZ2NMD57JK25P2YVPCMKW27W7HIE2OWI3UA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.158280779 (314h24m47.029363814s) [noise=true , address=45.91.116.11:4120 , piece_id=UDNDAZHIL3CV4LIC3RUFACBHCTQHF6PL6EGDJLWM4FSMYUXC4SQA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.159585022 (314h24m47.030668067s) [noise=true , address=145.79.227.41:4087 , piece_id=NKF2BL3HHWERKL5DPZ5SH73LS7XXT325TFWS3BSSXHETFMLL6VCQ, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.161483899 (314h24m47.032566944s) [noise=true , address=145.79.227.51:4411 , piece_id=RIEZSDOKZCRKA7KX775JPXEZVRZWIGOEAG62LSBRWPEVP4AQLUTA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.161534034 (314h24m47.032617079s) [noise=true , address=145.79.227.35:4481 , piece_id=ONU6UGBZKGYZN2KSAS57OCCKTFMDT5HMDCVJU5EIDYTENVRVOI2Q, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.163305847 (314h24m47.034388893s) [noise=true , address=145.79.227.45:4229 , piece_id=RDOEHHL2QNKQWZIRPWLOY6HW47OI4ZDXIPJHTR76BVQ3EFHZMKLA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.165089595 (314h24m47.03617264s) [noise=true , address=145.79.227.46:4601 , piece_id=3QTBNM4CT6QE74FQCOBDICM5UYXW32REQ25YEKQWXAVDC46RQBCA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.166839817 (314h24m47.037922862s) [noise=true , address=145.79.227.44:4195 , piece_id=3FIM4BHP2C62BSJXHU7GVYQ3DOC5YL3KOGDHNRMRXM3MJ6MWOQMQ, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.168326769 (314h24m47.039409814s) [noise=true , address=145.79.227.31:4798 , piece_id=W2VFZPXKGVORU5JCQIIP2IR5SCSEMWYRWJOKHFHBI3V7WCSMLBWA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:48.203795023 (314h24m48.074878059s) [noise=true , address=145.79.227.44:4195 , piece_id=3FIM4BHP2C62BSJXHU7GVYQ3DOC5YL3KOGDHNRMRXM3MJ6MWOQMQ, seg_pos={0 0}, upload=275696]: Done uploading piece. err: ecclient: upload failed (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, address:145.79.227.44:4195): protocol: storage node overloaded, request limit: 2000
Dec 18 17:53:48.207081231 (314h24m48.078164276s) [noise=true , address=145.79.227.31:4798 , piece_id=W2VFZPXKGVORU5JCQIIP2IR5SCSEMWYRWJOKHFHBI3V7WCSMLBWA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: storage node overloaded, request limit: 2000
Dec 18 17:53:48.863464901 (314h24m48.734547956s) [upload=275696]: Next returned nil segment
Dec 18 17:53:48.863489558 (314h24m48.734572623s) [upload=275696]: Waiting for error group managing segments...
Dec 18 17:53:49.149217626 (314h24m49.020300661s) [noise=true , address=145.79.227.51:4411 , piece_id=RIEZSDOKZCRKA7KX775JPXEZVRZWIGOEAG62LSBRWPEVP4AQLUTA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.151240638 (314h24m49.022323683s) [noise=true , address=145.79.227.52:4446 , piece_id=KOGZIRIVHIRYRX6B4HPS27EZKTPK4AK45HM644HLBOA5QUJLDB6A, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.159687424 (314h24m49.030770459s) [noise=true , address=145.79.227.46:4601 , piece_id=3QTBNM4CT6QE74FQCOBDICM5UYXW32REQ25YEKQWXAVDC46RQBCA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.163440961 (314h24m49.034524006s) [noise=true , address=145.79.227.37:4566 , piece_id=TUE5OHICYNTQQ26VILHTZXFMTIGWVUQS7JMYJEARRYVUT6CMCJPA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.166507440 (314h24m49.037590475s) [noise=true , address=145.79.227.49:4339 , piece_id=CGSAGFA7UHNQ53CLLYZ2NMD57JK25P2YVPCMKW27W7HIE2OWI3UA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.183896243 (314h24m49.054979288s) [noise=true , address=145.79.227.41:4087 , piece_id=NKF2BL3HHWERKL5DPZ5SH73LS7XXT325TFWS3BSSXHETFMLL6VCQ, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.219629698 (314h24m49.090712733s) [noise=true , address=145.79.227.45:4229 , piece_id=RDOEHHL2QNKQWZIRPWLOY6HW47OI4ZDXIPJHTR76BVQ3EFHZMKLA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.229654178 (314h24m49.100737213s) [noise=true , address=145.79.227.35:4481 , piece_id=ONU6UGBZKGYZN2KSAS57OCCKTFMDT5HMDCVJU5EIDYTENVRVOI2Q, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.348528404 (314h24m49.219611439s) [noise=true , address=145.79.227.42:4122 , piece_id=A245U2W7Y4XGA5JNGKPWBOM2HDQTOIZPMYF6GEZXZGJGIPCM4JPQ, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.361664419 (314h24m49.232747454s) [noise=true , address=45.91.116.11:4120 , piece_id=UDNDAZHIL3CV4LIC3RUFACBHCTQHF6PL6EGDJLWM4FSMYUXC4SQA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.524862349 (314h24m49.395945394s) [noise=true , address=45.91.116.10:4081 , piece_id=JYBWCPDYGATJSR3H2ISPX2KB4ZKBVIETIOEGRNSZZLQDE7IPAJHQ, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.698660224 (314h24m49.569743279s) [noise=true , address=145.79.227.14:4839 , piece_id=PW76MJZTJNUDTLCJ4GGQPLSEAJENKJYQOU5T62TAGCFTGQBNGFTA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:54:13.109258041 (314h25m12.980341086s) [noise=true , address=145.79.227.27:4691 , piece_id=4ROXPEVMSBJE7EF5MHM6RBEQKOPVGHPVV5RCCNGX6R2CKWWLIV7Q, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:54:13.109281937 (314h25m12.980364982s) [seg_pos={0 0}, upload=275696]: Segment reached optimal threshold of 13 pieces.
Dec 18 17:54:13.109482640 (314h25m12.980565705s) [noise=true , address=77.90.31.21:4880 , piece_id=EDFGJA5GIV2HY2KV55BDGMRZGMV3QDJ6OM5BOODGX4RJZBAAPAIA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF
Dec 18 17:54:13.109493139 (314h25m12.980576174s) [noise=true , address=77.90.31.21:4880 , piece_id=EDFGJA5GIV2HY2KV55BDGMRZGMV3QDJ6OM5BOODGX4RJZBAAPAIA, seg_pos={0 0}, upload=275696]: Long tail cancellation due to optimal threshold
Dec 18 17:54:13.109515152 (314h25m12.980598187s) [seg_pos={0 0}, upload=275696]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 18 17:54:13.109598921 (314h25m12.980681966s) [upload=275696]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 18 17:54:13.109606365 (314h25m12.9806894s) [upload=275696]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 18 17:54:13.397754392 (314h25m13.268837437s) [upload=275696]: Done uploading segments...
Dec 18 17:54:13.397776934 (314h25m13.268859969s) [upload=275696]: Upload finished. err: <nil>
Duration: 5.892487177s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
5.963932549s trace:
https://jaeger.dev.storj.io/trace/66a9c91bc4a6ccf0
FAIL (2025-12-18 17:59:04 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:41370->45.38.24.2:443: read: connection reset by peer
Duration: 5.963979639s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
5.815448824s trace:
https://jaeger.dev.storj.io/trace/66a9c91bc4a6ccf0
FAIL (2025-12-18 17:58:35 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:41370->45.38.24.2:443: read: connection reset by peer
Duration: 5.815484412s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.................................................X.XXX......] duration:
5.826130234s trace:
https://jaeger.dev.storj.io/trace/55e899dcc96ebc44
FAIL (2025-12-18 17:57:53 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: UDNDAZHIL3CV4LIC3RUFACBHCTQHF6PL6EGDJLWM4FSMYUXC4SQA) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: JYBWCPDYGATJSR3H2ISPX2KB4ZKBVIETIOEGRNSZZLQDE7IPAJHQ) context deadline exceeded
======LOGS======
Dec 18 17:53:46.907446718 (314h24m46.778529753s) [upload=275696]: Starting upload
Dec 18 17:53:46.907512523 (314h24m46.778595578s) [upload=275696]: Done starting upload
Dec 18 17:53:46.907634715 (314h24m46.77871775s) [upload=275696]: Uploading segments...
Dec 18 17:53:46.907721671 (314h24m46.778804716s) [upload=275696]: Got next segment. Inline: false
Dec 18 17:53:46.907766126 (314h24m46.778849161s) [upload=275696]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 18 17:53:46.907771726 (314h24m46.778854761s) [upload=275696]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 18 17:53:47.154892897 (314h24m47.025975943s) [seg_pos={0 0}, upload=275696]: Begin upload segment...
Dec 18 17:53:47.154946931 (314h24m47.026029976s) [seg_pos={0 0}, upload=275696]: Done begin upload segment.
Dec 18 17:53:47.154978721 (314h24m47.026061756s) [noise=true , address=145.79.227.42:4122 , piece_id=A245U2W7Y4XGA5JNGKPWBOM2HDQTOIZPMYF6GEZXZGJGIPCM4JPQ, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.155041712 (314h24m47.026124747s) [noise=true , address=77.90.31.21:4880 , piece_id=EDFGJA5GIV2HY2KV55BDGMRZGMV3QDJ6OM5BOODGX4RJZBAAPAIA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.155513359 (314h24m47.026596394s) [noise=true , address=145.79.227.52:4446 , piece_id=KOGZIRIVHIRYRX6B4HPS27EZKTPK4AK45HM644HLBOA5QUJLDB6A, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.155877683 (314h24m47.026960718s) [noise=true , address=145.79.227.37:4566 , piece_id=TUE5OHICYNTQQ26VILHTZXFMTIGWVUQS7JMYJEARRYVUT6CMCJPA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.156232409 (314h24m47.027315454s) [noise=true , address=145.79.227.14:4839 , piece_id=PW76MJZTJNUDTLCJ4GGQPLSEAJENKJYQOU5T62TAGCFTGQBNGFTA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.156821220 (314h24m47.027904265s) [noise=true , address=145.79.227.27:4691 , piece_id=4ROXPEVMSBJE7EF5MHM6RBEQKOPVGHPVV5RCCNGX6R2CKWWLIV7Q, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.157223125 (314h24m47.02830616s) [noise=true , address=45.91.116.10:4081 , piece_id=JYBWCPDYGATJSR3H2ISPX2KB4ZKBVIETIOEGRNSZZLQDE7IPAJHQ, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.157618027 (314h24m47.028701062s) [noise=true , address=145.79.227.49:4339 , piece_id=CGSAGFA7UHNQ53CLLYZ2NMD57JK25P2YVPCMKW27W7HIE2OWI3UA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.158280779 (314h24m47.029363814s) [noise=true , address=45.91.116.11:4120 , piece_id=UDNDAZHIL3CV4LIC3RUFACBHCTQHF6PL6EGDJLWM4FSMYUXC4SQA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.159585022 (314h24m47.030668067s) [noise=true , address=145.79.227.41:4087 , piece_id=NKF2BL3HHWERKL5DPZ5SH73LS7XXT325TFWS3BSSXHETFMLL6VCQ, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.161483899 (314h24m47.032566944s) [noise=true , address=145.79.227.51:4411 , piece_id=RIEZSDOKZCRKA7KX775JPXEZVRZWIGOEAG62LSBRWPEVP4AQLUTA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.161534034 (314h24m47.032617079s) [noise=true , address=145.79.227.35:4481 , piece_id=ONU6UGBZKGYZN2KSAS57OCCKTFMDT5HMDCVJU5EIDYTENVRVOI2Q, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.163305847 (314h24m47.034388893s) [noise=true , address=145.79.227.45:4229 , piece_id=RDOEHHL2QNKQWZIRPWLOY6HW47OI4ZDXIPJHTR76BVQ3EFHZMKLA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.165089595 (314h24m47.03617264s) [noise=true , address=145.79.227.46:4601 , piece_id=3QTBNM4CT6QE74FQCOBDICM5UYXW32REQ25YEKQWXAVDC46RQBCA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.166839817 (314h24m47.037922862s) [noise=true , address=145.79.227.44:4195 , piece_id=3FIM4BHP2C62BSJXHU7GVYQ3DOC5YL3KOGDHNRMRXM3MJ6MWOQMQ, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.168326769 (314h24m47.039409814s) [noise=true , address=145.79.227.31:4798 , piece_id=W2VFZPXKGVORU5JCQIIP2IR5SCSEMWYRWJOKHFHBI3V7WCSMLBWA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:48.203795023 (314h24m48.074878059s) [noise=true , address=145.79.227.44:4195 , piece_id=3FIM4BHP2C62BSJXHU7GVYQ3DOC5YL3KOGDHNRMRXM3MJ6MWOQMQ, seg_pos={0 0}, upload=275696]: Done uploading piece. err: ecclient: upload failed (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, address:145.79.227.44:4195): protocol: storage node overloaded, request limit: 2000
Dec 18 17:53:48.207081231 (314h24m48.078164276s) [noise=true , address=145.79.227.31:4798 , piece_id=W2VFZPXKGVORU5JCQIIP2IR5SCSEMWYRWJOKHFHBI3V7WCSMLBWA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: storage node overloaded, request limit: 2000
Dec 18 17:53:48.863464901 (314h24m48.734547956s) [upload=275696]: Next returned nil segment
Dec 18 17:53:48.863489558 (314h24m48.734572623s) [upload=275696]: Waiting for error group managing segments...
Dec 18 17:53:49.149217626 (314h24m49.020300661s) [noise=true , address=145.79.227.51:4411 , piece_id=RIEZSDOKZCRKA7KX775JPXEZVRZWIGOEAG62LSBRWPEVP4AQLUTA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.151240638 (314h24m49.022323683s) [noise=true , address=145.79.227.52:4446 , piece_id=KOGZIRIVHIRYRX6B4HPS27EZKTPK4AK45HM644HLBOA5QUJLDB6A, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.159687424 (314h24m49.030770459s) [noise=true , address=145.79.227.46:4601 , piece_id=3QTBNM4CT6QE74FQCOBDICM5UYXW32REQ25YEKQWXAVDC46RQBCA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.163440961 (314h24m49.034524006s) [noise=true , address=145.79.227.37:4566 , piece_id=TUE5OHICYNTQQ26VILHTZXFMTIGWVUQS7JMYJEARRYVUT6CMCJPA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.166507440 (314h24m49.037590475s) [noise=true , address=145.79.227.49:4339 , piece_id=CGSAGFA7UHNQ53CLLYZ2NMD57JK25P2YVPCMKW27W7HIE2OWI3UA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.183896243 (314h24m49.054979288s) [noise=true , address=145.79.227.41:4087 , piece_id=NKF2BL3HHWERKL5DPZ5SH73LS7XXT325TFWS3BSSXHETFMLL6VCQ, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.219629698 (314h24m49.090712733s) [noise=true , address=145.79.227.45:4229 , piece_id=RDOEHHL2QNKQWZIRPWLOY6HW47OI4ZDXIPJHTR76BVQ3EFHZMKLA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.229654178 (314h24m49.100737213s) [noise=true , address=145.79.227.35:4481 , piece_id=ONU6UGBZKGYZN2KSAS57OCCKTFMDT5HMDCVJU5EIDYTENVRVOI2Q, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.348528404 (314h24m49.219611439s) [noise=true , address=145.79.227.42:4122 , piece_id=A245U2W7Y4XGA5JNGKPWBOM2HDQTOIZPMYF6GEZXZGJGIPCM4JPQ, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.361664419 (314h24m49.232747454s) [noise=true , address=45.91.116.11:4120 , piece_id=UDNDAZHIL3CV4LIC3RUFACBHCTQHF6PL6EGDJLWM4FSMYUXC4SQA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.524862349 (314h24m49.395945394s) [noise=true , address=45.91.116.10:4081 , piece_id=JYBWCPDYGATJSR3H2ISPX2KB4ZKBVIETIOEGRNSZZLQDE7IPAJHQ, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.698660224 (314h24m49.569743279s) [noise=true , address=145.79.227.14:4839 , piece_id=PW76MJZTJNUDTLCJ4GGQPLSEAJENKJYQOU5T62TAGCFTGQBNGFTA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:54:13.109258041 (314h25m12.980341086s) [noise=true , address=145.79.227.27:4691 , piece_id=4ROXPEVMSBJE7EF5MHM6RBEQKOPVGHPVV5RCCNGX6R2CKWWLIV7Q, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:54:13.109281937 (314h25m12.980364982s) [seg_pos={0 0}, upload=275696]: Segment reached optimal threshold of 13 pieces.
Dec 18 17:54:13.109482640 (314h25m12.980565705s) [noise=true , address=77.90.31.21:4880 , piece_id=EDFGJA5GIV2HY2KV55BDGMRZGMV3QDJ6OM5BOODGX4RJZBAAPAIA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF
Dec 18 17:54:13.109493139 (314h25m12.980576174s) [noise=true , address=77.90.31.21:4880 , piece_id=EDFGJA5GIV2HY2KV55BDGMRZGMV3QDJ6OM5BOODGX4RJZBAAPAIA, seg_pos={0 0}, upload=275696]: Long tail cancellation due to optimal threshold
Dec 18 17:54:13.109515152 (314h25m12.980598187s) [seg_pos={0 0}, upload=275696]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 18 17:54:13.109598921 (314h25m12.980681966s) [upload=275696]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 18 17:54:13.109606365 (314h25m12.9806894s) [upload=275696]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 18 17:54:13.397754392 (314h25m13.268837437s) [upload=275696]: Done uploading segments...
Dec 18 17:54:13.397776934 (314h25m13.268859969s) [upload=275696]: Upload finished. err: <nil>
Duration: 5.82615898s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
6.101992982s trace:
https://jaeger.dev.storj.io/trace/66a9c91bc4a6ccf0
FAIL (2025-12-18 17:57:31 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:41370->45.38.24.2:443: read: connection reset by peer
Duration: 6.102024082s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [..................................................X.XXX.....] duration:
5.895890343s trace:
https://jaeger.dev.storj.io/trace/55e899dcc96ebc44
FAIL (2025-12-18 17:56:57 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: UDNDAZHIL3CV4LIC3RUFACBHCTQHF6PL6EGDJLWM4FSMYUXC4SQA) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: JYBWCPDYGATJSR3H2ISPX2KB4ZKBVIETIOEGRNSZZLQDE7IPAJHQ) context deadline exceeded
======LOGS======
Dec 18 17:53:46.907446718 (314h24m46.778529753s) [upload=275696]: Starting upload
Dec 18 17:53:46.907512523 (314h24m46.778595578s) [upload=275696]: Done starting upload
Dec 18 17:53:46.907634715 (314h24m46.77871775s) [upload=275696]: Uploading segments...
Dec 18 17:53:46.907721671 (314h24m46.778804716s) [upload=275696]: Got next segment. Inline: false
Dec 18 17:53:46.907766126 (314h24m46.778849161s) [upload=275696]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 18 17:53:46.907771726 (314h24m46.778854761s) [upload=275696]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 18 17:53:47.154892897 (314h24m47.025975943s) [seg_pos={0 0}, upload=275696]: Begin upload segment...
Dec 18 17:53:47.154946931 (314h24m47.026029976s) [seg_pos={0 0}, upload=275696]: Done begin upload segment.
Dec 18 17:53:47.154978721 (314h24m47.026061756s) [noise=true , address=145.79.227.42:4122 , piece_id=A245U2W7Y4XGA5JNGKPWBOM2HDQTOIZPMYF6GEZXZGJGIPCM4JPQ, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.155041712 (314h24m47.026124747s) [noise=true , address=77.90.31.21:4880 , piece_id=EDFGJA5GIV2HY2KV55BDGMRZGMV3QDJ6OM5BOODGX4RJZBAAPAIA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.155513359 (314h24m47.026596394s) [noise=true , address=145.79.227.52:4446 , piece_id=KOGZIRIVHIRYRX6B4HPS27EZKTPK4AK45HM644HLBOA5QUJLDB6A, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.155877683 (314h24m47.026960718s) [noise=true , address=145.79.227.37:4566 , piece_id=TUE5OHICYNTQQ26VILHTZXFMTIGWVUQS7JMYJEARRYVUT6CMCJPA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.156232409 (314h24m47.027315454s) [noise=true , address=145.79.227.14:4839 , piece_id=PW76MJZTJNUDTLCJ4GGQPLSEAJENKJYQOU5T62TAGCFTGQBNGFTA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.156821220 (314h24m47.027904265s) [noise=true , address=145.79.227.27:4691 , piece_id=4ROXPEVMSBJE7EF5MHM6RBEQKOPVGHPVV5RCCNGX6R2CKWWLIV7Q, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.157223125 (314h24m47.02830616s) [noise=true , address=45.91.116.10:4081 , piece_id=JYBWCPDYGATJSR3H2ISPX2KB4ZKBVIETIOEGRNSZZLQDE7IPAJHQ, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.157618027 (314h24m47.028701062s) [noise=true , address=145.79.227.49:4339 , piece_id=CGSAGFA7UHNQ53CLLYZ2NMD57JK25P2YVPCMKW27W7HIE2OWI3UA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.158280779 (314h24m47.029363814s) [noise=true , address=45.91.116.11:4120 , piece_id=UDNDAZHIL3CV4LIC3RUFACBHCTQHF6PL6EGDJLWM4FSMYUXC4SQA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.159585022 (314h24m47.030668067s) [noise=true , address=145.79.227.41:4087 , piece_id=NKF2BL3HHWERKL5DPZ5SH73LS7XXT325TFWS3BSSXHETFMLL6VCQ, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.161483899 (314h24m47.032566944s) [noise=true , address=145.79.227.51:4411 , piece_id=RIEZSDOKZCRKA7KX775JPXEZVRZWIGOEAG62LSBRWPEVP4AQLUTA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.161534034 (314h24m47.032617079s) [noise=true , address=145.79.227.35:4481 , piece_id=ONU6UGBZKGYZN2KSAS57OCCKTFMDT5HMDCVJU5EIDYTENVRVOI2Q, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.163305847 (314h24m47.034388893s) [noise=true , address=145.79.227.45:4229 , piece_id=RDOEHHL2QNKQWZIRPWLOY6HW47OI4ZDXIPJHTR76BVQ3EFHZMKLA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.165089595 (314h24m47.03617264s) [noise=true , address=145.79.227.46:4601 , piece_id=3QTBNM4CT6QE74FQCOBDICM5UYXW32REQ25YEKQWXAVDC46RQBCA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.166839817 (314h24m47.037922862s) [noise=true , address=145.79.227.44:4195 , piece_id=3FIM4BHP2C62BSJXHU7GVYQ3DOC5YL3KOGDHNRMRXM3MJ6MWOQMQ, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.168326769 (314h24m47.039409814s) [noise=true , address=145.79.227.31:4798 , piece_id=W2VFZPXKGVORU5JCQIIP2IR5SCSEMWYRWJOKHFHBI3V7WCSMLBWA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:48.203795023 (314h24m48.074878059s) [noise=true , address=145.79.227.44:4195 , piece_id=3FIM4BHP2C62BSJXHU7GVYQ3DOC5YL3KOGDHNRMRXM3MJ6MWOQMQ, seg_pos={0 0}, upload=275696]: Done uploading piece. err: ecclient: upload failed (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, address:145.79.227.44:4195): protocol: storage node overloaded, request limit: 2000
Dec 18 17:53:48.207081231 (314h24m48.078164276s) [noise=true , address=145.79.227.31:4798 , piece_id=W2VFZPXKGVORU5JCQIIP2IR5SCSEMWYRWJOKHFHBI3V7WCSMLBWA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: storage node overloaded, request limit: 2000
Dec 18 17:53:48.863464901 (314h24m48.734547956s) [upload=275696]: Next returned nil segment
Dec 18 17:53:48.863489558 (314h24m48.734572623s) [upload=275696]: Waiting for error group managing segments...
Dec 18 17:53:49.149217626 (314h24m49.020300661s) [noise=true , address=145.79.227.51:4411 , piece_id=RIEZSDOKZCRKA7KX775JPXEZVRZWIGOEAG62LSBRWPEVP4AQLUTA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.151240638 (314h24m49.022323683s) [noise=true , address=145.79.227.52:4446 , piece_id=KOGZIRIVHIRYRX6B4HPS27EZKTPK4AK45HM644HLBOA5QUJLDB6A, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.159687424 (314h24m49.030770459s) [noise=true , address=145.79.227.46:4601 , piece_id=3QTBNM4CT6QE74FQCOBDICM5UYXW32REQ25YEKQWXAVDC46RQBCA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.163440961 (314h24m49.034524006s) [noise=true , address=145.79.227.37:4566 , piece_id=TUE5OHICYNTQQ26VILHTZXFMTIGWVUQS7JMYJEARRYVUT6CMCJPA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.166507440 (314h24m49.037590475s) [noise=true , address=145.79.227.49:4339 , piece_id=CGSAGFA7UHNQ53CLLYZ2NMD57JK25P2YVPCMKW27W7HIE2OWI3UA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.183896243 (314h24m49.054979288s) [noise=true , address=145.79.227.41:4087 , piece_id=NKF2BL3HHWERKL5DPZ5SH73LS7XXT325TFWS3BSSXHETFMLL6VCQ, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.219629698 (314h24m49.090712733s) [noise=true , address=145.79.227.45:4229 , piece_id=RDOEHHL2QNKQWZIRPWLOY6HW47OI4ZDXIPJHTR76BVQ3EFHZMKLA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.229654178 (314h24m49.100737213s) [noise=true , address=145.79.227.35:4481 , piece_id=ONU6UGBZKGYZN2KSAS57OCCKTFMDT5HMDCVJU5EIDYTENVRVOI2Q, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.348528404 (314h24m49.219611439s) [noise=true , address=145.79.227.42:4122 , piece_id=A245U2W7Y4XGA5JNGKPWBOM2HDQTOIZPMYF6GEZXZGJGIPCM4JPQ, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.361664419 (314h24m49.232747454s) [noise=true , address=45.91.116.11:4120 , piece_id=UDNDAZHIL3CV4LIC3RUFACBHCTQHF6PL6EGDJLWM4FSMYUXC4SQA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.524862349 (314h24m49.395945394s) [noise=true , address=45.91.116.10:4081 , piece_id=JYBWCPDYGATJSR3H2ISPX2KB4ZKBVIETIOEGRNSZZLQDE7IPAJHQ, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.698660224 (314h24m49.569743279s) [noise=true , address=145.79.227.14:4839 , piece_id=PW76MJZTJNUDTLCJ4GGQPLSEAJENKJYQOU5T62TAGCFTGQBNGFTA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:54:13.109258041 (314h25m12.980341086s) [noise=true , address=145.79.227.27:4691 , piece_id=4ROXPEVMSBJE7EF5MHM6RBEQKOPVGHPVV5RCCNGX6R2CKWWLIV7Q, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:54:13.109281937 (314h25m12.980364982s) [seg_pos={0 0}, upload=275696]: Segment reached optimal threshold of 13 pieces.
Dec 18 17:54:13.109482640 (314h25m12.980565705s) [noise=true , address=77.90.31.21:4880 , piece_id=EDFGJA5GIV2HY2KV55BDGMRZGMV3QDJ6OM5BOODGX4RJZBAAPAIA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF
Dec 18 17:54:13.109493139 (314h25m12.980576174s) [noise=true , address=77.90.31.21:4880 , piece_id=EDFGJA5GIV2HY2KV55BDGMRZGMV3QDJ6OM5BOODGX4RJZBAAPAIA, seg_pos={0 0}, upload=275696]: Long tail cancellation due to optimal threshold
Dec 18 17:54:13.109515152 (314h25m12.980598187s) [seg_pos={0 0}, upload=275696]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 18 17:54:13.109598921 (314h25m12.980681966s) [upload=275696]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 18 17:54:13.109606365 (314h25m12.9806894s) [upload=275696]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 18 17:54:13.397754392 (314h25m13.268837437s) [upload=275696]: Done uploading segments...
Dec 18 17:54:13.397776934 (314h25m13.268859969s) [upload=275696]: Upload finished. err: <nil>
Duration: 5.895918386s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
5.80270904s trace:
https://jaeger.dev.storj.io/trace/66a9c91bc4a6ccf0
FAIL (2025-12-18 17:56:51 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:41370->45.38.24.2:443: read: connection reset by peer
Duration: 5.802738957s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [...................................................X.XXX....] duration:
5.835045031s trace:
https://jaeger.dev.storj.io/trace/55e899dcc96ebc44
FAIL (2025-12-18 17:56:36 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: UDNDAZHIL3CV4LIC3RUFACBHCTQHF6PL6EGDJLWM4FSMYUXC4SQA) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: JYBWCPDYGATJSR3H2ISPX2KB4ZKBVIETIOEGRNSZZLQDE7IPAJHQ) context deadline exceeded
======LOGS======
Dec 18 17:53:46.907446718 (314h24m46.778529753s) [upload=275696]: Starting upload
Dec 18 17:53:46.907512523 (314h24m46.778595578s) [upload=275696]: Done starting upload
Dec 18 17:53:46.907634715 (314h24m46.77871775s) [upload=275696]: Uploading segments...
Dec 18 17:53:46.907721671 (314h24m46.778804716s) [upload=275696]: Got next segment. Inline: false
Dec 18 17:53:46.907766126 (314h24m46.778849161s) [upload=275696]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 18 17:53:46.907771726 (314h24m46.778854761s) [upload=275696]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 18 17:53:47.154892897 (314h24m47.025975943s) [seg_pos={0 0}, upload=275696]: Begin upload segment...
Dec 18 17:53:47.154946931 (314h24m47.026029976s) [seg_pos={0 0}, upload=275696]: Done begin upload segment.
Dec 18 17:53:47.154978721 (314h24m47.026061756s) [noise=true , address=145.79.227.42:4122 , piece_id=A245U2W7Y4XGA5JNGKPWBOM2HDQTOIZPMYF6GEZXZGJGIPCM4JPQ, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.155041712 (314h24m47.026124747s) [noise=true , address=77.90.31.21:4880 , piece_id=EDFGJA5GIV2HY2KV55BDGMRZGMV3QDJ6OM5BOODGX4RJZBAAPAIA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.155513359 (314h24m47.026596394s) [noise=true , address=145.79.227.52:4446 , piece_id=KOGZIRIVHIRYRX6B4HPS27EZKTPK4AK45HM644HLBOA5QUJLDB6A, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.155877683 (314h24m47.026960718s) [noise=true , address=145.79.227.37:4566 , piece_id=TUE5OHICYNTQQ26VILHTZXFMTIGWVUQS7JMYJEARRYVUT6CMCJPA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.156232409 (314h24m47.027315454s) [noise=true , address=145.79.227.14:4839 , piece_id=PW76MJZTJNUDTLCJ4GGQPLSEAJENKJYQOU5T62TAGCFTGQBNGFTA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.156821220 (314h24m47.027904265s) [noise=true , address=145.79.227.27:4691 , piece_id=4ROXPEVMSBJE7EF5MHM6RBEQKOPVGHPVV5RCCNGX6R2CKWWLIV7Q, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.157223125 (314h24m47.02830616s) [noise=true , address=45.91.116.10:4081 , piece_id=JYBWCPDYGATJSR3H2ISPX2KB4ZKBVIETIOEGRNSZZLQDE7IPAJHQ, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.157618027 (314h24m47.028701062s) [noise=true , address=145.79.227.49:4339 , piece_id=CGSAGFA7UHNQ53CLLYZ2NMD57JK25P2YVPCMKW27W7HIE2OWI3UA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.158280779 (314h24m47.029363814s) [noise=true , address=45.91.116.11:4120 , piece_id=UDNDAZHIL3CV4LIC3RUFACBHCTQHF6PL6EGDJLWM4FSMYUXC4SQA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.159585022 (314h24m47.030668067s) [noise=true , address=145.79.227.41:4087 , piece_id=NKF2BL3HHWERKL5DPZ5SH73LS7XXT325TFWS3BSSXHETFMLL6VCQ, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.161483899 (314h24m47.032566944s) [noise=true , address=145.79.227.51:4411 , piece_id=RIEZSDOKZCRKA7KX775JPXEZVRZWIGOEAG62LSBRWPEVP4AQLUTA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.161534034 (314h24m47.032617079s) [noise=true , address=145.79.227.35:4481 , piece_id=ONU6UGBZKGYZN2KSAS57OCCKTFMDT5HMDCVJU5EIDYTENVRVOI2Q, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.163305847 (314h24m47.034388893s) [noise=true , address=145.79.227.45:4229 , piece_id=RDOEHHL2QNKQWZIRPWLOY6HW47OI4ZDXIPJHTR76BVQ3EFHZMKLA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.165089595 (314h24m47.03617264s) [noise=true , address=145.79.227.46:4601 , piece_id=3QTBNM4CT6QE74FQCOBDICM5UYXW32REQ25YEKQWXAVDC46RQBCA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.166839817 (314h24m47.037922862s) [noise=true , address=145.79.227.44:4195 , piece_id=3FIM4BHP2C62BSJXHU7GVYQ3DOC5YL3KOGDHNRMRXM3MJ6MWOQMQ, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.168326769 (314h24m47.039409814s) [noise=true , address=145.79.227.31:4798 , piece_id=W2VFZPXKGVORU5JCQIIP2IR5SCSEMWYRWJOKHFHBI3V7WCSMLBWA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:48.203795023 (314h24m48.074878059s) [noise=true , address=145.79.227.44:4195 , piece_id=3FIM4BHP2C62BSJXHU7GVYQ3DOC5YL3KOGDHNRMRXM3MJ6MWOQMQ, seg_pos={0 0}, upload=275696]: Done uploading piece. err: ecclient: upload failed (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, address:145.79.227.44:4195): protocol: storage node overloaded, request limit: 2000
Dec 18 17:53:48.207081231 (314h24m48.078164276s) [noise=true , address=145.79.227.31:4798 , piece_id=W2VFZPXKGVORU5JCQIIP2IR5SCSEMWYRWJOKHFHBI3V7WCSMLBWA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: storage node overloaded, request limit: 2000
Dec 18 17:53:48.863464901 (314h24m48.734547956s) [upload=275696]: Next returned nil segment
Dec 18 17:53:48.863489558 (314h24m48.734572623s) [upload=275696]: Waiting for error group managing segments...
Dec 18 17:53:49.149217626 (314h24m49.020300661s) [noise=true , address=145.79.227.51:4411 , piece_id=RIEZSDOKZCRKA7KX775JPXEZVRZWIGOEAG62LSBRWPEVP4AQLUTA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.151240638 (314h24m49.022323683s) [noise=true , address=145.79.227.52:4446 , piece_id=KOGZIRIVHIRYRX6B4HPS27EZKTPK4AK45HM644HLBOA5QUJLDB6A, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.159687424 (314h24m49.030770459s) [noise=true , address=145.79.227.46:4601 , piece_id=3QTBNM4CT6QE74FQCOBDICM5UYXW32REQ25YEKQWXAVDC46RQBCA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.163440961 (314h24m49.034524006s) [noise=true , address=145.79.227.37:4566 , piece_id=TUE5OHICYNTQQ26VILHTZXFMTIGWVUQS7JMYJEARRYVUT6CMCJPA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.166507440 (314h24m49.037590475s) [noise=true , address=145.79.227.49:4339 , piece_id=CGSAGFA7UHNQ53CLLYZ2NMD57JK25P2YVPCMKW27W7HIE2OWI3UA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.183896243 (314h24m49.054979288s) [noise=true , address=145.79.227.41:4087 , piece_id=NKF2BL3HHWERKL5DPZ5SH73LS7XXT325TFWS3BSSXHETFMLL6VCQ, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.219629698 (314h24m49.090712733s) [noise=true , address=145.79.227.45:4229 , piece_id=RDOEHHL2QNKQWZIRPWLOY6HW47OI4ZDXIPJHTR76BVQ3EFHZMKLA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.229654178 (314h24m49.100737213s) [noise=true , address=145.79.227.35:4481 , piece_id=ONU6UGBZKGYZN2KSAS57OCCKTFMDT5HMDCVJU5EIDYTENVRVOI2Q, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.348528404 (314h24m49.219611439s) [noise=true , address=145.79.227.42:4122 , piece_id=A245U2W7Y4XGA5JNGKPWBOM2HDQTOIZPMYF6GEZXZGJGIPCM4JPQ, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.361664419 (314h24m49.232747454s) [noise=true , address=45.91.116.11:4120 , piece_id=UDNDAZHIL3CV4LIC3RUFACBHCTQHF6PL6EGDJLWM4FSMYUXC4SQA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.524862349 (314h24m49.395945394s) [noise=true , address=45.91.116.10:4081 , piece_id=JYBWCPDYGATJSR3H2ISPX2KB4ZKBVIETIOEGRNSZZLQDE7IPAJHQ, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.698660224 (314h24m49.569743279s) [noise=true , address=145.79.227.14:4839 , piece_id=PW76MJZTJNUDTLCJ4GGQPLSEAJENKJYQOU5T62TAGCFTGQBNGFTA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:54:13.109258041 (314h25m12.980341086s) [noise=true , address=145.79.227.27:4691 , piece_id=4ROXPEVMSBJE7EF5MHM6RBEQKOPVGHPVV5RCCNGX6R2CKWWLIV7Q, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:54:13.109281937 (314h25m12.980364982s) [seg_pos={0 0}, upload=275696]: Segment reached optimal threshold of 13 pieces.
Dec 18 17:54:13.109482640 (314h25m12.980565705s) [noise=true , address=77.90.31.21:4880 , piece_id=EDFGJA5GIV2HY2KV55BDGMRZGMV3QDJ6OM5BOODGX4RJZBAAPAIA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF
Dec 18 17:54:13.109493139 (314h25m12.980576174s) [noise=true , address=77.90.31.21:4880 , piece_id=EDFGJA5GIV2HY2KV55BDGMRZGMV3QDJ6OM5BOODGX4RJZBAAPAIA, seg_pos={0 0}, upload=275696]: Long tail cancellation due to optimal threshold
Dec 18 17:54:13.109515152 (314h25m12.980598187s) [seg_pos={0 0}, upload=275696]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 18 17:54:13.109598921 (314h25m12.980681966s) [upload=275696]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 18 17:54:13.109606365 (314h25m12.9806894s) [upload=275696]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 18 17:54:13.397754392 (314h25m13.268837437s) [upload=275696]: Done uploading segments...
Dec 18 17:54:13.397776934 (314h25m13.268859969s) [upload=275696]: Upload finished. err: <nil>
Duration: 5.835074157s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
6.021175s trace:
https://jaeger.dev.storj.io/trace/66a9c91bc4a6ccf0
FAIL (2025-12-18 17:56:23 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:41370->45.38.24.2:443: read: connection reset by peer
Duration: 6.021207462s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [....................................................X.XXX...] duration:
5.896806312s trace:
https://jaeger.dev.storj.io/trace/55e899dcc96ebc44
FAIL (2025-12-18 17:56:19 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: UDNDAZHIL3CV4LIC3RUFACBHCTQHF6PL6EGDJLWM4FSMYUXC4SQA) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: JYBWCPDYGATJSR3H2ISPX2KB4ZKBVIETIOEGRNSZZLQDE7IPAJHQ) context deadline exceeded
======LOGS======
Dec 18 17:53:46.907446718 (314h24m46.778529753s) [upload=275696]: Starting upload
Dec 18 17:53:46.907512523 (314h24m46.778595578s) [upload=275696]: Done starting upload
Dec 18 17:53:46.907634715 (314h24m46.77871775s) [upload=275696]: Uploading segments...
Dec 18 17:53:46.907721671 (314h24m46.778804716s) [upload=275696]: Got next segment. Inline: false
Dec 18 17:53:46.907766126 (314h24m46.778849161s) [upload=275696]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 18 17:53:46.907771726 (314h24m46.778854761s) [upload=275696]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 18 17:53:47.154892897 (314h24m47.025975943s) [seg_pos={0 0}, upload=275696]: Begin upload segment...
Dec 18 17:53:47.154946931 (314h24m47.026029976s) [seg_pos={0 0}, upload=275696]: Done begin upload segment.
Dec 18 17:53:47.154978721 (314h24m47.026061756s) [noise=true , address=145.79.227.42:4122 , piece_id=A245U2W7Y4XGA5JNGKPWBOM2HDQTOIZPMYF6GEZXZGJGIPCM4JPQ, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.155041712 (314h24m47.026124747s) [noise=true , address=77.90.31.21:4880 , piece_id=EDFGJA5GIV2HY2KV55BDGMRZGMV3QDJ6OM5BOODGX4RJZBAAPAIA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.155513359 (314h24m47.026596394s) [noise=true , address=145.79.227.52:4446 , piece_id=KOGZIRIVHIRYRX6B4HPS27EZKTPK4AK45HM644HLBOA5QUJLDB6A, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.155877683 (314h24m47.026960718s) [noise=true , address=145.79.227.37:4566 , piece_id=TUE5OHICYNTQQ26VILHTZXFMTIGWVUQS7JMYJEARRYVUT6CMCJPA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.156232409 (314h24m47.027315454s) [noise=true , address=145.79.227.14:4839 , piece_id=PW76MJZTJNUDTLCJ4GGQPLSEAJENKJYQOU5T62TAGCFTGQBNGFTA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.156821220 (314h24m47.027904265s) [noise=true , address=145.79.227.27:4691 , piece_id=4ROXPEVMSBJE7EF5MHM6RBEQKOPVGHPVV5RCCNGX6R2CKWWLIV7Q, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.157223125 (314h24m47.02830616s) [noise=true , address=45.91.116.10:4081 , piece_id=JYBWCPDYGATJSR3H2ISPX2KB4ZKBVIETIOEGRNSZZLQDE7IPAJHQ, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.157618027 (314h24m47.028701062s) [noise=true , address=145.79.227.49:4339 , piece_id=CGSAGFA7UHNQ53CLLYZ2NMD57JK25P2YVPCMKW27W7HIE2OWI3UA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.158280779 (314h24m47.029363814s) [noise=true , address=45.91.116.11:4120 , piece_id=UDNDAZHIL3CV4LIC3RUFACBHCTQHF6PL6EGDJLWM4FSMYUXC4SQA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.159585022 (314h24m47.030668067s) [noise=true , address=145.79.227.41:4087 , piece_id=NKF2BL3HHWERKL5DPZ5SH73LS7XXT325TFWS3BSSXHETFMLL6VCQ, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.161483899 (314h24m47.032566944s) [noise=true , address=145.79.227.51:4411 , piece_id=RIEZSDOKZCRKA7KX775JPXEZVRZWIGOEAG62LSBRWPEVP4AQLUTA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.161534034 (314h24m47.032617079s) [noise=true , address=145.79.227.35:4481 , piece_id=ONU6UGBZKGYZN2KSAS57OCCKTFMDT5HMDCVJU5EIDYTENVRVOI2Q, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.163305847 (314h24m47.034388893s) [noise=true , address=145.79.227.45:4229 , piece_id=RDOEHHL2QNKQWZIRPWLOY6HW47OI4ZDXIPJHTR76BVQ3EFHZMKLA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.165089595 (314h24m47.03617264s) [noise=true , address=145.79.227.46:4601 , piece_id=3QTBNM4CT6QE74FQCOBDICM5UYXW32REQ25YEKQWXAVDC46RQBCA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.166839817 (314h24m47.037922862s) [noise=true , address=145.79.227.44:4195 , piece_id=3FIM4BHP2C62BSJXHU7GVYQ3DOC5YL3KOGDHNRMRXM3MJ6MWOQMQ, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.168326769 (314h24m47.039409814s) [noise=true , address=145.79.227.31:4798 , piece_id=W2VFZPXKGVORU5JCQIIP2IR5SCSEMWYRWJOKHFHBI3V7WCSMLBWA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:48.203795023 (314h24m48.074878059s) [noise=true , address=145.79.227.44:4195 , piece_id=3FIM4BHP2C62BSJXHU7GVYQ3DOC5YL3KOGDHNRMRXM3MJ6MWOQMQ, seg_pos={0 0}, upload=275696]: Done uploading piece. err: ecclient: upload failed (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, address:145.79.227.44:4195): protocol: storage node overloaded, request limit: 2000
Dec 18 17:53:48.207081231 (314h24m48.078164276s) [noise=true , address=145.79.227.31:4798 , piece_id=W2VFZPXKGVORU5JCQIIP2IR5SCSEMWYRWJOKHFHBI3V7WCSMLBWA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: storage node overloaded, request limit: 2000
Dec 18 17:53:48.863464901 (314h24m48.734547956s) [upload=275696]: Next returned nil segment
Dec 18 17:53:48.863489558 (314h24m48.734572623s) [upload=275696]: Waiting for error group managing segments...
Dec 18 17:53:49.149217626 (314h24m49.020300661s) [noise=true , address=145.79.227.51:4411 , piece_id=RIEZSDOKZCRKA7KX775JPXEZVRZWIGOEAG62LSBRWPEVP4AQLUTA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.151240638 (314h24m49.022323683s) [noise=true , address=145.79.227.52:4446 , piece_id=KOGZIRIVHIRYRX6B4HPS27EZKTPK4AK45HM644HLBOA5QUJLDB6A, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.159687424 (314h24m49.030770459s) [noise=true , address=145.79.227.46:4601 , piece_id=3QTBNM4CT6QE74FQCOBDICM5UYXW32REQ25YEKQWXAVDC46RQBCA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.163440961 (314h24m49.034524006s) [noise=true , address=145.79.227.37:4566 , piece_id=TUE5OHICYNTQQ26VILHTZXFMTIGWVUQS7JMYJEARRYVUT6CMCJPA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.166507440 (314h24m49.037590475s) [noise=true , address=145.79.227.49:4339 , piece_id=CGSAGFA7UHNQ53CLLYZ2NMD57JK25P2YVPCMKW27W7HIE2OWI3UA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.183896243 (314h24m49.054979288s) [noise=true , address=145.79.227.41:4087 , piece_id=NKF2BL3HHWERKL5DPZ5SH73LS7XXT325TFWS3BSSXHETFMLL6VCQ, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.219629698 (314h24m49.090712733s) [noise=true , address=145.79.227.45:4229 , piece_id=RDOEHHL2QNKQWZIRPWLOY6HW47OI4ZDXIPJHTR76BVQ3EFHZMKLA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.229654178 (314h24m49.100737213s) [noise=true , address=145.79.227.35:4481 , piece_id=ONU6UGBZKGYZN2KSAS57OCCKTFMDT5HMDCVJU5EIDYTENVRVOI2Q, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.348528404 (314h24m49.219611439s) [noise=true , address=145.79.227.42:4122 , piece_id=A245U2W7Y4XGA5JNGKPWBOM2HDQTOIZPMYF6GEZXZGJGIPCM4JPQ, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.361664419 (314h24m49.232747454s) [noise=true , address=45.91.116.11:4120 , piece_id=UDNDAZHIL3CV4LIC3RUFACBHCTQHF6PL6EGDJLWM4FSMYUXC4SQA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.524862349 (314h24m49.395945394s) [noise=true , address=45.91.116.10:4081 , piece_id=JYBWCPDYGATJSR3H2ISPX2KB4ZKBVIETIOEGRNSZZLQDE7IPAJHQ, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.698660224 (314h24m49.569743279s) [noise=true , address=145.79.227.14:4839 , piece_id=PW76MJZTJNUDTLCJ4GGQPLSEAJENKJYQOU5T62TAGCFTGQBNGFTA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:54:13.109258041 (314h25m12.980341086s) [noise=true , address=145.79.227.27:4691 , piece_id=4ROXPEVMSBJE7EF5MHM6RBEQKOPVGHPVV5RCCNGX6R2CKWWLIV7Q, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:54:13.109281937 (314h25m12.980364982s) [seg_pos={0 0}, upload=275696]: Segment reached optimal threshold of 13 pieces.
Dec 18 17:54:13.109482640 (314h25m12.980565705s) [noise=true , address=77.90.31.21:4880 , piece_id=EDFGJA5GIV2HY2KV55BDGMRZGMV3QDJ6OM5BOODGX4RJZBAAPAIA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF
Dec 18 17:54:13.109493139 (314h25m12.980576174s) [noise=true , address=77.90.31.21:4880 , piece_id=EDFGJA5GIV2HY2KV55BDGMRZGMV3QDJ6OM5BOODGX4RJZBAAPAIA, seg_pos={0 0}, upload=275696]: Long tail cancellation due to optimal threshold
Dec 18 17:54:13.109515152 (314h25m12.980598187s) [seg_pos={0 0}, upload=275696]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 18 17:54:13.109598921 (314h25m12.980681966s) [upload=275696]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 18 17:54:13.109606365 (314h25m12.9806894s) [upload=275696]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 18 17:54:13.397754392 (314h25m13.268837437s) [upload=275696]: Done uploading segments...
Dec 18 17:54:13.397776934 (314h25m13.268859969s) [upload=275696]: Upload finished. err: <nil>
Duration: 5.896831319s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.....................................................X.XXX..] duration:
5.925615548s trace:
https://jaeger.dev.storj.io/trace/55e899dcc96ebc44
FAIL (2025-12-18 17:55:37 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: UDNDAZHIL3CV4LIC3RUFACBHCTQHF6PL6EGDJLWM4FSMYUXC4SQA) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: JYBWCPDYGATJSR3H2ISPX2KB4ZKBVIETIOEGRNSZZLQDE7IPAJHQ) context deadline exceeded
======LOGS======
Dec 18 17:53:46.907446718 (314h24m46.778529753s) [upload=275696]: Starting upload
Dec 18 17:53:46.907512523 (314h24m46.778595578s) [upload=275696]: Done starting upload
Dec 18 17:53:46.907634715 (314h24m46.77871775s) [upload=275696]: Uploading segments...
Dec 18 17:53:46.907721671 (314h24m46.778804716s) [upload=275696]: Got next segment. Inline: false
Dec 18 17:53:46.907766126 (314h24m46.778849161s) [upload=275696]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 18 17:53:46.907771726 (314h24m46.778854761s) [upload=275696]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 18 17:53:47.154892897 (314h24m47.025975943s) [seg_pos={0 0}, upload=275696]: Begin upload segment...
Dec 18 17:53:47.154946931 (314h24m47.026029976s) [seg_pos={0 0}, upload=275696]: Done begin upload segment.
Dec 18 17:53:47.154978721 (314h24m47.026061756s) [noise=true , address=145.79.227.42:4122 , piece_id=A245U2W7Y4XGA5JNGKPWBOM2HDQTOIZPMYF6GEZXZGJGIPCM4JPQ, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.155041712 (314h24m47.026124747s) [noise=true , address=77.90.31.21:4880 , piece_id=EDFGJA5GIV2HY2KV55BDGMRZGMV3QDJ6OM5BOODGX4RJZBAAPAIA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.155513359 (314h24m47.026596394s) [noise=true , address=145.79.227.52:4446 , piece_id=KOGZIRIVHIRYRX6B4HPS27EZKTPK4AK45HM644HLBOA5QUJLDB6A, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.155877683 (314h24m47.026960718s) [noise=true , address=145.79.227.37:4566 , piece_id=TUE5OHICYNTQQ26VILHTZXFMTIGWVUQS7JMYJEARRYVUT6CMCJPA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.156232409 (314h24m47.027315454s) [noise=true , address=145.79.227.14:4839 , piece_id=PW76MJZTJNUDTLCJ4GGQPLSEAJENKJYQOU5T62TAGCFTGQBNGFTA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.156821220 (314h24m47.027904265s) [noise=true , address=145.79.227.27:4691 , piece_id=4ROXPEVMSBJE7EF5MHM6RBEQKOPVGHPVV5RCCNGX6R2CKWWLIV7Q, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.157223125 (314h24m47.02830616s) [noise=true , address=45.91.116.10:4081 , piece_id=JYBWCPDYGATJSR3H2ISPX2KB4ZKBVIETIOEGRNSZZLQDE7IPAJHQ, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.157618027 (314h24m47.028701062s) [noise=true , address=145.79.227.49:4339 , piece_id=CGSAGFA7UHNQ53CLLYZ2NMD57JK25P2YVPCMKW27W7HIE2OWI3UA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.158280779 (314h24m47.029363814s) [noise=true , address=45.91.116.11:4120 , piece_id=UDNDAZHIL3CV4LIC3RUFACBHCTQHF6PL6EGDJLWM4FSMYUXC4SQA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.159585022 (314h24m47.030668067s) [noise=true , address=145.79.227.41:4087 , piece_id=NKF2BL3HHWERKL5DPZ5SH73LS7XXT325TFWS3BSSXHETFMLL6VCQ, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.161483899 (314h24m47.032566944s) [noise=true , address=145.79.227.51:4411 , piece_id=RIEZSDOKZCRKA7KX775JPXEZVRZWIGOEAG62LSBRWPEVP4AQLUTA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.161534034 (314h24m47.032617079s) [noise=true , address=145.79.227.35:4481 , piece_id=ONU6UGBZKGYZN2KSAS57OCCKTFMDT5HMDCVJU5EIDYTENVRVOI2Q, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.163305847 (314h24m47.034388893s) [noise=true , address=145.79.227.45:4229 , piece_id=RDOEHHL2QNKQWZIRPWLOY6HW47OI4ZDXIPJHTR76BVQ3EFHZMKLA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.165089595 (314h24m47.03617264s) [noise=true , address=145.79.227.46:4601 , piece_id=3QTBNM4CT6QE74FQCOBDICM5UYXW32REQ25YEKQWXAVDC46RQBCA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.166839817 (314h24m47.037922862s) [noise=true , address=145.79.227.44:4195 , piece_id=3FIM4BHP2C62BSJXHU7GVYQ3DOC5YL3KOGDHNRMRXM3MJ6MWOQMQ, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.168326769 (314h24m47.039409814s) [noise=true , address=145.79.227.31:4798 , piece_id=W2VFZPXKGVORU5JCQIIP2IR5SCSEMWYRWJOKHFHBI3V7WCSMLBWA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:48.203795023 (314h24m48.074878059s) [noise=true , address=145.79.227.44:4195 , piece_id=3FIM4BHP2C62BSJXHU7GVYQ3DOC5YL3KOGDHNRMRXM3MJ6MWOQMQ, seg_pos={0 0}, upload=275696]: Done uploading piece. err: ecclient: upload failed (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, address:145.79.227.44:4195): protocol: storage node overloaded, request limit: 2000
Dec 18 17:53:48.207081231 (314h24m48.078164276s) [noise=true , address=145.79.227.31:4798 , piece_id=W2VFZPXKGVORU5JCQIIP2IR5SCSEMWYRWJOKHFHBI3V7WCSMLBWA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: storage node overloaded, request limit: 2000
Dec 18 17:53:48.863464901 (314h24m48.734547956s) [upload=275696]: Next returned nil segment
Dec 18 17:53:48.863489558 (314h24m48.734572623s) [upload=275696]: Waiting for error group managing segments...
Dec 18 17:53:49.149217626 (314h24m49.020300661s) [noise=true , address=145.79.227.51:4411 , piece_id=RIEZSDOKZCRKA7KX775JPXEZVRZWIGOEAG62LSBRWPEVP4AQLUTA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.151240638 (314h24m49.022323683s) [noise=true , address=145.79.227.52:4446 , piece_id=KOGZIRIVHIRYRX6B4HPS27EZKTPK4AK45HM644HLBOA5QUJLDB6A, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.159687424 (314h24m49.030770459s) [noise=true , address=145.79.227.46:4601 , piece_id=3QTBNM4CT6QE74FQCOBDICM5UYXW32REQ25YEKQWXAVDC46RQBCA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.163440961 (314h24m49.034524006s) [noise=true , address=145.79.227.37:4566 , piece_id=TUE5OHICYNTQQ26VILHTZXFMTIGWVUQS7JMYJEARRYVUT6CMCJPA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.166507440 (314h24m49.037590475s) [noise=true , address=145.79.227.49:4339 , piece_id=CGSAGFA7UHNQ53CLLYZ2NMD57JK25P2YVPCMKW27W7HIE2OWI3UA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.183896243 (314h24m49.054979288s) [noise=true , address=145.79.227.41:4087 , piece_id=NKF2BL3HHWERKL5DPZ5SH73LS7XXT325TFWS3BSSXHETFMLL6VCQ, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.219629698 (314h24m49.090712733s) [noise=true , address=145.79.227.45:4229 , piece_id=RDOEHHL2QNKQWZIRPWLOY6HW47OI4ZDXIPJHTR76BVQ3EFHZMKLA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.229654178 (314h24m49.100737213s) [noise=true , address=145.79.227.35:4481 , piece_id=ONU6UGBZKGYZN2KSAS57OCCKTFMDT5HMDCVJU5EIDYTENVRVOI2Q, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.348528404 (314h24m49.219611439s) [noise=true , address=145.79.227.42:4122 , piece_id=A245U2W7Y4XGA5JNGKPWBOM2HDQTOIZPMYF6GEZXZGJGIPCM4JPQ, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.361664419 (314h24m49.232747454s) [noise=true , address=45.91.116.11:4120 , piece_id=UDNDAZHIL3CV4LIC3RUFACBHCTQHF6PL6EGDJLWM4FSMYUXC4SQA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.524862349 (314h24m49.395945394s) [noise=true , address=45.91.116.10:4081 , piece_id=JYBWCPDYGATJSR3H2ISPX2KB4ZKBVIETIOEGRNSZZLQDE7IPAJHQ, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.698660224 (314h24m49.569743279s) [noise=true , address=145.79.227.14:4839 , piece_id=PW76MJZTJNUDTLCJ4GGQPLSEAJENKJYQOU5T62TAGCFTGQBNGFTA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:54:13.109258041 (314h25m12.980341086s) [noise=true , address=145.79.227.27:4691 , piece_id=4ROXPEVMSBJE7EF5MHM6RBEQKOPVGHPVV5RCCNGX6R2CKWWLIV7Q, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:54:13.109281937 (314h25m12.980364982s) [seg_pos={0 0}, upload=275696]: Segment reached optimal threshold of 13 pieces.
Dec 18 17:54:13.109482640 (314h25m12.980565705s) [noise=true , address=77.90.31.21:4880 , piece_id=EDFGJA5GIV2HY2KV55BDGMRZGMV3QDJ6OM5BOODGX4RJZBAAPAIA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF
Dec 18 17:54:13.109493139 (314h25m12.980576174s) [noise=true , address=77.90.31.21:4880 , piece_id=EDFGJA5GIV2HY2KV55BDGMRZGMV3QDJ6OM5BOODGX4RJZBAAPAIA, seg_pos={0 0}, upload=275696]: Long tail cancellation due to optimal threshold
Dec 18 17:54:13.109515152 (314h25m12.980598187s) [seg_pos={0 0}, upload=275696]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 18 17:54:13.109598921 (314h25m12.980681966s) [upload=275696]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 18 17:54:13.109606365 (314h25m12.9806894s) [upload=275696]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 18 17:54:13.397754392 (314h25m13.268837437s) [upload=275696]: Done uploading segments...
Dec 18 17:54:13.397776934 (314h25m13.268859969s) [upload=275696]: Upload finished. err: <nil>
Duration: 5.925643472s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
6.076283878s trace:
https://jaeger.dev.storj.io/trace/66a9c91bc4a6ccf0
FAIL (2025-12-18 17:55:31 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:41370->45.38.24.2:443: read: connection reset by peer
Duration: 6.076309748s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [......................................................X.XXX.] duration:
6.125579638s trace:
https://jaeger.dev.storj.io/trace/55e899dcc96ebc44
FAIL (2025-12-18 17:54:36 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: UDNDAZHIL3CV4LIC3RUFACBHCTQHF6PL6EGDJLWM4FSMYUXC4SQA) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: JYBWCPDYGATJSR3H2ISPX2KB4ZKBVIETIOEGRNSZZLQDE7IPAJHQ) context deadline exceeded
======LOGS======
Dec 18 17:53:46.907446718 (314h24m46.778529753s) [upload=275696]: Starting upload
Dec 18 17:53:46.907512523 (314h24m46.778595578s) [upload=275696]: Done starting upload
Dec 18 17:53:46.907634715 (314h24m46.77871775s) [upload=275696]: Uploading segments...
Dec 18 17:53:46.907721671 (314h24m46.778804716s) [upload=275696]: Got next segment. Inline: false
Dec 18 17:53:46.907766126 (314h24m46.778849161s) [upload=275696]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 18 17:53:46.907771726 (314h24m46.778854761s) [upload=275696]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 18 17:53:47.154892897 (314h24m47.025975943s) [seg_pos={0 0}, upload=275696]: Begin upload segment...
Dec 18 17:53:47.154946931 (314h24m47.026029976s) [seg_pos={0 0}, upload=275696]: Done begin upload segment.
Dec 18 17:53:47.154978721 (314h24m47.026061756s) [noise=true , address=145.79.227.42:4122 , piece_id=A245U2W7Y4XGA5JNGKPWBOM2HDQTOIZPMYF6GEZXZGJGIPCM4JPQ, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.155041712 (314h24m47.026124747s) [noise=true , address=77.90.31.21:4880 , piece_id=EDFGJA5GIV2HY2KV55BDGMRZGMV3QDJ6OM5BOODGX4RJZBAAPAIA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.155513359 (314h24m47.026596394s) [noise=true , address=145.79.227.52:4446 , piece_id=KOGZIRIVHIRYRX6B4HPS27EZKTPK4AK45HM644HLBOA5QUJLDB6A, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.155877683 (314h24m47.026960718s) [noise=true , address=145.79.227.37:4566 , piece_id=TUE5OHICYNTQQ26VILHTZXFMTIGWVUQS7JMYJEARRYVUT6CMCJPA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.156232409 (314h24m47.027315454s) [noise=true , address=145.79.227.14:4839 , piece_id=PW76MJZTJNUDTLCJ4GGQPLSEAJENKJYQOU5T62TAGCFTGQBNGFTA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.156821220 (314h24m47.027904265s) [noise=true , address=145.79.227.27:4691 , piece_id=4ROXPEVMSBJE7EF5MHM6RBEQKOPVGHPVV5RCCNGX6R2CKWWLIV7Q, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.157223125 (314h24m47.02830616s) [noise=true , address=45.91.116.10:4081 , piece_id=JYBWCPDYGATJSR3H2ISPX2KB4ZKBVIETIOEGRNSZZLQDE7IPAJHQ, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.157618027 (314h24m47.028701062s) [noise=true , address=145.79.227.49:4339 , piece_id=CGSAGFA7UHNQ53CLLYZ2NMD57JK25P2YVPCMKW27W7HIE2OWI3UA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.158280779 (314h24m47.029363814s) [noise=true , address=45.91.116.11:4120 , piece_id=UDNDAZHIL3CV4LIC3RUFACBHCTQHF6PL6EGDJLWM4FSMYUXC4SQA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.159585022 (314h24m47.030668067s) [noise=true , address=145.79.227.41:4087 , piece_id=NKF2BL3HHWERKL5DPZ5SH73LS7XXT325TFWS3BSSXHETFMLL6VCQ, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.161483899 (314h24m47.032566944s) [noise=true , address=145.79.227.51:4411 , piece_id=RIEZSDOKZCRKA7KX775JPXEZVRZWIGOEAG62LSBRWPEVP4AQLUTA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.161534034 (314h24m47.032617079s) [noise=true , address=145.79.227.35:4481 , piece_id=ONU6UGBZKGYZN2KSAS57OCCKTFMDT5HMDCVJU5EIDYTENVRVOI2Q, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.163305847 (314h24m47.034388893s) [noise=true , address=145.79.227.45:4229 , piece_id=RDOEHHL2QNKQWZIRPWLOY6HW47OI4ZDXIPJHTR76BVQ3EFHZMKLA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.165089595 (314h24m47.03617264s) [noise=true , address=145.79.227.46:4601 , piece_id=3QTBNM4CT6QE74FQCOBDICM5UYXW32REQ25YEKQWXAVDC46RQBCA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.166839817 (314h24m47.037922862s) [noise=true , address=145.79.227.44:4195 , piece_id=3FIM4BHP2C62BSJXHU7GVYQ3DOC5YL3KOGDHNRMRXM3MJ6MWOQMQ, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.168326769 (314h24m47.039409814s) [noise=true , address=145.79.227.31:4798 , piece_id=W2VFZPXKGVORU5JCQIIP2IR5SCSEMWYRWJOKHFHBI3V7WCSMLBWA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:48.203795023 (314h24m48.074878059s) [noise=true , address=145.79.227.44:4195 , piece_id=3FIM4BHP2C62BSJXHU7GVYQ3DOC5YL3KOGDHNRMRXM3MJ6MWOQMQ, seg_pos={0 0}, upload=275696]: Done uploading piece. err: ecclient: upload failed (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, address:145.79.227.44:4195): protocol: storage node overloaded, request limit: 2000
Dec 18 17:53:48.207081231 (314h24m48.078164276s) [noise=true , address=145.79.227.31:4798 , piece_id=W2VFZPXKGVORU5JCQIIP2IR5SCSEMWYRWJOKHFHBI3V7WCSMLBWA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: storage node overloaded, request limit: 2000
Dec 18 17:53:48.863464901 (314h24m48.734547956s) [upload=275696]: Next returned nil segment
Dec 18 17:53:48.863489558 (314h24m48.734572623s) [upload=275696]: Waiting for error group managing segments...
Dec 18 17:53:49.149217626 (314h24m49.020300661s) [noise=true , address=145.79.227.51:4411 , piece_id=RIEZSDOKZCRKA7KX775JPXEZVRZWIGOEAG62LSBRWPEVP4AQLUTA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.151240638 (314h24m49.022323683s) [noise=true , address=145.79.227.52:4446 , piece_id=KOGZIRIVHIRYRX6B4HPS27EZKTPK4AK45HM644HLBOA5QUJLDB6A, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.159687424 (314h24m49.030770459s) [noise=true , address=145.79.227.46:4601 , piece_id=3QTBNM4CT6QE74FQCOBDICM5UYXW32REQ25YEKQWXAVDC46RQBCA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.163440961 (314h24m49.034524006s) [noise=true , address=145.79.227.37:4566 , piece_id=TUE5OHICYNTQQ26VILHTZXFMTIGWVUQS7JMYJEARRYVUT6CMCJPA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.166507440 (314h24m49.037590475s) [noise=true , address=145.79.227.49:4339 , piece_id=CGSAGFA7UHNQ53CLLYZ2NMD57JK25P2YVPCMKW27W7HIE2OWI3UA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.183896243 (314h24m49.054979288s) [noise=true , address=145.79.227.41:4087 , piece_id=NKF2BL3HHWERKL5DPZ5SH73LS7XXT325TFWS3BSSXHETFMLL6VCQ, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.219629698 (314h24m49.090712733s) [noise=true , address=145.79.227.45:4229 , piece_id=RDOEHHL2QNKQWZIRPWLOY6HW47OI4ZDXIPJHTR76BVQ3EFHZMKLA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.229654178 (314h24m49.100737213s) [noise=true , address=145.79.227.35:4481 , piece_id=ONU6UGBZKGYZN2KSAS57OCCKTFMDT5HMDCVJU5EIDYTENVRVOI2Q, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.348528404 (314h24m49.219611439s) [noise=true , address=145.79.227.42:4122 , piece_id=A245U2W7Y4XGA5JNGKPWBOM2HDQTOIZPMYF6GEZXZGJGIPCM4JPQ, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.361664419 (314h24m49.232747454s) [noise=true , address=45.91.116.11:4120 , piece_id=UDNDAZHIL3CV4LIC3RUFACBHCTQHF6PL6EGDJLWM4FSMYUXC4SQA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.524862349 (314h24m49.395945394s) [noise=true , address=45.91.116.10:4081 , piece_id=JYBWCPDYGATJSR3H2ISPX2KB4ZKBVIETIOEGRNSZZLQDE7IPAJHQ, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.698660224 (314h24m49.569743279s) [noise=true , address=145.79.227.14:4839 , piece_id=PW76MJZTJNUDTLCJ4GGQPLSEAJENKJYQOU5T62TAGCFTGQBNGFTA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:54:13.109258041 (314h25m12.980341086s) [noise=true , address=145.79.227.27:4691 , piece_id=4ROXPEVMSBJE7EF5MHM6RBEQKOPVGHPVV5RCCNGX6R2CKWWLIV7Q, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:54:13.109281937 (314h25m12.980364982s) [seg_pos={0 0}, upload=275696]: Segment reached optimal threshold of 13 pieces.
Dec 18 17:54:13.109482640 (314h25m12.980565705s) [noise=true , address=77.90.31.21:4880 , piece_id=EDFGJA5GIV2HY2KV55BDGMRZGMV3QDJ6OM5BOODGX4RJZBAAPAIA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF
Dec 18 17:54:13.109493139 (314h25m12.980576174s) [noise=true , address=77.90.31.21:4880 , piece_id=EDFGJA5GIV2HY2KV55BDGMRZGMV3QDJ6OM5BOODGX4RJZBAAPAIA, seg_pos={0 0}, upload=275696]: Long tail cancellation due to optimal threshold
Dec 18 17:54:13.109515152 (314h25m12.980598187s) [seg_pos={0 0}, upload=275696]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 18 17:54:13.109598921 (314h25m12.980681966s) [upload=275696]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 18 17:54:13.109606365 (314h25m12.9806894s) [upload=275696]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 18 17:54:13.397754392 (314h25m13.268837437s) [upload=275696]: Done uploading segments...
Dec 18 17:54:13.397776934 (314h25m13.268859969s) [upload=275696]: Upload finished. err: <nil>
Duration: 6.125604746s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
6.497719829s trace:
https://jaeger.dev.storj.io/trace/66a9c91bc4a6ccf0
FAIL (2025-12-18 17:54:28 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:41370->45.38.24.2:443: read: connection reset by peer
Duration: 6.497745738s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.......................................................X.XXX] duration:
30.003959425s trace:
https://jaeger.dev.storj.io/trace/55e899dcc96ebc44
FAIL (2025-12-18 17:54:16 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: UDNDAZHIL3CV4LIC3RUFACBHCTQHF6PL6EGDJLWM4FSMYUXC4SQA) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: JYBWCPDYGATJSR3H2ISPX2KB4ZKBVIETIOEGRNSZZLQDE7IPAJHQ) context deadline exceeded
======LOGS======
Dec 18 17:53:46.907446718 (314h24m46.778529753s) [upload=275696]: Starting upload
Dec 18 17:53:46.907512523 (314h24m46.778595578s) [upload=275696]: Done starting upload
Dec 18 17:53:46.907634715 (314h24m46.77871775s) [upload=275696]: Uploading segments...
Dec 18 17:53:46.907721671 (314h24m46.778804716s) [upload=275696]: Got next segment. Inline: false
Dec 18 17:53:46.907766126 (314h24m46.778849161s) [upload=275696]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 18 17:53:46.907771726 (314h24m46.778854761s) [upload=275696]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 18 17:53:47.154892897 (314h24m47.025975943s) [seg_pos={0 0}, upload=275696]: Begin upload segment...
Dec 18 17:53:47.154946931 (314h24m47.026029976s) [seg_pos={0 0}, upload=275696]: Done begin upload segment.
Dec 18 17:53:47.154978721 (314h24m47.026061756s) [noise=true , address=145.79.227.42:4122 , piece_id=A245U2W7Y4XGA5JNGKPWBOM2HDQTOIZPMYF6GEZXZGJGIPCM4JPQ, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.155041712 (314h24m47.026124747s) [noise=true , address=77.90.31.21:4880 , piece_id=EDFGJA5GIV2HY2KV55BDGMRZGMV3QDJ6OM5BOODGX4RJZBAAPAIA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.155513359 (314h24m47.026596394s) [noise=true , address=145.79.227.52:4446 , piece_id=KOGZIRIVHIRYRX6B4HPS27EZKTPK4AK45HM644HLBOA5QUJLDB6A, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.155877683 (314h24m47.026960718s) [noise=true , address=145.79.227.37:4566 , piece_id=TUE5OHICYNTQQ26VILHTZXFMTIGWVUQS7JMYJEARRYVUT6CMCJPA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.156232409 (314h24m47.027315454s) [noise=true , address=145.79.227.14:4839 , piece_id=PW76MJZTJNUDTLCJ4GGQPLSEAJENKJYQOU5T62TAGCFTGQBNGFTA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.156821220 (314h24m47.027904265s) [noise=true , address=145.79.227.27:4691 , piece_id=4ROXPEVMSBJE7EF5MHM6RBEQKOPVGHPVV5RCCNGX6R2CKWWLIV7Q, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.157223125 (314h24m47.02830616s) [noise=true , address=45.91.116.10:4081 , piece_id=JYBWCPDYGATJSR3H2ISPX2KB4ZKBVIETIOEGRNSZZLQDE7IPAJHQ, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.157618027 (314h24m47.028701062s) [noise=true , address=145.79.227.49:4339 , piece_id=CGSAGFA7UHNQ53CLLYZ2NMD57JK25P2YVPCMKW27W7HIE2OWI3UA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.158280779 (314h24m47.029363814s) [noise=true , address=45.91.116.11:4120 , piece_id=UDNDAZHIL3CV4LIC3RUFACBHCTQHF6PL6EGDJLWM4FSMYUXC4SQA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.159585022 (314h24m47.030668067s) [noise=true , address=145.79.227.41:4087 , piece_id=NKF2BL3HHWERKL5DPZ5SH73LS7XXT325TFWS3BSSXHETFMLL6VCQ, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.161483899 (314h24m47.032566944s) [noise=true , address=145.79.227.51:4411 , piece_id=RIEZSDOKZCRKA7KX775JPXEZVRZWIGOEAG62LSBRWPEVP4AQLUTA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.161534034 (314h24m47.032617079s) [noise=true , address=145.79.227.35:4481 , piece_id=ONU6UGBZKGYZN2KSAS57OCCKTFMDT5HMDCVJU5EIDYTENVRVOI2Q, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.163305847 (314h24m47.034388893s) [noise=true , address=145.79.227.45:4229 , piece_id=RDOEHHL2QNKQWZIRPWLOY6HW47OI4ZDXIPJHTR76BVQ3EFHZMKLA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.165089595 (314h24m47.03617264s) [noise=true , address=145.79.227.46:4601 , piece_id=3QTBNM4CT6QE74FQCOBDICM5UYXW32REQ25YEKQWXAVDC46RQBCA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.166839817 (314h24m47.037922862s) [noise=true , address=145.79.227.44:4195 , piece_id=3FIM4BHP2C62BSJXHU7GVYQ3DOC5YL3KOGDHNRMRXM3MJ6MWOQMQ, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:47.168326769 (314h24m47.039409814s) [noise=true , address=145.79.227.31:4798 , piece_id=W2VFZPXKGVORU5JCQIIP2IR5SCSEMWYRWJOKHFHBI3V7WCSMLBWA, seg_pos={0 0}, upload=275696]: Uploading piece...
Dec 18 17:53:48.203795023 (314h24m48.074878059s) [noise=true , address=145.79.227.44:4195 , piece_id=3FIM4BHP2C62BSJXHU7GVYQ3DOC5YL3KOGDHNRMRXM3MJ6MWOQMQ, seg_pos={0 0}, upload=275696]: Done uploading piece. err: ecclient: upload failed (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, address:145.79.227.44:4195): protocol: storage node overloaded, request limit: 2000
Dec 18 17:53:48.207081231 (314h24m48.078164276s) [noise=true , address=145.79.227.31:4798 , piece_id=W2VFZPXKGVORU5JCQIIP2IR5SCSEMWYRWJOKHFHBI3V7WCSMLBWA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: storage node overloaded, request limit: 2000
Dec 18 17:53:48.863464901 (314h24m48.734547956s) [upload=275696]: Next returned nil segment
Dec 18 17:53:48.863489558 (314h24m48.734572623s) [upload=275696]: Waiting for error group managing segments...
Dec 18 17:53:49.149217626 (314h24m49.020300661s) [noise=true , address=145.79.227.51:4411 , piece_id=RIEZSDOKZCRKA7KX775JPXEZVRZWIGOEAG62LSBRWPEVP4AQLUTA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.151240638 (314h24m49.022323683s) [noise=true , address=145.79.227.52:4446 , piece_id=KOGZIRIVHIRYRX6B4HPS27EZKTPK4AK45HM644HLBOA5QUJLDB6A, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.159687424 (314h24m49.030770459s) [noise=true , address=145.79.227.46:4601 , piece_id=3QTBNM4CT6QE74FQCOBDICM5UYXW32REQ25YEKQWXAVDC46RQBCA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.163440961 (314h24m49.034524006s) [noise=true , address=145.79.227.37:4566 , piece_id=TUE5OHICYNTQQ26VILHTZXFMTIGWVUQS7JMYJEARRYVUT6CMCJPA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.166507440 (314h24m49.037590475s) [noise=true , address=145.79.227.49:4339 , piece_id=CGSAGFA7UHNQ53CLLYZ2NMD57JK25P2YVPCMKW27W7HIE2OWI3UA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.183896243 (314h24m49.054979288s) [noise=true , address=145.79.227.41:4087 , piece_id=NKF2BL3HHWERKL5DPZ5SH73LS7XXT325TFWS3BSSXHETFMLL6VCQ, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.219629698 (314h24m49.090712733s) [noise=true , address=145.79.227.45:4229 , piece_id=RDOEHHL2QNKQWZIRPWLOY6HW47OI4ZDXIPJHTR76BVQ3EFHZMKLA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.229654178 (314h24m49.100737213s) [noise=true , address=145.79.227.35:4481 , piece_id=ONU6UGBZKGYZN2KSAS57OCCKTFMDT5HMDCVJU5EIDYTENVRVOI2Q, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.348528404 (314h24m49.219611439s) [noise=true , address=145.79.227.42:4122 , piece_id=A245U2W7Y4XGA5JNGKPWBOM2HDQTOIZPMYF6GEZXZGJGIPCM4JPQ, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.361664419 (314h24m49.232747454s) [noise=true , address=45.91.116.11:4120 , piece_id=UDNDAZHIL3CV4LIC3RUFACBHCTQHF6PL6EGDJLWM4FSMYUXC4SQA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.524862349 (314h24m49.395945394s) [noise=true , address=45.91.116.10:4081 , piece_id=JYBWCPDYGATJSR3H2ISPX2KB4ZKBVIETIOEGRNSZZLQDE7IPAJHQ, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:53:49.698660224 (314h24m49.569743279s) [noise=true , address=145.79.227.14:4839 , piece_id=PW76MJZTJNUDTLCJ4GGQPLSEAJENKJYQOU5T62TAGCFTGQBNGFTA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:54:13.109258041 (314h25m12.980341086s) [noise=true , address=145.79.227.27:4691 , piece_id=4ROXPEVMSBJE7EF5MHM6RBEQKOPVGHPVV5RCCNGX6R2CKWWLIV7Q, seg_pos={0 0}, upload=275696]: Done uploading piece. err: <nil>
Dec 18 17:54:13.109281937 (314h25m12.980364982s) [seg_pos={0 0}, upload=275696]: Segment reached optimal threshold of 13 pieces.
Dec 18 17:54:13.109482640 (314h25m12.980565705s) [noise=true , address=77.90.31.21:4880 , piece_id=EDFGJA5GIV2HY2KV55BDGMRZGMV3QDJ6OM5BOODGX4RJZBAAPAIA, seg_pos={0 0}, upload=275696]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF
Dec 18 17:54:13.109493139 (314h25m12.980576174s) [noise=true , address=77.90.31.21:4880 , piece_id=EDFGJA5GIV2HY2KV55BDGMRZGMV3QDJ6OM5BOODGX4RJZBAAPAIA, seg_pos={0 0}, upload=275696]: Long tail cancellation due to optimal threshold
Dec 18 17:54:13.109515152 (314h25m12.980598187s) [seg_pos={0 0}, upload=275696]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 18 17:54:13.109598921 (314h25m12.980681966s) [upload=275696]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 18 17:54:13.109606365 (314h25m12.9806894s) [upload=275696]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 18 17:54:13.397754392 (314h25m13.268837437s) [upload=275696]: Done uploading segments...
Dec 18 17:54:13.397776934 (314h25m13.268859969s) [upload=275696]: Upload finished. err: <nil>
Duration: 30.003973942s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [........................................................X.XX] duration:
30.246127349s trace:
https://jaeger.dev.storj.io/trace/59a4497d48618fd7
FAIL (2025-12-18 17:53:45 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 18 17:53:16.435581396 (314h24m16.306664441s) [upload=275686]: Starting upload
Dec 18 17:53:16.435657491 (314h24m16.306740536s) [upload=275686]: Done starting upload
Dec 18 17:53:16.435685555 (314h24m16.30676859s) [upload=275686]: Uploading segments...
Dec 18 17:53:16.435901205 (314h24m16.30698425s) [upload=275686]: Got next segment. Inline: false
Dec 18 17:53:16.435941623 (314h24m16.307024658s) [upload=275686]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 18 17:53:16.435946783 (314h24m16.307029818s) [upload=275686]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 18 17:53:16.685100133 (314h24m16.556183178s) [seg_pos={0 0}, upload=275686]: Begin upload segment...
Dec 18 17:53:16.685153314 (314h24m16.556236359s) [seg_pos={0 0}, upload=275686]: Done begin upload segment.
Dec 18 17:53:16.685181608 (314h24m16.556264653s) [noise=true , address=77.90.31.14:4663 , piece_id=LS73IYJ7ERIX5NY5Z4XXFGQ5R7LJRIFSRAN2PFHI74IJ5OAYJFDA, seg_pos={0 0}, upload=275686]: Uploading piece...
Dec 18 17:53:16.685193211 (314h24m16.556276256s) [noise=true , address=145.79.227.29:4943 , piece_id=G46DM5DAQ5WI6QFBSX65XJZWBBLSPU7VCLMQWUZAHUHDREGOGTXQ, seg_pos={0 0}, upload=275686]: Uploading piece...
Dec 18 17:53:16.685748969 (314h24m16.556832014s) [noise=true , address=145.79.227.37:4567 , piece_id=H6UX36WVFZQYA2JHHCPCSO544GFJPOH7PUBON36PQU24HGUXNYHQ, seg_pos={0 0}, upload=275686]: Uploading piece...
Dec 18 17:53:16.686303004 (314h24m16.557386049s) [noise=true , address=145.79.227.28:4726 , piece_id=3S3VUJ3XXDBFHKPFREMOWVWHA2QQUUKGMWN26UPC672MCBTUZ6YQ, seg_pos={0 0}, upload=275686]: Uploading piece...
Dec 18 17:53:16.686736038 (314h24m16.557819083s) [noise=true , address=145.79.227.48:4302 , piece_id=VL2HUMQM3C6Q3HJDAGSOX2OK74BDOU7NE7OAES3RA3E4GIDNURSA, seg_pos={0 0}, upload=275686]: Uploading piece...
Dec 18 17:53:16.687895315 (314h24m16.55897836s) [noise=true , address=77.90.31.17:4773 , piece_id=SUNV3BYXCAVFPLYLCKATNVYVCM7RY6JGMBMAHNYQYKCRKG2D2R2A, seg_pos={0 0}, upload=275686]: Uploading piece...
Dec 18 17:53:16.688991413 (314h24m16.560074448s) [noise=true , address=145.79.227.35:4481 , piece_id=CVXGM5WOD67XC36K7IXINXZQLRSGH43GZYNOUEIIW44YC4DP7BSQ, seg_pos={0 0}, upload=275686]: Uploading piece...
Dec 18 17:53:16.689280563 (314h24m16.560363598s) [noise=true , address=145.79.227.19:4079 , piece_id=XXNPL6Y27RFYIHFQVEAA5VWVEF2JTNZ4Z4NS2OYAIPKTFFETOWMQ, seg_pos={0 0}, upload=275686]: Uploading piece...
Dec 18 17:53:16.691000158 (314h24m16.562083203s) [noise=true , address=145.79.227.51:4410 , piece_id=UDOYF2XRWSBS3RAJIMSE7C7QYWHIBQ5CP55PXLJFDWCPJ7XT6JHQ, seg_pos={0 0}, upload=275686]: Uploading piece...
Dec 18 17:53:16.692912249 (314h24m16.563995294s) [noise=true , address=45.91.116.6:4974 , piece_id=XEMOVDY5ETYPWKWMSTMME5YAY3PF4RJR4XAHNY7J6SMCL3DWVD3Q, seg_pos={0 0}, upload=275686]: Uploading piece...
Dec 18 17:53:16.694741663 (314h24m16.565824708s) [noise=true , address=145.79.227.41:4085 , piece_id=OEP5TKT7FYOENTIFBI67MSXND3RN3SM33QW53DLFR6VMRIK74KDA, seg_pos={0 0}, upload=275686]: Uploading piece...
Dec 18 17:53:16.695487293 (314h24m16.566570328s) [noise=true , address=145.79.227.49:4339 , piece_id=AUEAPAY7FAVJBO6ILFTCST2ILMAMN6PCDH3TSJXAWWZ4Y3DZREFQ, seg_pos={0 0}, upload=275686]: Uploading piece...
Dec 18 17:53:16.696194219 (314h24m16.567277265s) [noise=true , address=145.79.227.38:4979 , piece_id=YJ77EH6WMQX3ZB6WC2M5EM2RNFPY74BMWA4JERDWXPJIDUQFLF6Q, seg_pos={0 0}, upload=275686]: Uploading piece...
Dec 18 17:53:16.697298522 (314h24m16.568381567s) [noise=true , address=77.90.31.16:4736 , piece_id=AHHE23UPYUGEHP65GSM2WXVJQXN3MXISZTOHKGVEDJENLWASU3QQ, seg_pos={0 0}, upload=275686]: Uploading piece...
Dec 18 17:53:16.697839412 (314h24m16.568922457s) [noise=true , address=145.79.227.32:4835 , piece_id=X4ENVFONL4BMQZZY7S73XFDBREKT7GKPK6LCUWZTENEHEMYA63ZA, seg_pos={0 0}, upload=275686]: Uploading piece...
Dec 18 17:53:16.699115041 (314h24m16.570198076s) [noise=true , address=145.79.227.40:4051 , piece_id=5KRKYJQPVKYRVELUW7BUMRJHHPT6MHY5JZE4R67BU772IX2V56YQ, seg_pos={0 0}, upload=275686]: Uploading piece...
Dec 18 17:53:17.979191831 (314h24m17.850274886s) [noise=true , address=145.79.227.38:4979 , piece_id=YJ77EH6WMQX3ZB6WC2M5EM2RNFPY74BMWA4JERDWXPJIDUQFLF6Q, seg_pos={0 0}, upload=275686]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: storage node overloaded, request limit: 2000
Dec 18 17:53:18.372465093 (314h24m18.243548129s) [upload=275686]: Next returned nil segment
Dec 18 17:53:18.372479340 (314h24m18.243562375s) [upload=275686]: Waiting for error group managing segments...
Dec 18 17:53:18.671149584 (314h24m18.542232629s) [noise=true , address=145.79.227.37:4567 , piece_id=H6UX36WVFZQYA2JHHCPCSO544GFJPOH7PUBON36PQU24HGUXNYHQ, seg_pos={0 0}, upload=275686]: Done uploading piece. err: <nil>
Dec 18 17:53:18.678018804 (314h24m18.549101839s) [noise=true , address=145.79.227.35:4481 , piece_id=CVXGM5WOD67XC36K7IXINXZQLRSGH43GZYNOUEIIW44YC4DP7BSQ, seg_pos={0 0}, upload=275686]: Done uploading piece. err: <nil>
Dec 18 17:53:18.686832508 (314h24m18.557915553s) [noise=true , address=145.79.227.48:4302 , piece_id=VL2HUMQM3C6Q3HJDAGSOX2OK74BDOU7NE7OAES3RA3E4GIDNURSA, seg_pos={0 0}, upload=275686]: Done uploading piece. err: <nil>
Dec 18 17:53:18.693921187 (314h24m18.565004232s) [noise=true , address=145.79.227.29:4943 , piece_id=G46DM5DAQ5WI6QFBSX65XJZWBBLSPU7VCLMQWUZAHUHDREGOGTXQ, seg_pos={0 0}, upload=275686]: Done uploading piece. err: <nil>
Dec 18 17:53:18.699251437 (314h24m18.570334472s) [noise=true , address=145.79.227.28:4726 , piece_id=3S3VUJ3XXDBFHKPFREMOWVWHA2QQUUKGMWN26UPC672MCBTUZ6YQ, seg_pos={0 0}, upload=275686]: Done uploading piece. err: <nil>
Dec 18 17:53:18.699691244 (314h24m18.570774289s) [noise=true , address=145.79.227.49:4339 , piece_id=AUEAPAY7FAVJBO6ILFTCST2ILMAMN6PCDH3TSJXAWWZ4Y3DZREFQ, seg_pos={0 0}, upload=275686]: Done uploading piece. err: <nil>
Dec 18 17:53:18.700014229 (314h24m18.571097264s) [noise=true , address=145.79.227.19:4079 , piece_id=XXNPL6Y27RFYIHFQVEAA5VWVEF2JTNZ4Z4NS2OYAIPKTFFETOWMQ, seg_pos={0 0}, upload=275686]: Done uploading piece. err: <nil>
Dec 18 17:53:18.736991483 (314h24m18.608074518s) [noise=true , address=145.79.227.51:4410 , piece_id=UDOYF2XRWSBS3RAJIMSE7C7QYWHIBQ5CP55PXLJFDWCPJ7XT6JHQ, seg_pos={0 0}, upload=275686]: Done uploading piece. err: <nil>
Dec 18 17:53:18.836077098 (314h24m18.707160143s) [noise=true , address=77.90.31.17:4773 , piece_id=SUNV3BYXCAVFPLYLCKATNVYVCM7RY6JGMBMAHNYQYKCRKG2D2R2A, seg_pos={0 0}, upload=275686]: Done uploading piece. err: <nil>
Dec 18 17:53:18.908648502 (314h24m18.779731537s) [noise=true , address=77.90.31.16:4736 , piece_id=AHHE23UPYUGEHP65GSM2WXVJQXN3MXISZTOHKGVEDJENLWASU3QQ, seg_pos={0 0}, upload=275686]: Done uploading piece. err: <nil>
Dec 18 17:53:18.979319226 (314h24m18.850402262s) [noise=true , address=77.90.31.14:4663 , piece_id=LS73IYJ7ERIX5NY5Z4XXFGQ5R7LJRIFSRAN2PFHI74IJ5OAYJFDA, seg_pos={0 0}, upload=275686]: Done uploading piece. err: <nil>
Dec 18 17:53:19.521117648 (314h24m19.392200683s) [noise=true , address=45.91.116.6:4974 , piece_id=XEMOVDY5ETYPWKWMSTMME5YAY3PF4RJR4XAHNY7J6SMCL3DWVD3Q, seg_pos={0 0}, upload=275686]: Done uploading piece. err: <nil>
Dec 18 17:53:45.751762793 (314h24m45.622845828s) [noise=true , address=145.79.227.32:4835 , piece_id=X4ENVFONL4BMQZZY7S73XFDBREKT7GKPK6LCUWZTENEHEMYA63ZA, seg_pos={0 0}, upload=275686]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF
Dec 18 17:53:45.751830853 (314h24m45.622913888s) [noise=true , address=145.79.227.40:4051 , piece_id=5KRKYJQPVKYRVELUW7BUMRJHHPT6MHY5JZE4R67BU772IX2V56YQ, seg_pos={0 0}, upload=275686]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: expected piece hash; context deadline exceeded; EOF
Dec 18 17:53:45.751849178 (314h24m45.622932223s) [noise=true , address=145.79.227.41:4085 , piece_id=OEP5TKT7FYOENTIFBI67MSXND3RN3SM33QW53DLFR6VMRIK74KDA, seg_pos={0 0}, upload=275686]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:145.79.227.41:4085): protocol: expected piece hash; context deadline exceeded; EOF
Dec 18 17:53:45.751880488 (314h24m45.622963523s) [seg_pos={0 0}, upload=275686]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 18 17:53:45.995474753 (314h24m45.866557788s) [upload=275686]: Done uploading segments...
Dec 18 17:53:45.995509449 (314h24m45.866592484s) [upload=275686]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 30.246152666s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
5.915169509s trace:
https://jaeger.dev.storj.io/trace/66a9c91bc4a6ccf0
FAIL (2025-12-18 17:53:37 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:41370->45.38.24.2:443: read: connection reset by peer
Duration: 5.915194696s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........................................................X.X] duration:
30.24862049s trace:
https://jaeger.dev.storj.io/trace/437bc02b3adf475f
FAIL (2025-12-18 17:53:14 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 18 17:52:44.740777961 (314h23m44.611861006s) [upload=275679]: Starting upload
Dec 18 17:52:44.740858534 (314h23m44.611941579s) [upload=275679]: Done starting upload
Dec 18 17:52:44.740995044 (314h23m44.612078089s) [upload=275679]: Uploading segments...
Dec 18 17:52:44.741641195 (314h23m44.61272425s) [upload=275679]: Got next segment. Inline: false
Dec 18 17:52:44.741689917 (314h23m44.612772962s) [upload=275679]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 18 17:52:44.741696661 (314h23m44.612779696s) [upload=275679]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 18 17:52:44.991895803 (314h23m44.862978868s) [seg_pos={0 0}, upload=275679]: Begin upload segment...
Dec 18 17:52:44.991960025 (314h23m44.86304308s) [seg_pos={0 0}, upload=275679]: Done begin upload segment.
Dec 18 17:52:44.992001244 (314h23m44.863084299s) [noise=true , address=77.90.31.11:4553 , piece_id=5JR4WR7UM45LIMLMYPTO5N3UEFBS7JBYWCDKUDMPVMNVZJWAMRCQ, seg_pos={0 0}, upload=275679]: Uploading piece...
Dec 18 17:52:44.993332879 (314h23m44.864415914s) [noise=true , address=145.79.227.50:4374 , piece_id=2HRM2RBR3SYHPEM4HSPCQUNWTEXERVHP3U5XKOQDI46CPYELJECQ, seg_pos={0 0}, upload=275679]: Uploading piece...
Dec 18 17:52:44.993850485 (314h23m44.86493353s) [noise=true , address=145.79.227.49:4337 , piece_id=YUBQJZBTELQN4FAEFPZKETB7EPLMYIPQD74EYIW3IV3Z5ZEV43EA, seg_pos={0 0}, upload=275679]: Uploading piece...
Dec 18 17:52:44.994230208 (314h23m44.865313243s) [noise=true , address=145.79.227.45:4230 , piece_id=ZXCAIAA6F3VK7YEEM6KJNEQBK6OW76VAZUZJDREMDXU26ZBTEVAQ, seg_pos={0 0}, upload=275679]: Uploading piece...
Dec 18 17:52:44.994547352 (314h23m44.865630397s) [noise=true , address=145.79.227.39:4014 , piece_id=SBOPD3T6BA3TYAUDATBGR2TFXRJR4UBSEUFBRFEOPR3IORUDXX5A, seg_pos={0 0}, upload=275679]: Uploading piece...
Dec 18 17:52:44.994867923 (314h23m44.865950958s) [noise=true , address=77.90.31.13:4627 , piece_id=VMT2UW5V7I35CQ4T5MQAPG2Q5TQT7ZWZ26R3H4UQZNWS3QWOXWAA, seg_pos={0 0}, upload=275679]: Uploading piece...
Dec 18 17:52:44.995155520 (314h23m44.866238555s) [noise=true , address=145.79.227.44:4195 , piece_id=N7HNJJ3WUPCTESCL2757P2T2CY5WLJ4234NFDH5NZULJMXTHKN2A, seg_pos={0 0}, upload=275679]: Uploading piece...
Dec 18 17:52:44.995509264 (314h23m44.866592309s) [noise=true , address=145.79.227.9:4366 , piece_id=LC3BXTGNY6HZVKVRCWXCHXI2HCER2XYXA5EY3B5EDZOROBSXTOUA, seg_pos={0 0}, upload=275679]: Uploading piece...
Dec 18 17:52:44.995832479 (314h23m44.866915514s) [noise=true , address=145.79.227.46:4602 , piece_id=BRDASXODM3KXOXXZHOHWZH57ROCHAMTYPX3RHK4HB3P3TB2R6O5A, seg_pos={0 0}, upload=275679]: Uploading piece...
Dec 18 17:52:44.996171324 (314h23m44.867254359s) [noise=true , address=45.91.116.12:4226 , piece_id=7SG6A2PFALY2J3EXUGSAENBASROUNU6CDADDL63NW5HMBRSOCWLA, seg_pos={0 0}, upload=275679]: Uploading piece...
Dec 18 17:52:44.996535729 (314h23m44.867618774s) [noise=true , address=45.91.116.14:4191 , piece_id=PHZXDLNEZIFFCCPIAX25DOFFUTRNQ2T6QXYHNMC3BS2FIE65CETA, seg_pos={0 0}, upload=275679]: Uploading piece...
Dec 18 17:52:44.996872900 (314h23m44.867955945s) [noise=true , address=145.79.227.38:4978 , piece_id=WMSYETGNJNU2AS5U3YPB4TU3EV2BRCDE4CRSTYQCYNPIEQSZRO7Q, seg_pos={0 0}, upload=275679]: Uploading piece...
Dec 18 17:52:44.997246812 (314h23m44.868329847s) [noise=true , address=145.79.227.32:4835 , piece_id=BDCCTSNIO5XWLP5G4VOXKCIDWJ3LFKEQWEGM37EYGQYSN4TNTAEQ, seg_pos={0 0}, upload=275679]: Uploading piece...
Dec 18 17:52:44.997648127 (314h23m44.868731172s) [noise=true , address=145.79.227.51:4410 , piece_id=4N5X2DROGKKWVKRZ45AF2EQTOKIAJCIY7EAS7T7HLL5ONC4WHKXA, seg_pos={0 0}, upload=275679]: Uploading piece...
Dec 18 17:52:44.998625608 (314h23m44.869708643s) [noise=true , address=77.90.31.10:4518 , piece_id=LALBF2ROEGO5MRGBZW7HSOIL53C3AY2HEIBRLK6HNUCDAOCDWFPA, seg_pos={0 0}, upload=275679]: Uploading piece...
Dec 18 17:52:44.998994370 (314h23m44.870077415s) [noise=true , address=145.79.227.47:4267 , piece_id=TTDS56UKSLXX4PQUUSBRETQE2ZXRCA2GAJQBGBUIRYKK3LSS2CBA, seg_pos={0 0}, upload=275679]: Uploading piece...
Dec 18 17:52:46.008177570 (314h23m45.879260625s) [noise=true , address=145.79.227.44:4195 , piece_id=N7HNJJ3WUPCTESCL2757P2T2CY5WLJ4234NFDH5NZULJMXTHKN2A, seg_pos={0 0}, upload=275679]: Done uploading piece. err: ecclient: upload failed (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, address:145.79.227.44:4195): protocol: storage node overloaded, request limit: 2000
Dec 18 17:52:46.035403435 (314h23m45.90648647s) [noise=true , address=145.79.227.38:4978 , piece_id=WMSYETGNJNU2AS5U3YPB4TU3EV2BRCDE4CRSTYQCYNPIEQSZRO7Q, seg_pos={0 0}, upload=275679]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: storage node overloaded, request limit: 2000
Dec 18 17:52:46.601523981 (314h23m46.472607026s) [upload=275679]: Next returned nil segment
Dec 18 17:52:46.601538860 (314h23m46.472621905s) [upload=275679]: Waiting for error group managing segments...
Dec 18 17:52:46.997264972 (314h23m46.868348017s) [noise=true , address=145.79.227.50:4374 , piece_id=2HRM2RBR3SYHPEM4HSPCQUNWTEXERVHP3U5XKOQDI46CPYELJECQ, seg_pos={0 0}, upload=275679]: Done uploading piece. err: <nil>
Dec 18 17:52:47.001736858 (314h23m46.872819903s) [noise=true , address=145.79.227.51:4410 , piece_id=4N5X2DROGKKWVKRZ45AF2EQTOKIAJCIY7EAS7T7HLL5ONC4WHKXA, seg_pos={0 0}, upload=275679]: Done uploading piece. err: <nil>
Dec 18 17:52:47.013564090 (314h23m46.884647135s) [noise=true , address=145.79.227.49:4337 , piece_id=YUBQJZBTELQN4FAEFPZKETB7EPLMYIPQD74EYIW3IV3Z5ZEV43EA, seg_pos={0 0}, upload=275679]: Done uploading piece. err: <nil>
Dec 18 17:52:47.015131835 (314h23m46.88621488s) [noise=true , address=145.79.227.45:4230 , piece_id=ZXCAIAA6F3VK7YEEM6KJNEQBK6OW76VAZUZJDREMDXU26ZBTEVAQ, seg_pos={0 0}, upload=275679]: Done uploading piece. err: <nil>
Dec 18 17:52:47.027554080 (314h23m46.898637115s) [noise=true , address=145.79.227.47:4267 , piece_id=TTDS56UKSLXX4PQUUSBRETQE2ZXRCA2GAJQBGBUIRYKK3LSS2CBA, seg_pos={0 0}, upload=275679]: Done uploading piece. err: <nil>
Dec 18 17:52:47.063316270 (314h23m46.934399315s) [noise=true , address=145.79.227.46:4602 , piece_id=BRDASXODM3KXOXXZHOHWZH57ROCHAMTYPX3RHK4HB3P3TB2R6O5A, seg_pos={0 0}, upload=275679]: Done uploading piece. err: <nil>
Dec 18 17:52:47.158803993 (314h23m47.029887039s) [noise=true , address=77.90.31.13:4627 , piece_id=VMT2UW5V7I35CQ4T5MQAPG2Q5TQT7ZWZ26R3H4UQZNWS3QWOXWAA, seg_pos={0 0}, upload=275679]: Done uploading piece. err: <nil>
Dec 18 17:52:47.223458552 (314h23m47.094541607s) [noise=true , address=77.90.31.10:4518 , piece_id=LALBF2ROEGO5MRGBZW7HSOIL53C3AY2HEIBRLK6HNUCDAOCDWFPA, seg_pos={0 0}, upload=275679]: Done uploading piece. err: <nil>
Dec 18 17:52:47.244689391 (314h23m47.115772426s) [noise=true , address=45.91.116.12:4226 , piece_id=7SG6A2PFALY2J3EXUGSAENBASROUNU6CDADDL63NW5HMBRSOCWLA, seg_pos={0 0}, upload=275679]: Done uploading piece. err: <nil>
Dec 18 17:52:47.258941461 (314h23m47.130024496s) [noise=true , address=45.91.116.14:4191 , piece_id=PHZXDLNEZIFFCCPIAX25DOFFUTRNQ2T6QXYHNMC3BS2FIE65CETA, seg_pos={0 0}, upload=275679]: Done uploading piece. err: <nil>
Dec 18 17:52:47.282675285 (314h23m47.15375833s) [noise=true , address=77.90.31.11:4553 , piece_id=5JR4WR7UM45LIMLMYPTO5N3UEFBS7JBYWCDKUDMPVMNVZJWAMRCQ, seg_pos={0 0}, upload=275679]: Done uploading piece. err: <nil>
Dec 18 17:53:13.004530920 (314h24m12.875613985s) [noise=true , address=145.79.227.39:4014 , piece_id=SBOPD3T6BA3TYAUDATBGR2TFXRJR4UBSEUFBRFEOPR3IORUDXX5A, seg_pos={0 0}, upload=275679]: Done uploading piece. err: <nil>
Dec 18 17:53:14.042758338 (314h24m13.913841374s) [noise=true , address=145.79.227.9:4366 , piece_id=LC3BXTGNY6HZVKVRCWXCHXI2HCER2XYXA5EY3B5EDZOROBSXTOUA, seg_pos={0 0}, upload=275679]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF
Dec 18 17:53:14.042938541 (314h24m13.914021577s) [noise=true , address=145.79.227.32:4835 , piece_id=BDCCTSNIO5XWLP5G4VOXKCIDWJ3LFKEQWEGM37EYGQYSN4TNTAEQ, seg_pos={0 0}, upload=275679]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: expected piece hash; context deadline exceeded; EOF
Dec 18 17:53:14.042970603 (314h24m13.914053648s) [seg_pos={0 0}, upload=275679]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 18 17:53:14.289403563 (314h24m14.160486608s) [upload=275679]: Done uploading segments...
Dec 18 17:53:14.289463628 (314h24m14.160546663s) [upload=275679]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 30.248643373s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
6.271749484s trace:
https://jaeger.dev.storj.io/trace/66a9c91bc4a6ccf0
FAIL (2025-12-18 17:52:45 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:41370->45.38.24.2:443: read: connection reset by peer
Duration: 6.271780894s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
11.157904651s trace:
https://jaeger.dev.storj.io/trace/542c87d49235b036
FAIL (2025-12-18 17:52:10 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Dec 18 17:51:29.160346640 (314h22m29.031429685s) [upload=275659]: Starting upload
Dec 18 17:51:29.160434988 (314h22m29.031518033s) [upload=275659]: Done starting upload
Dec 18 17:51:29.160533456 (314h22m29.031616491s) [upload=275659]: Uploading segments...
Dec 18 17:51:29.160655689 (314h22m29.031738724s) [upload=275659]: Got next segment. Inline: false
Dec 18 17:51:29.160705403 (314h22m29.031788448s) [upload=275659]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 18 17:51:29.160712416 (314h22m29.031795461s) [upload=275659]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 18 17:51:29.408277098 (314h22m29.279360133s) [seg_pos={0 0}, upload=275659]: Begin upload segment...
Dec 18 17:51:29.408320331 (314h22m29.279403376s) [seg_pos={0 0}, upload=275659]: Done begin upload segment.
Dec 18 17:51:29.408346160 (314h22m29.279429205s) [noise=true , address=145.79.227.29:4941 , piece_id=OB5APOYWO4YOOGFYHYFPN2LLZ67KJIQ6MWMPL3D2G4DBSUXDJR2Q, seg_pos={0 0}, upload=275659]: Uploading piece...
Dec 18 17:51:29.408396085 (314h22m29.27947913s) [noise=true , address=145.79.227.35:4482 , piece_id=6FBX5THUQIUUPU4DEGTE3BNGAIYPGJEDMWL5LLN7UKG2HMVRWDPA, seg_pos={0 0}, upload=275659]: Uploading piece...
Dec 18 17:51:29.408909313 (314h22m29.279992368s) [noise=true , address=145.79.227.31:4799 , piece_id=NW7PWMP475FEDCYBHJHSZRRTJPX5CVCKQD3B4LIWVKHW7LT5YXFA, seg_pos={0 0}, upload=275659]: Uploading piece...
Dec 18 17:51:29.409343249 (314h22m29.280426284s) [noise=true , address=77.90.31.16:4736 , piece_id=WUTZYLFG3SDPFCBLEPCMQGWZT3P5ZL5QXV4AH42WTTKZ7TMQTYWA, seg_pos={0 0}, upload=275659]: Uploading piece...
Dec 18 17:51:29.409731658 (314h22m29.280814703s) [noise=true , address=145.79.227.43:4158 , piece_id=JEEHOYDYIM24NIGMSDMKHDVFOVZ26EA7NQN36WI2G6CV4TYN3QFQ, seg_pos={0 0}, upload=275659]: Uploading piece...
Dec 18 17:51:29.410190232 (314h22m29.281273267s) [noise=true , address=45.91.116.12:4225 , piece_id=MHBRUDZU2XDTJHXQWBTK6D5GXUPMSZOLUY3XLZL4HU7OYLXGPFEQ, seg_pos={0 0}, upload=275659]: Uploading piece...
Dec 18 17:51:29.410672620 (314h22m29.281755655s) [noise=true , address=77.90.31.19:4844 , piece_id=TUIKVIPAPHLOYBTW2X3ZAV3MQBLJFGJMSRI5UCLQLXWYLJ2KBLOA, seg_pos={0 0}, upload=275659]: Uploading piece...
Dec 18 17:51:29.411095015 (314h22m29.28217805s) [noise=true , address=145.79.227.28:4725 , piece_id=G4FKWZ2WA2FPTZRFQGLQCAUHE4S3YLKRVNU6ZAAEMEEHB7WIGP5Q, seg_pos={0 0}, upload=275659]: Uploading piece...
Dec 18 17:51:29.411474818 (314h22m29.282557863s) [noise=true , address=145.79.227.44:4193 , piece_id=H63FDE3TBZRZH2RTB3ICE4KR3KVJUV5VSD3FXYPSQFL6Z5KLFF7A, seg_pos={0 0}, upload=275659]: Uploading piece...
Dec 18 17:51:29.413122305 (314h22m29.28420535s) [noise=true , address=145.79.227.9:4366 , piece_id=QB5RP5V4JHCHQ35VJ5XM6I2AF6D3FSJCGGIHMFTWKQ2OSA6BTBXQ, seg_pos={0 0}, upload=275659]: Uploading piece...
Dec 18 17:51:29.414758731 (314h22m29.285841776s) [noise=true , address=145.79.227.19:4079 , piece_id=BL7Y4X4WEIZELLKWFZOPG2R5BTHF4DZBUOSH2Y4JIDYCV3AHDF5Q, seg_pos={0 0}, upload=275659]: Uploading piece...
Dec 18 17:51:29.416324342 (314h22m29.287407377s) [noise=true , address=145.79.227.38:4978 , piece_id=HRH56KIJFUNULBA22IUMERCV55GXYS2L7D33NXDWALSRSQFPS36Q, seg_pos={0 0}, upload=275659]: Uploading piece...
Dec 18 17:51:29.418015904 (314h22m29.289098959s) [noise=true , address=145.79.227.50:4373 , piece_id=535S2AUG472UTMGZVZH6JSIVR463KKHYM4F5YMTCZTT7STCCUCGA, seg_pos={0 0}, upload=275659]: Uploading piece...
Dec 18 17:51:29.419582727 (314h22m29.290665772s) [noise=true , address=145.79.227.52:4445 , piece_id=6536JA7244RODOVIXE5HEL773CP6NOKND3MAMDLUPOUWUIQHFB7Q, seg_pos={0 0}, upload=275659]: Uploading piece...
Dec 18 17:51:29.422560899 (314h22m29.293643944s) [noise=true , address=145.79.227.49:4339 , piece_id=S3O2OBKWCTJB6S4QBZT3GCQHTXKU3UUTV2LVHDD35YV7MHIPYSTQ, seg_pos={0 0}, upload=275659]: Uploading piece...
Dec 18 17:51:29.426335437 (314h22m29.297418482s) [noise=true , address=145.79.227.47:4267 , piece_id=EJAMIMLRRTOCHPG7MFWDHD3OERJUPDQVISP4UTJOJFIHHU2S77NA, seg_pos={0 0}, upload=275659]: Uploading piece...
Dec 18 17:51:30.417534100 (314h22m30.288617145s) [noise=true , address=145.79.227.31:4799 , piece_id=NW7PWMP475FEDCYBHJHSZRRTJPX5CVCKQD3B4LIWVKHW7LT5YXFA, seg_pos={0 0}, upload=275659]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: storage node overloaded, request limit: 2000
Dec 18 17:51:30.749557654 (314h22m30.620640699s) [noise=true , address=145.79.227.38:4978 , piece_id=HRH56KIJFUNULBA22IUMERCV55GXYS2L7D33NXDWALSRSQFPS36Q, seg_pos={0 0}, upload=275659]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: storage node overloaded, request limit: 2000
Dec 18 17:51:31.009906529 (314h22m30.880989564s) [upload=275659]: Next returned nil segment
Dec 18 17:51:31.009943740 (314h22m30.881026775s) [upload=275659]: Waiting for error group managing segments...
Dec 18 17:51:31.408854866 (314h22m31.279937911s) [noise=true , address=145.79.227.47:4267 , piece_id=EJAMIMLRRTOCHPG7MFWDHD3OERJUPDQVISP4UTJOJFIHHU2S77NA, seg_pos={0 0}, upload=275659]: Done uploading piece. err: <nil>
Dec 18 17:51:31.431991915 (314h22m31.303074961s) [noise=true , address=145.79.227.35:4482 , piece_id=6FBX5THUQIUUPU4DEGTE3BNGAIYPGJEDMWL5LLN7UKG2HMVRWDPA, seg_pos={0 0}, upload=275659]: Done uploading piece. err: <nil>
Dec 18 17:51:31.433564681 (314h22m31.304647716s) [noise=true , address=145.79.227.52:4445 , piece_id=6536JA7244RODOVIXE5HEL773CP6NOKND3MAMDLUPOUWUIQHFB7Q, seg_pos={0 0}, upload=275659]: Done uploading piece. err: <nil>
Dec 18 17:51:31.434252902 (314h22m31.305335937s) [noise=true , address=145.79.227.49:4339 , piece_id=S3O2OBKWCTJB6S4QBZT3GCQHTXKU3UUTV2LVHDD35YV7MHIPYSTQ, seg_pos={0 0}, upload=275659]: Done uploading piece. err: <nil>
Dec 18 17:51:31.435794707 (314h22m31.306877752s) [noise=true , address=145.79.227.19:4079 , piece_id=BL7Y4X4WEIZELLKWFZOPG2R5BTHF4DZBUOSH2Y4JIDYCV3AHDF5Q, seg_pos={0 0}, upload=275659]: Done uploading piece. err: <nil>
Dec 18 17:51:31.461664671 (314h22m31.332747706s) [noise=true , address=145.79.227.28:4725 , piece_id=G4FKWZ2WA2FPTZRFQGLQCAUHE4S3YLKRVNU6ZAAEMEEHB7WIGP5Q, seg_pos={0 0}, upload=275659]: Done uploading piece. err: <nil>
Dec 18 17:51:31.505662054 (314h22m31.376745099s) [noise=true , address=145.79.227.50:4373 , piece_id=535S2AUG472UTMGZVZH6JSIVR463KKHYM4F5YMTCZTT7STCCUCGA, seg_pos={0 0}, upload=275659]: Done uploading piece. err: <nil>
Dec 18 17:51:31.585254921 (314h22m31.456337956s) [noise=true , address=77.90.31.16:4736 , piece_id=WUTZYLFG3SDPFCBLEPCMQGWZT3P5ZL5QXV4AH42WTTKZ7TMQTYWA, seg_pos={0 0}, upload=275659]: Done uploading piece. err: <nil>
Dec 18 17:51:31.671072451 (314h22m31.542155486s) [noise=true , address=77.90.31.19:4844 , piece_id=TUIKVIPAPHLOYBTW2X3ZAV3MQBLJFGJMSRI5UCLQLXWYLJ2KBLOA, seg_pos={0 0}, upload=275659]: Done uploading piece. err: <nil>
Dec 18 17:51:31.853611024 (314h22m31.724694069s) [noise=true , address=45.91.116.12:4225 , piece_id=MHBRUDZU2XDTJHXQWBTK6D5GXUPMSZOLUY3XLZL4HU7OYLXGPFEQ, seg_pos={0 0}, upload=275659]: Done uploading piece. err: <nil>
Dec 18 17:51:32.113732186 (314h22m31.984815231s) [noise=true , address=145.79.227.29:4941 , piece_id=OB5APOYWO4YOOGFYHYFPN2LLZ67KJIQ6MWMPL3D2G4DBSUXDJR2Q, seg_pos={0 0}, upload=275659]: Done uploading piece. err: <nil>
Dec 18 17:51:58.472630565 (314h22m58.34371361s) [noise=true , address=145.79.227.44:4193 , piece_id=H63FDE3TBZRZH2RTB3ICE4KR3KVJUV5VSD3FXYPSQFL6Z5KLFF7A, seg_pos={0 0}, upload=275659]: Done uploading piece. err: ecclient: upload failed (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, address:145.79.227.44:4193): protocol: expected piece hash; context deadline exceeded; EOF
Dec 18 17:51:58.472821189 (314h22m58.343904224s) [noise=true , address=145.79.227.9:4366 , piece_id=QB5RP5V4JHCHQ35VJ5XM6I2AF6D3FSJCGGIHMFTWKQ2OSA6BTBXQ, seg_pos={0 0}, upload=275659]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF
Dec 18 17:51:58.473011471 (314h22m58.344094536s) [noise=true , address=145.79.227.43:4158 , piece_id=JEEHOYDYIM24NIGMSDMKHDVFOVZ26EA7NQN36WI2G6CV4TYN3QFQ, seg_pos={0 0}, upload=275659]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: expected piece hash; context deadline exceeded; EOF
Dec 18 17:51:58.473063671 (314h22m58.344146706s) [seg_pos={0 0}, upload=275659]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Dec 18 17:51:58.710525563 (314h22m58.581608598s) [upload=275659]: Done uploading segments...
Dec 18 17:51:58.710566231 (314h22m58.581649276s) [upload=275659]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 11.157922785s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.239568248s trace:
https://jaeger.dev.storj.io/trace/542c87d49235b036
FAIL (2025-12-18 17:51:58 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Dec 18 17:51:29.160346640 (314h22m29.031429685s) [upload=275659]: Starting upload
Dec 18 17:51:29.160434988 (314h22m29.031518033s) [upload=275659]: Done starting upload
Dec 18 17:51:29.160533456 (314h22m29.031616491s) [upload=275659]: Uploading segments...
Dec 18 17:51:29.160655689 (314h22m29.031738724s) [upload=275659]: Got next segment. Inline: false
Dec 18 17:51:29.160705403 (314h22m29.031788448s) [upload=275659]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 18 17:51:29.160712416 (314h22m29.031795461s) [upload=275659]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 18 17:51:29.408277098 (314h22m29.279360133s) [seg_pos={0 0}, upload=275659]: Begin upload segment...
Dec 18 17:51:29.408320331 (314h22m29.279403376s) [seg_pos={0 0}, upload=275659]: Done begin upload segment.
Dec 18 17:51:29.408346160 (314h22m29.279429205s) [noise=true , address=145.79.227.29:4941 , piece_id=OB5APOYWO4YOOGFYHYFPN2LLZ67KJIQ6MWMPL3D2G4DBSUXDJR2Q, seg_pos={0 0}, upload=275659]: Uploading piece...
Dec 18 17:51:29.408396085 (314h22m29.27947913s) [noise=true , address=145.79.227.35:4482 , piece_id=6FBX5THUQIUUPU4DEGTE3BNGAIYPGJEDMWL5LLN7UKG2HMVRWDPA, seg_pos={0 0}, upload=275659]: Uploading piece...
Dec 18 17:51:29.408909313 (314h22m29.279992368s) [noise=true , address=145.79.227.31:4799 , piece_id=NW7PWMP475FEDCYBHJHSZRRTJPX5CVCKQD3B4LIWVKHW7LT5YXFA, seg_pos={0 0}, upload=275659]: Uploading piece...
Dec 18 17:51:29.409343249 (314h22m29.280426284s) [noise=true , address=77.90.31.16:4736 , piece_id=WUTZYLFG3SDPFCBLEPCMQGWZT3P5ZL5QXV4AH42WTTKZ7TMQTYWA, seg_pos={0 0}, upload=275659]: Uploading piece...
Dec 18 17:51:29.409731658 (314h22m29.280814703s) [noise=true , address=145.79.227.43:4158 , piece_id=JEEHOYDYIM24NIGMSDMKHDVFOVZ26EA7NQN36WI2G6CV4TYN3QFQ, seg_pos={0 0}, upload=275659]: Uploading piece...
Dec 18 17:51:29.410190232 (314h22m29.281273267s) [noise=true , address=45.91.116.12:4225 , piece_id=MHBRUDZU2XDTJHXQWBTK6D5GXUPMSZOLUY3XLZL4HU7OYLXGPFEQ, seg_pos={0 0}, upload=275659]: Uploading piece...
Dec 18 17:51:29.410672620 (314h22m29.281755655s) [noise=true , address=77.90.31.19:4844 , piece_id=TUIKVIPAPHLOYBTW2X3ZAV3MQBLJFGJMSRI5UCLQLXWYLJ2KBLOA, seg_pos={0 0}, upload=275659]: Uploading piece...
Dec 18 17:51:29.411095015 (314h22m29.28217805s) [noise=true , address=145.79.227.28:4725 , piece_id=G4FKWZ2WA2FPTZRFQGLQCAUHE4S3YLKRVNU6ZAAEMEEHB7WIGP5Q, seg_pos={0 0}, upload=275659]: Uploading piece...
Dec 18 17:51:29.411474818 (314h22m29.282557863s) [noise=true , address=145.79.227.44:4193 , piece_id=H63FDE3TBZRZH2RTB3ICE4KR3KVJUV5VSD3FXYPSQFL6Z5KLFF7A, seg_pos={0 0}, upload=275659]: Uploading piece...
Dec 18 17:51:29.413122305 (314h22m29.28420535s) [noise=true , address=145.79.227.9:4366 , piece_id=QB5RP5V4JHCHQ35VJ5XM6I2AF6D3FSJCGGIHMFTWKQ2OSA6BTBXQ, seg_pos={0 0}, upload=275659]: Uploading piece...
Dec 18 17:51:29.414758731 (314h22m29.285841776s) [noise=true , address=145.79.227.19:4079 , piece_id=BL7Y4X4WEIZELLKWFZOPG2R5BTHF4DZBUOSH2Y4JIDYCV3AHDF5Q, seg_pos={0 0}, upload=275659]: Uploading piece...
Dec 18 17:51:29.416324342 (314h22m29.287407377s) [noise=true , address=145.79.227.38:4978 , piece_id=HRH56KIJFUNULBA22IUMERCV55GXYS2L7D33NXDWALSRSQFPS36Q, seg_pos={0 0}, upload=275659]: Uploading piece...
Dec 18 17:51:29.418015904 (314h22m29.289098959s) [noise=true , address=145.79.227.50:4373 , piece_id=535S2AUG472UTMGZVZH6JSIVR463KKHYM4F5YMTCZTT7STCCUCGA, seg_pos={0 0}, upload=275659]: Uploading piece...
Dec 18 17:51:29.419582727 (314h22m29.290665772s) [noise=true , address=145.79.227.52:4445 , piece_id=6536JA7244RODOVIXE5HEL773CP6NOKND3MAMDLUPOUWUIQHFB7Q, seg_pos={0 0}, upload=275659]: Uploading piece...
Dec 18 17:51:29.422560899 (314h22m29.293643944s) [noise=true , address=145.79.227.49:4339 , piece_id=S3O2OBKWCTJB6S4QBZT3GCQHTXKU3UUTV2LVHDD35YV7MHIPYSTQ, seg_pos={0 0}, upload=275659]: Uploading piece...
Dec 18 17:51:29.426335437 (314h22m29.297418482s) [noise=true , address=145.79.227.47:4267 , piece_id=EJAMIMLRRTOCHPG7MFWDHD3OERJUPDQVISP4UTJOJFIHHU2S77NA, seg_pos={0 0}, upload=275659]: Uploading piece...
Dec 18 17:51:30.417534100 (314h22m30.288617145s) [noise=true , address=145.79.227.31:4799 , piece_id=NW7PWMP475FEDCYBHJHSZRRTJPX5CVCKQD3B4LIWVKHW7LT5YXFA, seg_pos={0 0}, upload=275659]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: storage node overloaded, request limit: 2000
Dec 18 17:51:30.749557654 (314h22m30.620640699s) [noise=true , address=145.79.227.38:4978 , piece_id=HRH56KIJFUNULBA22IUMERCV55GXYS2L7D33NXDWALSRSQFPS36Q, seg_pos={0 0}, upload=275659]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: storage node overloaded, request limit: 2000
Dec 18 17:51:31.009906529 (314h22m30.880989564s) [upload=275659]: Next returned nil segment
Dec 18 17:51:31.009943740 (314h22m30.881026775s) [upload=275659]: Waiting for error group managing segments...
Dec 18 17:51:31.408854866 (314h22m31.279937911s) [noise=true , address=145.79.227.47:4267 , piece_id=EJAMIMLRRTOCHPG7MFWDHD3OERJUPDQVISP4UTJOJFIHHU2S77NA, seg_pos={0 0}, upload=275659]: Done uploading piece. err: <nil>
Dec 18 17:51:31.431991915 (314h22m31.303074961s) [noise=true , address=145.79.227.35:4482 , piece_id=6FBX5THUQIUUPU4DEGTE3BNGAIYPGJEDMWL5LLN7UKG2HMVRWDPA, seg_pos={0 0}, upload=275659]: Done uploading piece. err: <nil>
Dec 18 17:51:31.433564681 (314h22m31.304647716s) [noise=true , address=145.79.227.52:4445 , piece_id=6536JA7244RODOVIXE5HEL773CP6NOKND3MAMDLUPOUWUIQHFB7Q, seg_pos={0 0}, upload=275659]: Done uploading piece. err: <nil>
Dec 18 17:51:31.434252902 (314h22m31.305335937s) [noise=true , address=145.79.227.49:4339 , piece_id=S3O2OBKWCTJB6S4QBZT3GCQHTXKU3UUTV2LVHDD35YV7MHIPYSTQ, seg_pos={0 0}, upload=275659]: Done uploading piece. err: <nil>
Dec 18 17:51:31.435794707 (314h22m31.306877752s) [noise=true , address=145.79.227.19:4079 , piece_id=BL7Y4X4WEIZELLKWFZOPG2R5BTHF4DZBUOSH2Y4JIDYCV3AHDF5Q, seg_pos={0 0}, upload=275659]: Done uploading piece. err: <nil>
Dec 18 17:51:31.461664671 (314h22m31.332747706s) [noise=true , address=145.79.227.28:4725 , piece_id=G4FKWZ2WA2FPTZRFQGLQCAUHE4S3YLKRVNU6ZAAEMEEHB7WIGP5Q, seg_pos={0 0}, upload=275659]: Done uploading piece. err: <nil>
Dec 18 17:51:31.505662054 (314h22m31.376745099s) [noise=true , address=145.79.227.50:4373 , piece_id=535S2AUG472UTMGZVZH6JSIVR463KKHYM4F5YMTCZTT7STCCUCGA, seg_pos={0 0}, upload=275659]: Done uploading piece. err: <nil>
Dec 18 17:51:31.585254921 (314h22m31.456337956s) [noise=true , address=77.90.31.16:4736 , piece_id=WUTZYLFG3SDPFCBLEPCMQGWZT3P5ZL5QXV4AH42WTTKZ7TMQTYWA, seg_pos={0 0}, upload=275659]: Done uploading piece. err: <nil>
Dec 18 17:51:31.671072451 (314h22m31.542155486s) [noise=true , address=77.90.31.19:4844 , piece_id=TUIKVIPAPHLOYBTW2X3ZAV3MQBLJFGJMSRI5UCLQLXWYLJ2KBLOA, seg_pos={0 0}, upload=275659]: Done uploading piece. err: <nil>
Dec 18 17:51:31.853611024 (314h22m31.724694069s) [noise=true , address=45.91.116.12:4225 , piece_id=MHBRUDZU2XDTJHXQWBTK6D5GXUPMSZOLUY3XLZL4HU7OYLXGPFEQ, seg_pos={0 0}, upload=275659]: Done uploading piece. err: <nil>
Dec 18 17:51:32.113732186 (314h22m31.984815231s) [noise=true , address=145.79.227.29:4941 , piece_id=OB5APOYWO4YOOGFYHYFPN2LLZ67KJIQ6MWMPL3D2G4DBSUXDJR2Q, seg_pos={0 0}, upload=275659]: Done uploading piece. err: <nil>
Dec 18 17:51:58.472630565 (314h22m58.34371361s) [noise=true , address=145.79.227.44:4193 , piece_id=H63FDE3TBZRZH2RTB3ICE4KR3KVJUV5VSD3FXYPSQFL6Z5KLFF7A, seg_pos={0 0}, upload=275659]: Done uploading piece. err: ecclient: upload failed (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, address:145.79.227.44:4193): protocol: expected piece hash; context deadline exceeded; EOF
Dec 18 17:51:58.472821189 (314h22m58.343904224s) [noise=true , address=145.79.227.9:4366 , piece_id=QB5RP5V4JHCHQ35VJ5XM6I2AF6D3FSJCGGIHMFTWKQ2OSA6BTBXQ, seg_pos={0 0}, upload=275659]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF
Dec 18 17:51:58.473011471 (314h22m58.344094536s) [noise=true , address=145.79.227.43:4158 , piece_id=JEEHOYDYIM24NIGMSDMKHDVFOVZ26EA7NQN36WI2G6CV4TYN3QFQ, seg_pos={0 0}, upload=275659]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: expected piece hash; context deadline exceeded; EOF
Dec 18 17:51:58.473063671 (314h22m58.344146706s) [seg_pos={0 0}, upload=275659]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Dec 18 17:51:58.710525563 (314h22m58.581608598s) [upload=275659]: Done uploading segments...
Dec 18 17:51:58.710566231 (314h22m58.581649276s) [upload=275659]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 30.239591171s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
5.961193934s trace:
https://jaeger.dev.storj.io/trace/66a9c91bc4a6ccf0
FAIL (2025-12-18 17:51:45 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:41370->45.38.24.2:443: read: connection reset by peer
Duration: 5.961221357s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
6.43719317s trace:
https://jaeger.dev.storj.io/trace/66a9c91bc4a6ccf0
FAIL (2025-12-18 17:50:41 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:41370->45.38.24.2:443: read: connection reset by peer
Duration: 6.437221302s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
6.320731191s trace:
https://jaeger.dev.storj.io/trace/66a9c91bc4a6ccf0
FAIL (2025-12-18 17:50:07 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:41370->45.38.24.2:443: read: connection reset by peer
Duration: 6.320755627s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
6.266304096s trace:
https://jaeger.dev.storj.io/trace/66a9c91bc4a6ccf0
FAIL (2025-12-18 17:49:07 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:41370->45.38.24.2:443: read: connection reset by peer
Duration: 6.266333022s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
6.521359552s trace:
https://jaeger.dev.storj.io/trace/66a9c91bc4a6ccf0
FAIL (2025-12-18 17:48:07 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:41370->45.38.24.2:443: read: connection reset by peer
Duration: 6.521378178s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
6.031149934s trace:
https://jaeger.dev.storj.io/trace/66a9c91bc4a6ccf0
FAIL (2025-12-18 17:47:31 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:41370->45.38.24.2:443: read: connection reset by peer
Duration: 6.031186014s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
6.115547367s trace:
https://jaeger.dev.storj.io/trace/66a9c91bc4a6ccf0
FAIL (2025-12-18 17:46:56 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:41370->45.38.24.2:443: read: connection reset by peer
Duration: 6.115563958s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
510.790134ms trace:
https://jaeger.dev.storj.io/trace/66a9c91bc4a6ccf0
FAIL (2025-12-18 17:46:24 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:41370->45.38.24.2:443: read: connection reset by peer
Duration: 510.818488ms
edge
gateway
us-select-1
|
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)antiblip:
1 out of 15 is failed (threshold=6) [X..............] duration:
317.555283ms remote:
"tcp:[2620:cb:2000::1]:443" response headers:
Cf-Ray=9b006960ed100bfb-SIN Date=Thu, 18 Dec 2025 17:34:33 GMT Content-Type=text/html; charset=UTF-8 Content-Length=4316 X-Frame-Options=SAMEORIGIN Expires=Thu, 01 Jan 1970 00:00:01 GMT Alt-Svc=h3=":443"; ma=86400 Referrer-Policy=same-origin Cache-Control=private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Vary=Accept-Encoding Set-Cookie=_cfuvid=2JVf_Sw9oAzc2nvftjzJtNuERbpSRNcl3qk.txEthk0-1766079273220-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Server=cloudflare target:
https://www.storj.io/dcs-satellites trace:
https://jaeger.dev.storj.io/trace/50fdaf5d6d738382
FAIL (2025-12-18 17:45:31 UTC)
orbiter warning: unexpected status code "500 Internal Server Error"
"<!DOCTYPE html>\n<!--[if lt IE 7]> <html class=\"no-js ie6 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if IE 7]> <html class=\"no-js ie7 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if IE 8]> <html class=\"no-js ie8 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if gt IE 8]><!--> <html class=\"no-js\" lang=\"en-US\"> <!--<![endif]-->\n<head>\n<title>Worker threw exception | www.storj.io | Cloudflare</title>\n<meta charset=\"UTF-8\" />\n<meta http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\" />\n<meta http-equiv=\"X-UA-Compatible\" content=\"IE=Edge\" />\n<meta name=\"robots\" content=\"noindex, nofollow\" />\n<meta name=\"viewport\" content=\"width=device-width,initial-scale=1\" />\n<link rel=\"stylesheet\" id=\"cf_styles-css\" href=\"/cdn-cgi/styles/cf.errors.css\" />\n<!--[if lt IE 9]><link rel=\"stylesheet\" id='cf_styles-ie-css' href=\"/cdn-cgi/styles/cf.errors.ie.css\" /><![endif]-->\n<style>body{margin:0;padding:0}</style>\n\n\n<!--[if gte IE 10]><!-->\n<script>\n if (!navigator.cookieEnabled) {\n window.addEventListener('DO"
Duration: 317.64318ms
core
website
|
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)antiblip:
1 out of 15 is failed (threshold=6) [.X.............] duration:
308.692213ms remote:
"tcp:[2620:cb:2000::1]:443" response headers:
Cf-Ray=9b006960ed100bfb-SIN Date=Thu, 18 Dec 2025 17:34:33 GMT Content-Type=text/html; charset=UTF-8 Content-Length=4316 X-Frame-Options=SAMEORIGIN Expires=Thu, 01 Jan 1970 00:00:01 GMT Alt-Svc=h3=":443"; ma=86400 Referrer-Policy=same-origin Cache-Control=private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Vary=Accept-Encoding Set-Cookie=_cfuvid=2JVf_Sw9oAzc2nvftjzJtNuERbpSRNcl3qk.txEthk0-1766079273220-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Server=cloudflare target:
https://www.storj.io/dcs-satellites trace:
https://jaeger.dev.storj.io/trace/50fdaf5d6d738382
FAIL (2025-12-18 17:44:37 UTC)
orbiter warning: unexpected status code "500 Internal Server Error"
"<!DOCTYPE html>\n<!--[if lt IE 7]> <html class=\"no-js ie6 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if IE 7]> <html class=\"no-js ie7 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if IE 8]> <html class=\"no-js ie8 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if gt IE 8]><!--> <html class=\"no-js\" lang=\"en-US\"> <!--<![endif]-->\n<head>\n<title>Worker threw exception | www.storj.io | Cloudflare</title>\n<meta charset=\"UTF-8\" />\n<meta http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\" />\n<meta http-equiv=\"X-UA-Compatible\" content=\"IE=Edge\" />\n<meta name=\"robots\" content=\"noindex, nofollow\" />\n<meta name=\"viewport\" content=\"width=device-width,initial-scale=1\" />\n<link rel=\"stylesheet\" id=\"cf_styles-css\" href=\"/cdn-cgi/styles/cf.errors.css\" />\n<!--[if lt IE 9]><link rel=\"stylesheet\" id='cf_styles-ie-css' href=\"/cdn-cgi/styles/cf.errors.ie.css\" /><![endif]-->\n<style>body{margin:0;padding:0}</style>\n\n\n<!--[if gte IE 10]><!-->\n<script>\n if (!navigator.cookieEnabled) {\n window.addEventListener('DO"
Duration: 308.720387ms
core
website
|
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)antiblip:
1 out of 15 is failed (threshold=6) [..X............] duration:
306.002493ms remote:
"tcp:[2620:cb:2000::1]:443" response headers:
Cf-Ray=9b006960ed100bfb-SIN Date=Thu, 18 Dec 2025 17:34:33 GMT Content-Type=text/html; charset=UTF-8 Content-Length=4316 X-Frame-Options=SAMEORIGIN Expires=Thu, 01 Jan 1970 00:00:01 GMT Alt-Svc=h3=":443"; ma=86400 Referrer-Policy=same-origin Cache-Control=private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Vary=Accept-Encoding Set-Cookie=_cfuvid=2JVf_Sw9oAzc2nvftjzJtNuERbpSRNcl3qk.txEthk0-1766079273220-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Server=cloudflare target:
https://www.storj.io/dcs-satellites trace:
https://jaeger.dev.storj.io/trace/50fdaf5d6d738382
FAIL (2025-12-18 17:44:05 UTC)
orbiter warning: unexpected status code "500 Internal Server Error"
"<!DOCTYPE html>\n<!--[if lt IE 7]> <html class=\"no-js ie6 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if IE 7]> <html class=\"no-js ie7 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if IE 8]> <html class=\"no-js ie8 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if gt IE 8]><!--> <html class=\"no-js\" lang=\"en-US\"> <!--<![endif]-->\n<head>\n<title>Worker threw exception | www.storj.io | Cloudflare</title>\n<meta charset=\"UTF-8\" />\n<meta http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\" />\n<meta http-equiv=\"X-UA-Compatible\" content=\"IE=Edge\" />\n<meta name=\"robots\" content=\"noindex, nofollow\" />\n<meta name=\"viewport\" content=\"width=device-width,initial-scale=1\" />\n<link rel=\"stylesheet\" id=\"cf_styles-css\" href=\"/cdn-cgi/styles/cf.errors.css\" />\n<!--[if lt IE 9]><link rel=\"stylesheet\" id='cf_styles-ie-css' href=\"/cdn-cgi/styles/cf.errors.ie.css\" /><![endif]-->\n<style>body{margin:0;padding:0}</style>\n\n\n<!--[if gte IE 10]><!-->\n<script>\n if (!navigator.cookieEnabled) {\n window.addEventListener('DO"
Duration: 306.028433ms
core
website
|
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)antiblip:
1 out of 15 is failed (threshold=6) [...X...........] duration:
308.875713ms remote:
"tcp:[2620:cb:2000::1]:443" response headers:
Cf-Ray=9b006960ed100bfb-SIN Date=Thu, 18 Dec 2025 17:34:33 GMT Content-Type=text/html; charset=UTF-8 Content-Length=4316 X-Frame-Options=SAMEORIGIN Expires=Thu, 01 Jan 1970 00:00:01 GMT Alt-Svc=h3=":443"; ma=86400 Referrer-Policy=same-origin Cache-Control=private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Vary=Accept-Encoding Set-Cookie=_cfuvid=2JVf_Sw9oAzc2nvftjzJtNuERbpSRNcl3qk.txEthk0-1766079273220-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Server=cloudflare target:
https://www.storj.io/dcs-satellites trace:
https://jaeger.dev.storj.io/trace/50fdaf5d6d738382
FAIL (2025-12-18 17:43:44 UTC)
orbiter warning: unexpected status code "500 Internal Server Error"
"<!DOCTYPE html>\n<!--[if lt IE 7]> <html class=\"no-js ie6 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if IE 7]> <html class=\"no-js ie7 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if IE 8]> <html class=\"no-js ie8 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if gt IE 8]><!--> <html class=\"no-js\" lang=\"en-US\"> <!--<![endif]-->\n<head>\n<title>Worker threw exception | www.storj.io | Cloudflare</title>\n<meta charset=\"UTF-8\" />\n<meta http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\" />\n<meta http-equiv=\"X-UA-Compatible\" content=\"IE=Edge\" />\n<meta name=\"robots\" content=\"noindex, nofollow\" />\n<meta name=\"viewport\" content=\"width=device-width,initial-scale=1\" />\n<link rel=\"stylesheet\" id=\"cf_styles-css\" href=\"/cdn-cgi/styles/cf.errors.css\" />\n<!--[if lt IE 9]><link rel=\"stylesheet\" id='cf_styles-ie-css' href=\"/cdn-cgi/styles/cf.errors.ie.css\" /><![endif]-->\n<style>body{margin:0;padding:0}</style>\n\n\n<!--[if gte IE 10]><!-->\n<script>\n if (!navigator.cookieEnabled) {\n window.addEventListener('DO"
Duration: 308.934285ms
core
website
|
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)antiblip:
1 out of 15 is failed (threshold=6) [....X..........] duration:
300.844269ms remote:
"tcp:[2620:cb:2000::1]:443" response headers:
Cf-Ray=9b006960ed100bfb-SIN Date=Thu, 18 Dec 2025 17:34:33 GMT Content-Type=text/html; charset=UTF-8 Content-Length=4316 X-Frame-Options=SAMEORIGIN Expires=Thu, 01 Jan 1970 00:00:01 GMT Alt-Svc=h3=":443"; ma=86400 Referrer-Policy=same-origin Cache-Control=private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Vary=Accept-Encoding Set-Cookie=_cfuvid=2JVf_Sw9oAzc2nvftjzJtNuERbpSRNcl3qk.txEthk0-1766079273220-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Server=cloudflare target:
https://www.storj.io/dcs-satellites trace:
https://jaeger.dev.storj.io/trace/50fdaf5d6d738382
FAIL (2025-12-18 17:43:07 UTC)
orbiter warning: unexpected status code "500 Internal Server Error"
"<!DOCTYPE html>\n<!--[if lt IE 7]> <html class=\"no-js ie6 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if IE 7]> <html class=\"no-js ie7 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if IE 8]> <html class=\"no-js ie8 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if gt IE 8]><!--> <html class=\"no-js\" lang=\"en-US\"> <!--<![endif]-->\n<head>\n<title>Worker threw exception | www.storj.io | Cloudflare</title>\n<meta charset=\"UTF-8\" />\n<meta http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\" />\n<meta http-equiv=\"X-UA-Compatible\" content=\"IE=Edge\" />\n<meta name=\"robots\" content=\"noindex, nofollow\" />\n<meta name=\"viewport\" content=\"width=device-width,initial-scale=1\" />\n<link rel=\"stylesheet\" id=\"cf_styles-css\" href=\"/cdn-cgi/styles/cf.errors.css\" />\n<!--[if lt IE 9]><link rel=\"stylesheet\" id='cf_styles-ie-css' href=\"/cdn-cgi/styles/cf.errors.ie.css\" /><![endif]-->\n<style>body{margin:0;padding:0}</style>\n\n\n<!--[if gte IE 10]><!-->\n<script>\n if (!navigator.cookieEnabled) {\n window.addEventListener('DO"
Duration: 300.87582ms
core
website
|
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)antiblip:
1 out of 15 is failed (threshold=6) [.....X.........] duration:
312.369867ms remote:
"tcp:[2620:cb:2000::1]:443" response headers:
Cf-Ray=9b006960ed100bfb-SIN Date=Thu, 18 Dec 2025 17:34:33 GMT Content-Type=text/html; charset=UTF-8 Content-Length=4316 X-Frame-Options=SAMEORIGIN Expires=Thu, 01 Jan 1970 00:00:01 GMT Alt-Svc=h3=":443"; ma=86400 Referrer-Policy=same-origin Cache-Control=private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Vary=Accept-Encoding Set-Cookie=_cfuvid=2JVf_Sw9oAzc2nvftjzJtNuERbpSRNcl3qk.txEthk0-1766079273220-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Server=cloudflare target:
https://www.storj.io/dcs-satellites trace:
https://jaeger.dev.storj.io/trace/50fdaf5d6d738382
FAIL (2025-12-18 17:42:23 UTC)
orbiter warning: unexpected status code "500 Internal Server Error"
"<!DOCTYPE html>\n<!--[if lt IE 7]> <html class=\"no-js ie6 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if IE 7]> <html class=\"no-js ie7 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if IE 8]> <html class=\"no-js ie8 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if gt IE 8]><!--> <html class=\"no-js\" lang=\"en-US\"> <!--<![endif]-->\n<head>\n<title>Worker threw exception | www.storj.io | Cloudflare</title>\n<meta charset=\"UTF-8\" />\n<meta http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\" />\n<meta http-equiv=\"X-UA-Compatible\" content=\"IE=Edge\" />\n<meta name=\"robots\" content=\"noindex, nofollow\" />\n<meta name=\"viewport\" content=\"width=device-width,initial-scale=1\" />\n<link rel=\"stylesheet\" id=\"cf_styles-css\" href=\"/cdn-cgi/styles/cf.errors.css\" />\n<!--[if lt IE 9]><link rel=\"stylesheet\" id='cf_styles-ie-css' href=\"/cdn-cgi/styles/cf.errors.ie.css\" /><![endif]-->\n<style>body{margin:0;padding:0}</style>\n\n\n<!--[if gte IE 10]><!-->\n<script>\n if (!navigator.cookieEnabled) {\n window.addEventListener('DO"
Duration: 312.435322ms
core
website
|
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)antiblip:
1 out of 15 is failed (threshold=6) [......X........] duration:
318.648773ms remote:
"tcp:[2620:cb:2000::1]:443" response headers:
Cf-Ray=9b006960ed100bfb-SIN Date=Thu, 18 Dec 2025 17:34:33 GMT Content-Type=text/html; charset=UTF-8 Content-Length=4316 X-Frame-Options=SAMEORIGIN Expires=Thu, 01 Jan 1970 00:00:01 GMT Alt-Svc=h3=":443"; ma=86400 Referrer-Policy=same-origin Cache-Control=private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Vary=Accept-Encoding Set-Cookie=_cfuvid=2JVf_Sw9oAzc2nvftjzJtNuERbpSRNcl3qk.txEthk0-1766079273220-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Server=cloudflare target:
https://www.storj.io/dcs-satellites trace:
https://jaeger.dev.storj.io/trace/50fdaf5d6d738382
FAIL (2025-12-18 17:41:43 UTC)
orbiter warning: unexpected status code "500 Internal Server Error"
"<!DOCTYPE html>\n<!--[if lt IE 7]> <html class=\"no-js ie6 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if IE 7]> <html class=\"no-js ie7 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if IE 8]> <html class=\"no-js ie8 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if gt IE 8]><!--> <html class=\"no-js\" lang=\"en-US\"> <!--<![endif]-->\n<head>\n<title>Worker threw exception | www.storj.io | Cloudflare</title>\n<meta charset=\"UTF-8\" />\n<meta http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\" />\n<meta http-equiv=\"X-UA-Compatible\" content=\"IE=Edge\" />\n<meta name=\"robots\" content=\"noindex, nofollow\" />\n<meta name=\"viewport\" content=\"width=device-width,initial-scale=1\" />\n<link rel=\"stylesheet\" id=\"cf_styles-css\" href=\"/cdn-cgi/styles/cf.errors.css\" />\n<!--[if lt IE 9]><link rel=\"stylesheet\" id='cf_styles-ie-css' href=\"/cdn-cgi/styles/cf.errors.ie.css\" /><![endif]-->\n<style>body{margin:0;padding:0}</style>\n\n\n<!--[if gte IE 10]><!-->\n<script>\n if (!navigator.cookieEnabled) {\n window.addEventListener('DO"
Duration: 318.670326ms
core
website
|
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)antiblip:
1 out of 15 is failed (threshold=6) [.......X.......] duration:
314.319018ms remote:
"tcp:[2620:cb:2000::1]:443" response headers:
Cf-Ray=9b006960ed100bfb-SIN Date=Thu, 18 Dec 2025 17:34:33 GMT Content-Type=text/html; charset=UTF-8 Content-Length=4316 X-Frame-Options=SAMEORIGIN Expires=Thu, 01 Jan 1970 00:00:01 GMT Alt-Svc=h3=":443"; ma=86400 Referrer-Policy=same-origin Cache-Control=private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Vary=Accept-Encoding Set-Cookie=_cfuvid=2JVf_Sw9oAzc2nvftjzJtNuERbpSRNcl3qk.txEthk0-1766079273220-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Server=cloudflare target:
https://www.storj.io/dcs-satellites trace:
https://jaeger.dev.storj.io/trace/50fdaf5d6d738382
FAIL (2025-12-18 17:40:34 UTC)
orbiter warning: unexpected status code "500 Internal Server Error"
"<!DOCTYPE html>\n<!--[if lt IE 7]> <html class=\"no-js ie6 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if IE 7]> <html class=\"no-js ie7 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if IE 8]> <html class=\"no-js ie8 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if gt IE 8]><!--> <html class=\"no-js\" lang=\"en-US\"> <!--<![endif]-->\n<head>\n<title>Worker threw exception | www.storj.io | Cloudflare</title>\n<meta charset=\"UTF-8\" />\n<meta http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\" />\n<meta http-equiv=\"X-UA-Compatible\" content=\"IE=Edge\" />\n<meta name=\"robots\" content=\"noindex, nofollow\" />\n<meta name=\"viewport\" content=\"width=device-width,initial-scale=1\" />\n<link rel=\"stylesheet\" id=\"cf_styles-css\" href=\"/cdn-cgi/styles/cf.errors.css\" />\n<!--[if lt IE 9]><link rel=\"stylesheet\" id='cf_styles-ie-css' href=\"/cdn-cgi/styles/cf.errors.ie.css\" /><![endif]-->\n<style>body{margin:0;padding:0}</style>\n\n\n<!--[if gte IE 10]><!-->\n<script>\n if (!navigator.cookieEnabled) {\n window.addEventListener('DO"
Duration: 314.349105ms
core
website
|
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)antiblip:
1 out of 15 is failed (threshold=6) [........X......] duration:
323.852473ms remote:
"tcp:[2620:cb:2000::1]:443" response headers:
Cf-Ray=9b006960ed100bfb-SIN Date=Thu, 18 Dec 2025 17:34:33 GMT Content-Type=text/html; charset=UTF-8 Content-Length=4316 X-Frame-Options=SAMEORIGIN Expires=Thu, 01 Jan 1970 00:00:01 GMT Alt-Svc=h3=":443"; ma=86400 Referrer-Policy=same-origin Cache-Control=private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Vary=Accept-Encoding Set-Cookie=_cfuvid=2JVf_Sw9oAzc2nvftjzJtNuERbpSRNcl3qk.txEthk0-1766079273220-0.0.1.1-604800000; path=/; domain=.www.storj.io; HttpOnly; Secure; SameSite=None Server=cloudflare target:
https://www.storj.io/dcs-satellites trace:
https://jaeger.dev.storj.io/trace/50fdaf5d6d738382
FAIL (2025-12-18 17:39:46 UTC)
orbiter warning: unexpected status code "500 Internal Server Error"
"<!DOCTYPE html>\n<!--[if lt IE 7]> <html class=\"no-js ie6 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if IE 7]> <html class=\"no-js ie7 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if IE 8]> <html class=\"no-js ie8 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if gt IE 8]><!--> <html class=\"no-js\" lang=\"en-US\"> <!--<![endif]-->\n<head>\n<title>Worker threw exception | www.storj.io | Cloudflare</title>\n<meta charset=\"UTF-8\" />\n<meta http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\" />\n<meta http-equiv=\"X-UA-Compatible\" content=\"IE=Edge\" />\n<meta name=\"robots\" content=\"noindex, nofollow\" />\n<meta name=\"viewport\" content=\"width=device-width,initial-scale=1\" />\n<link rel=\"stylesheet\" id=\"cf_styles-css\" href=\"/cdn-cgi/styles/cf.errors.css\" />\n<!--[if lt IE 9]><link rel=\"stylesheet\" id='cf_styles-ie-css' href=\"/cdn-cgi/styles/cf.errors.ie.css\" /><![endif]-->\n<style>body{margin:0;padding:0}</style>\n\n\n<!--[if gte IE 10]><!-->\n<script>\n if (!navigator.cookieEnabled) {\n window.addEventListener('DO"
Duration: 323.877442ms
core
website
|