Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
14 out of 60 is failed (threshold=6) [..............................................XXXXXXXXXXXXXX] duration:
5.003121536s trace:
https://jaeger.dev.storj.io/trace/1dbd683e1ebc1f69
FAIL (2025-11-21 11:11:01 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 5.003146463s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [..................................................X.........] duration:
447.331137ms remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/e2bae25cf885024
FAIL (2025-11-21 11:10:57 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 447.440726ms
edge
linksharing
bgp
eu1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [.................................................X..........] duration:
386.069865ms remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/41cfd75a627799ce
FAIL (2025-11-21 11:10:49 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 386.096877ms
edge
linksharing
bgp
us1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [..................................................X.........] duration:
391.230532ms remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/41cfd75a627799ce
FAIL (2025-11-21 11:10:05 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 391.270438ms
edge
linksharing
bgp
us1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
401.66552ms remote:
"tcp:207.211.208.129:443" target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace:
https://jaeger.dev.storj.io/trace/3e5c193367f9a974
FAIL (2025-11-21 11:10:04 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded
Duration: 401.698916ms
edge
gateway
vhost
bgp
us1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
452.94077ms remote:
"tcp:207.211.208.129:443" target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318 trace:
https://jaeger.dev.storj.io/trace/5285054b3a776da7
FAIL (2025-11-21 11:09:59 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318": context deadline exceeded
Duration: 452.987718ms
edge
gateway
vhost
bgp
eu1
|
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)antiblip:
1 out of 15 is failed (threshold=6) [.....X.........] duration:
860.583543ms remote:
"tcp:[2620:cb:2000::1]:443" target:
https://www.storj.io/dcs-satellites trace:
https://jaeger.dev.storj.io/trace/3d967bc11165dc1c
FAIL (2025-11-21 11:09:47 UTC)
orbiter warning: Get "https://www.storj.io/dcs-satellites": context deadline exceeded
Duration: 860.746083ms
core
website
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
13 out of 60 is failed (threshold=6) [...............................................XXXXXXXXXXXXX] duration:
5.000727366s trace:
https://jaeger.dev.storj.io/trace/1891ae4d8c33ba79
FAIL (2025-11-21 11:09:47 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 5.000755489s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [...................................................X........] duration:
508.799353ms remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/e2bae25cf885024
FAIL (2025-11-21 11:09:46 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 508.859267ms
edge
linksharing
bgp
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [......................................................X...X.] duration:
6.214600144s trace:
https://jaeger.dev.storj.io/trace/3c71c0f34134d3e8
FAIL (2025-11-21 11:09:39 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: MVWVSRLOEPNDQNAJJ2TF5H3LVGNAG2POUVGCEHSNEIOWEJ37E7FA) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: 7L25GK2GXANF42FOGD7XJXJP4CCXLXAMKCRHLVVJMEU3C3AOXIBA) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: F552HWH6XT2YJQABRP35GRN3KBZUTOL5DBK46HRSPXDS7G7PSIGQ) context deadline exceeded
======LOGS======
Nov 21 11:08:49.220727082 (291h4m47.772197543s) [upload=256340]: Starting upload
Nov 21 11:08:49.220828385 (291h4m47.772298846s) [upload=256340]: Done starting upload
Nov 21 11:08:49.221068072 (291h4m47.772538533s) [upload=256340]: Uploading segments...
Nov 21 11:08:49.221175045 (291h4m47.772645506s) [upload=256340]: Got next segment. Inline: false
Nov 21 11:08:49.221239588 (291h4m47.772710049s) [upload=256340]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 21 11:08:49.221247403 (291h4m47.772717864s) [upload=256340]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 21 11:08:49.463941805 (291h4m48.015412266s) [seg_pos={0 0}, upload=256340]: Begin upload segment...
Nov 21 11:08:49.464007140 (291h4m48.015477601s) [seg_pos={0 0}, upload=256340]: Done begin upload segment.
Nov 21 11:08:49.464040994 (291h4m48.015511455s) [noise=true , address=45.91.116.7:4010 , piece_id=F552HWH6XT2YJQABRP35GRN3KBZUTOL5DBK46HRSPXDS7G7PSIGQ, seg_pos={0 0}, upload=256340]: Uploading piece...
Nov 21 11:08:49.464432970 (291h4m48.015903431s) [noise=true , address=145.79.227.21:4475 , piece_id=IMP7XE2IM235OC6CZIDPS7KCQHPVQWK3TU2BPTGANTEN55BPN7XQ, seg_pos={0 0}, upload=256340]: Uploading piece...
Nov 21 11:08:49.465081406 (291h4m48.016551867s) [noise=true , address=77.90.31.15:4699 , piece_id=GW4M5UA3WL5OFBT3IHBPAXUUBCDCPK74WOLSXNRJ3YMALQ3VCNPQ, seg_pos={0 0}, upload=256340]: Uploading piece...
Nov 21 11:08:49.465556651 (291h4m48.017027112s) [noise=true , address=77.90.31.11:4554 , piece_id=I5TZO3QETRCMEUIJZOA45K3KFGYOG573Z6VSZAWMO6FGPYENZC7A, seg_pos={0 0}, upload=256340]: Uploading piece...
Nov 21 11:08:49.466129752 (291h4m48.017600213s) [noise=true , address=145.79.227.28:4725 , piece_id=CM4ARJXFPQ5L73AVRKXC3W2PHL5EAPBAU6FFWLO5FHBHHWZFCMHQ, seg_pos={0 0}, upload=256340]: Uploading piece...
Nov 21 11:08:49.466926569 (291h4m48.01839703s) [noise=true , address=77.90.31.10:4520 , piece_id=ZWSQH4GSREBKPOTJFREGXLKMKK63WY6HUH5XBFGRBMEAG34TH5PA, seg_pos={0 0}, upload=256340]: Uploading piece...
Nov 21 11:08:49.467485784 (291h4m48.018956245s) [noise=true , address=77.90.31.19:4842 , piece_id=MVWVSRLOEPNDQNAJJ2TF5H3LVGNAG2POUVGCEHSNEIOWEJ37E7FA, seg_pos={0 0}, upload=256340]: Uploading piece...
Nov 21 11:08:49.468203270 (291h4m48.019673721s) [noise=true , address=45.91.116.4:4940 , piece_id=MNN2SGDFXIPT5FFUQMLOCSNFZLIREI3B6O2KHOM43GPEWOTETJYQ, seg_pos={0 0}, upload=256340]: Uploading piece...
Nov 21 11:08:49.468858177 (291h4m48.020328638s) [noise=true , address=145.79.227.31:4797 , piece_id=ZYT3EYKH7F2D4P4Q37JRXSBCLHRACWDFMJOXSUA2BRILP7XNH22A, seg_pos={0 0}, upload=256340]: Uploading piece...
Nov 21 11:08:49.470354446 (291h4m48.021824907s) [noise=true , address=145.79.227.24:4583 , piece_id=7W2WYOH4QUDVTNIZER7ASJ7ECZE4ASS6WD5DDZMUEPOOPCL5M7QA, seg_pos={0 0}, upload=256340]: Uploading piece...
Nov 21 11:08:49.472376998 (291h4m48.023847459s) [noise=true , address=145.79.227.17:4971 , piece_id=JBCGNQCBZKRVESI25WPG7UEGQELZLNT6ZDAUBSDI2XE5ITU3K6NQ, seg_pos={0 0}, upload=256340]: Uploading piece...
Nov 21 11:08:49.475441563 (291h4m48.026912025s) [noise=true , address=45.91.116.11:4117 , piece_id=7L25GK2GXANF42FOGD7XJXJP4CCXLXAMKCRHLVVJMEU3C3AOXIBA, seg_pos={0 0}, upload=256340]: Uploading piece...
Nov 21 11:08:49.477522856 (291h4m48.028993317s) [noise=true , address=145.79.227.27:4689 , piece_id=B2ARHC33I7MAVPVUM5BHYQ7VXIB5UVJWCZ4KJCQGYDXA5J3OTTHA, seg_pos={0 0}, upload=256340]: Uploading piece...
Nov 21 11:08:49.488511291 (291h4m48.039981752s) [noise=true , address=145.79.227.28:4727 , piece_id=GJOERP3WIDS2C4VQGEYA4H3OHV64V4GOWYDGPIMW5SLZRMZMUBIQ, seg_pos={0 0}, upload=256340]: Uploading piece...
Nov 21 11:08:49.489403249 (291h4m48.04087372s) [noise=true , address=45.91.116.11:4119 , piece_id=PT3JGFZJK542ZTOETTB5DUF2T5QXBPJD4ETULT4BVQTYBH73CQLA, seg_pos={0 0}, upload=256340]: Uploading piece...
Nov 21 11:08:49.492101117 (291h4m48.043571578s) [noise=true , address=145.79.227.9:4365 , piece_id=HCEAP2KU7CN7RG4V4TC3ONKVJ6ZTNMGI6XRLSDHFW4NOE6D2JJLA, seg_pos={0 0}, upload=256340]: Uploading piece...
Nov 21 11:08:51.131275763 (291h4m49.682746224s) [upload=256340]: Next returned nil segment
Nov 21 11:08:51.131303967 (291h4m49.682774428s) [upload=256340]: Waiting for error group managing segments...
Nov 21 11:08:51.456208272 (291h4m50.007678733s) [noise=true , address=145.79.227.21:4475 , piece_id=IMP7XE2IM235OC6CZIDPS7KCQHPVQWK3TU2BPTGANTEN55BPN7XQ, seg_pos={0 0}, upload=256340]: Done uploading piece. err: <nil>
Nov 21 11:08:51.459688008 (291h4m50.011158469s) [noise=true , address=145.79.227.28:4725 , piece_id=CM4ARJXFPQ5L73AVRKXC3W2PHL5EAPBAU6FFWLO5FHBHHWZFCMHQ, seg_pos={0 0}, upload=256340]: Done uploading piece. err: <nil>
Nov 21 11:08:51.464453732 (291h4m50.015924193s) [noise=true , address=145.79.227.31:4797 , piece_id=ZYT3EYKH7F2D4P4Q37JRXSBCLHRACWDFMJOXSUA2BRILP7XNH22A, seg_pos={0 0}, upload=256340]: Done uploading piece. err: <nil>
Nov 21 11:08:51.473249190 (291h4m50.024719652s) [noise=true , address=145.79.227.9:4365 , piece_id=HCEAP2KU7CN7RG4V4TC3ONKVJ6ZTNMGI6XRLSDHFW4NOE6D2JJLA, seg_pos={0 0}, upload=256340]: Done uploading piece. err: <nil>
Nov 21 11:08:51.475322479 (291h4m50.02679294s) [noise=true , address=145.79.227.17:4971 , piece_id=JBCGNQCBZKRVESI25WPG7UEGQELZLNT6ZDAUBSDI2XE5ITU3K6NQ, seg_pos={0 0}, upload=256340]: Done uploading piece. err: <nil>
Nov 21 11:08:51.475584277 (291h4m50.027054738s) [noise=true , address=145.79.227.27:4689 , piece_id=B2ARHC33I7MAVPVUM5BHYQ7VXIB5UVJWCZ4KJCQGYDXA5J3OTTHA, seg_pos={0 0}, upload=256340]: Done uploading piece. err: <nil>
Nov 21 11:08:51.477161892 (291h4m50.028632353s) [noise=true , address=145.79.227.28:4727 , piece_id=GJOERP3WIDS2C4VQGEYA4H3OHV64V4GOWYDGPIMW5SLZRMZMUBIQ, seg_pos={0 0}, upload=256340]: Done uploading piece. err: <nil>
Nov 21 11:08:51.509396121 (291h4m50.060866582s) [noise=true , address=145.79.227.24:4583 , piece_id=7W2WYOH4QUDVTNIZER7ASJ7ECZE4ASS6WD5DDZMUEPOOPCL5M7QA, seg_pos={0 0}, upload=256340]: Done uploading piece. err: <nil>
Nov 21 11:08:52.022114547 (291h4m50.573585008s) [noise=true , address=77.90.31.15:4699 , piece_id=GW4M5UA3WL5OFBT3IHBPAXUUBCDCPK74WOLSXNRJ3YMALQ3VCNPQ, seg_pos={0 0}, upload=256340]: Done uploading piece. err: <nil>
Nov 21 11:08:52.100240452 (291h4m50.651710913s) [noise=true , address=77.90.31.19:4842 , piece_id=MVWVSRLOEPNDQNAJJ2TF5H3LVGNAG2POUVGCEHSNEIOWEJ37E7FA, seg_pos={0 0}, upload=256340]: Done uploading piece. err: <nil>
Nov 21 11:08:52.167139940 (291h4m50.718610401s) [noise=true , address=77.90.31.10:4520 , piece_id=ZWSQH4GSREBKPOTJFREGXLKMKK63WY6HUH5XBFGRBMEAG34TH5PA, seg_pos={0 0}, upload=256340]: Done uploading piece. err: <nil>
Nov 21 11:08:52.416887434 (291h4m50.968357885s) [noise=true , address=45.91.116.11:4117 , piece_id=7L25GK2GXANF42FOGD7XJXJP4CCXLXAMKCRHLVVJMEU3C3AOXIBA, seg_pos={0 0}, upload=256340]: Done uploading piece. err: <nil>
Nov 21 11:08:52.771095498 (291h4m51.322565959s) [noise=true , address=45.91.116.7:4010 , piece_id=F552HWH6XT2YJQABRP35GRN3KBZUTOL5DBK46HRSPXDS7G7PSIGQ, seg_pos={0 0}, upload=256340]: Done uploading piece. err: <nil>
Nov 21 11:08:52.771116288 (291h4m51.322586749s) [seg_pos={0 0}, upload=256340]: Segment reached optimal threshold of 13 pieces.
Nov 21 11:08:52.771337910 (291h4m51.322808361s) [noise=true , address=77.90.31.11:4554 , piece_id=I5TZO3QETRCMEUIJZOA45K3KFGYOG573Z6VSZAWMO6FGPYENZC7A, seg_pos={0 0}, upload=256340]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF
Nov 21 11:08:52.771346997 (291h4m51.322817458s) [noise=true , address=77.90.31.11:4554 , piece_id=I5TZO3QETRCMEUIJZOA45K3KFGYOG573Z6VSZAWMO6FGPYENZC7A, seg_pos={0 0}, upload=256340]: Long tail cancellation due to optimal threshold
Nov 21 11:08:52.771483657 (291h4m51.322954158s) [noise=true , address=45.91.116.4:4940 , piece_id=MNN2SGDFXIPT5FFUQMLOCSNFZLIREI3B6O2KHOM43GPEWOTETJYQ, seg_pos={0 0}, upload=256340]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF
Nov 21 11:08:52.771496833 (291h4m51.322967284s) [noise=true , address=45.91.116.4:4940 , piece_id=MNN2SGDFXIPT5FFUQMLOCSNFZLIREI3B6O2KHOM43GPEWOTETJYQ, seg_pos={0 0}, upload=256340]: Long tail cancellation due to optimal threshold
Nov 21 11:08:52.771504006 (291h4m51.322974467s) [noise=true , address=45.91.116.11:4119 , piece_id=PT3JGFZJK542ZTOETTB5DUF2T5QXBPJD4ETULT4BVQTYBH73CQLA, seg_pos={0 0}, upload=256340]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: write tcp 188.166.225.170:39358->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:39358->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39358->45.91.116.11:4119: use of closed network connection
Nov 21 11:08:52.771512422 (291h4m51.322982883s) [noise=true , address=45.91.116.11:4119 , piece_id=PT3JGFZJK542ZTOETTB5DUF2T5QXBPJD4ETULT4BVQTYBH73CQLA, seg_pos={0 0}, upload=256340]: Long tail cancellation due to optimal threshold
Nov 21 11:08:52.771686634 (291h4m51.323157095s) [seg_pos={0 0}, upload=256340]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Nov 21 11:08:52.771775644 (291h4m51.323246105s) [upload=256340]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Nov 21 11:08:52.771781364 (291h4m51.323251825s) [upload=256340]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Nov 21 11:08:53.069498780 (291h4m51.620969261s) [upload=256340]: Done uploading segments...
Nov 21 11:08:53.069524039 (291h4m51.6209945s) [upload=256340]: Upload finished. err: <nil>
Duration: 6.214634939s
core
uplink
us-select-1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [...................................................X........] duration:
388.668023ms remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/41cfd75a627799ce
FAIL (2025-11-21 11:09:37 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 388.690786ms
edge
linksharing
bgp
us1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
473.397785ms remote:
"tcp:207.211.208.129:443" target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318 trace:
https://jaeger.dev.storj.io/trace/5285054b3a776da7
FAIL (2025-11-21 11:09:24 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318": context deadline exceeded
Duration: 473.42698ms
edge
gateway
vhost
bgp
eu1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
385.717132ms remote:
"tcp:207.211.208.129:443" target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace:
https://jaeger.dev.storj.io/trace/3e5c193367f9a974
FAIL (2025-11-21 11:09:22 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded
Duration: 385.746529ms
edge
gateway
vhost
bgp
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
12 out of 60 is failed (threshold=6) [................................................XXXXXXXXXXXX] duration:
5.001090266s trace:
https://jaeger.dev.storj.io/trace/503f47f758b860b9
FAIL (2025-11-21 11:09:21 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 5.0011136s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......................................................X...X] duration:
30.003723508s trace:
https://jaeger.dev.storj.io/trace/3c71c0f34134d3e8
FAIL (2025-11-21 11:09:18 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: MVWVSRLOEPNDQNAJJ2TF5H3LVGNAG2POUVGCEHSNEIOWEJ37E7FA) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: 7L25GK2GXANF42FOGD7XJXJP4CCXLXAMKCRHLVVJMEU3C3AOXIBA) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: F552HWH6XT2YJQABRP35GRN3KBZUTOL5DBK46HRSPXDS7G7PSIGQ) context deadline exceeded
======LOGS======
Nov 21 11:08:49.220727082 (291h4m47.772197543s) [upload=256340]: Starting upload
Nov 21 11:08:49.220828385 (291h4m47.772298846s) [upload=256340]: Done starting upload
Nov 21 11:08:49.221068072 (291h4m47.772538533s) [upload=256340]: Uploading segments...
Nov 21 11:08:49.221175045 (291h4m47.772645506s) [upload=256340]: Got next segment. Inline: false
Nov 21 11:08:49.221239588 (291h4m47.772710049s) [upload=256340]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 21 11:08:49.221247403 (291h4m47.772717864s) [upload=256340]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 21 11:08:49.463941805 (291h4m48.015412266s) [seg_pos={0 0}, upload=256340]: Begin upload segment...
Nov 21 11:08:49.464007140 (291h4m48.015477601s) [seg_pos={0 0}, upload=256340]: Done begin upload segment.
Nov 21 11:08:49.464040994 (291h4m48.015511455s) [noise=true , address=45.91.116.7:4010 , piece_id=F552HWH6XT2YJQABRP35GRN3KBZUTOL5DBK46HRSPXDS7G7PSIGQ, seg_pos={0 0}, upload=256340]: Uploading piece...
Nov 21 11:08:49.464432970 (291h4m48.015903431s) [noise=true , address=145.79.227.21:4475 , piece_id=IMP7XE2IM235OC6CZIDPS7KCQHPVQWK3TU2BPTGANTEN55BPN7XQ, seg_pos={0 0}, upload=256340]: Uploading piece...
Nov 21 11:08:49.465081406 (291h4m48.016551867s) [noise=true , address=77.90.31.15:4699 , piece_id=GW4M5UA3WL5OFBT3IHBPAXUUBCDCPK74WOLSXNRJ3YMALQ3VCNPQ, seg_pos={0 0}, upload=256340]: Uploading piece...
Nov 21 11:08:49.465556651 (291h4m48.017027112s) [noise=true , address=77.90.31.11:4554 , piece_id=I5TZO3QETRCMEUIJZOA45K3KFGYOG573Z6VSZAWMO6FGPYENZC7A, seg_pos={0 0}, upload=256340]: Uploading piece...
Nov 21 11:08:49.466129752 (291h4m48.017600213s) [noise=true , address=145.79.227.28:4725 , piece_id=CM4ARJXFPQ5L73AVRKXC3W2PHL5EAPBAU6FFWLO5FHBHHWZFCMHQ, seg_pos={0 0}, upload=256340]: Uploading piece...
Nov 21 11:08:49.466926569 (291h4m48.01839703s) [noise=true , address=77.90.31.10:4520 , piece_id=ZWSQH4GSREBKPOTJFREGXLKMKK63WY6HUH5XBFGRBMEAG34TH5PA, seg_pos={0 0}, upload=256340]: Uploading piece...
Nov 21 11:08:49.467485784 (291h4m48.018956245s) [noise=true , address=77.90.31.19:4842 , piece_id=MVWVSRLOEPNDQNAJJ2TF5H3LVGNAG2POUVGCEHSNEIOWEJ37E7FA, seg_pos={0 0}, upload=256340]: Uploading piece...
Nov 21 11:08:49.468203270 (291h4m48.019673721s) [noise=true , address=45.91.116.4:4940 , piece_id=MNN2SGDFXIPT5FFUQMLOCSNFZLIREI3B6O2KHOM43GPEWOTETJYQ, seg_pos={0 0}, upload=256340]: Uploading piece...
Nov 21 11:08:49.468858177 (291h4m48.020328638s) [noise=true , address=145.79.227.31:4797 , piece_id=ZYT3EYKH7F2D4P4Q37JRXSBCLHRACWDFMJOXSUA2BRILP7XNH22A, seg_pos={0 0}, upload=256340]: Uploading piece...
Nov 21 11:08:49.470354446 (291h4m48.021824907s) [noise=true , address=145.79.227.24:4583 , piece_id=7W2WYOH4QUDVTNIZER7ASJ7ECZE4ASS6WD5DDZMUEPOOPCL5M7QA, seg_pos={0 0}, upload=256340]: Uploading piece...
Nov 21 11:08:49.472376998 (291h4m48.023847459s) [noise=true , address=145.79.227.17:4971 , piece_id=JBCGNQCBZKRVESI25WPG7UEGQELZLNT6ZDAUBSDI2XE5ITU3K6NQ, seg_pos={0 0}, upload=256340]: Uploading piece...
Nov 21 11:08:49.475441563 (291h4m48.026912025s) [noise=true , address=45.91.116.11:4117 , piece_id=7L25GK2GXANF42FOGD7XJXJP4CCXLXAMKCRHLVVJMEU3C3AOXIBA, seg_pos={0 0}, upload=256340]: Uploading piece...
Nov 21 11:08:49.477522856 (291h4m48.028993317s) [noise=true , address=145.79.227.27:4689 , piece_id=B2ARHC33I7MAVPVUM5BHYQ7VXIB5UVJWCZ4KJCQGYDXA5J3OTTHA, seg_pos={0 0}, upload=256340]: Uploading piece...
Nov 21 11:08:49.488511291 (291h4m48.039981752s) [noise=true , address=145.79.227.28:4727 , piece_id=GJOERP3WIDS2C4VQGEYA4H3OHV64V4GOWYDGPIMW5SLZRMZMUBIQ, seg_pos={0 0}, upload=256340]: Uploading piece...
Nov 21 11:08:49.489403249 (291h4m48.04087372s) [noise=true , address=45.91.116.11:4119 , piece_id=PT3JGFZJK542ZTOETTB5DUF2T5QXBPJD4ETULT4BVQTYBH73CQLA, seg_pos={0 0}, upload=256340]: Uploading piece...
Nov 21 11:08:49.492101117 (291h4m48.043571578s) [noise=true , address=145.79.227.9:4365 , piece_id=HCEAP2KU7CN7RG4V4TC3ONKVJ6ZTNMGI6XRLSDHFW4NOE6D2JJLA, seg_pos={0 0}, upload=256340]: Uploading piece...
Nov 21 11:08:51.131275763 (291h4m49.682746224s) [upload=256340]: Next returned nil segment
Nov 21 11:08:51.131303967 (291h4m49.682774428s) [upload=256340]: Waiting for error group managing segments...
Nov 21 11:08:51.456208272 (291h4m50.007678733s) [noise=true , address=145.79.227.21:4475 , piece_id=IMP7XE2IM235OC6CZIDPS7KCQHPVQWK3TU2BPTGANTEN55BPN7XQ, seg_pos={0 0}, upload=256340]: Done uploading piece. err: <nil>
Nov 21 11:08:51.459688008 (291h4m50.011158469s) [noise=true , address=145.79.227.28:4725 , piece_id=CM4ARJXFPQ5L73AVRKXC3W2PHL5EAPBAU6FFWLO5FHBHHWZFCMHQ, seg_pos={0 0}, upload=256340]: Done uploading piece. err: <nil>
Nov 21 11:08:51.464453732 (291h4m50.015924193s) [noise=true , address=145.79.227.31:4797 , piece_id=ZYT3EYKH7F2D4P4Q37JRXSBCLHRACWDFMJOXSUA2BRILP7XNH22A, seg_pos={0 0}, upload=256340]: Done uploading piece. err: <nil>
Nov 21 11:08:51.473249190 (291h4m50.024719652s) [noise=true , address=145.79.227.9:4365 , piece_id=HCEAP2KU7CN7RG4V4TC3ONKVJ6ZTNMGI6XRLSDHFW4NOE6D2JJLA, seg_pos={0 0}, upload=256340]: Done uploading piece. err: <nil>
Nov 21 11:08:51.475322479 (291h4m50.02679294s) [noise=true , address=145.79.227.17:4971 , piece_id=JBCGNQCBZKRVESI25WPG7UEGQELZLNT6ZDAUBSDI2XE5ITU3K6NQ, seg_pos={0 0}, upload=256340]: Done uploading piece. err: <nil>
Nov 21 11:08:51.475584277 (291h4m50.027054738s) [noise=true , address=145.79.227.27:4689 , piece_id=B2ARHC33I7MAVPVUM5BHYQ7VXIB5UVJWCZ4KJCQGYDXA5J3OTTHA, seg_pos={0 0}, upload=256340]: Done uploading piece. err: <nil>
Nov 21 11:08:51.477161892 (291h4m50.028632353s) [noise=true , address=145.79.227.28:4727 , piece_id=GJOERP3WIDS2C4VQGEYA4H3OHV64V4GOWYDGPIMW5SLZRMZMUBIQ, seg_pos={0 0}, upload=256340]: Done uploading piece. err: <nil>
Nov 21 11:08:51.509396121 (291h4m50.060866582s) [noise=true , address=145.79.227.24:4583 , piece_id=7W2WYOH4QUDVTNIZER7ASJ7ECZE4ASS6WD5DDZMUEPOOPCL5M7QA, seg_pos={0 0}, upload=256340]: Done uploading piece. err: <nil>
Nov 21 11:08:52.022114547 (291h4m50.573585008s) [noise=true , address=77.90.31.15:4699 , piece_id=GW4M5UA3WL5OFBT3IHBPAXUUBCDCPK74WOLSXNRJ3YMALQ3VCNPQ, seg_pos={0 0}, upload=256340]: Done uploading piece. err: <nil>
Nov 21 11:08:52.100240452 (291h4m50.651710913s) [noise=true , address=77.90.31.19:4842 , piece_id=MVWVSRLOEPNDQNAJJ2TF5H3LVGNAG2POUVGCEHSNEIOWEJ37E7FA, seg_pos={0 0}, upload=256340]: Done uploading piece. err: <nil>
Nov 21 11:08:52.167139940 (291h4m50.718610401s) [noise=true , address=77.90.31.10:4520 , piece_id=ZWSQH4GSREBKPOTJFREGXLKMKK63WY6HUH5XBFGRBMEAG34TH5PA, seg_pos={0 0}, upload=256340]: Done uploading piece. err: <nil>
Nov 21 11:08:52.416887434 (291h4m50.968357885s) [noise=true , address=45.91.116.11:4117 , piece_id=7L25GK2GXANF42FOGD7XJXJP4CCXLXAMKCRHLVVJMEU3C3AOXIBA, seg_pos={0 0}, upload=256340]: Done uploading piece. err: <nil>
Nov 21 11:08:52.771095498 (291h4m51.322565959s) [noise=true , address=45.91.116.7:4010 , piece_id=F552HWH6XT2YJQABRP35GRN3KBZUTOL5DBK46HRSPXDS7G7PSIGQ, seg_pos={0 0}, upload=256340]: Done uploading piece. err: <nil>
Nov 21 11:08:52.771116288 (291h4m51.322586749s) [seg_pos={0 0}, upload=256340]: Segment reached optimal threshold of 13 pieces.
Nov 21 11:08:52.771337910 (291h4m51.322808361s) [noise=true , address=77.90.31.11:4554 , piece_id=I5TZO3QETRCMEUIJZOA45K3KFGYOG573Z6VSZAWMO6FGPYENZC7A, seg_pos={0 0}, upload=256340]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF
Nov 21 11:08:52.771346997 (291h4m51.322817458s) [noise=true , address=77.90.31.11:4554 , piece_id=I5TZO3QETRCMEUIJZOA45K3KFGYOG573Z6VSZAWMO6FGPYENZC7A, seg_pos={0 0}, upload=256340]: Long tail cancellation due to optimal threshold
Nov 21 11:08:52.771483657 (291h4m51.322954158s) [noise=true , address=45.91.116.4:4940 , piece_id=MNN2SGDFXIPT5FFUQMLOCSNFZLIREI3B6O2KHOM43GPEWOTETJYQ, seg_pos={0 0}, upload=256340]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF
Nov 21 11:08:52.771496833 (291h4m51.322967284s) [noise=true , address=45.91.116.4:4940 , piece_id=MNN2SGDFXIPT5FFUQMLOCSNFZLIREI3B6O2KHOM43GPEWOTETJYQ, seg_pos={0 0}, upload=256340]: Long tail cancellation due to optimal threshold
Nov 21 11:08:52.771504006 (291h4m51.322974467s) [noise=true , address=45.91.116.11:4119 , piece_id=PT3JGFZJK542ZTOETTB5DUF2T5QXBPJD4ETULT4BVQTYBH73CQLA, seg_pos={0 0}, upload=256340]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: write tcp 188.166.225.170:39358->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:39358->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39358->45.91.116.11:4119: use of closed network connection
Nov 21 11:08:52.771512422 (291h4m51.322982883s) [noise=true , address=45.91.116.11:4119 , piece_id=PT3JGFZJK542ZTOETTB5DUF2T5QXBPJD4ETULT4BVQTYBH73CQLA, seg_pos={0 0}, upload=256340]: Long tail cancellation due to optimal threshold
Nov 21 11:08:52.771686634 (291h4m51.323157095s) [seg_pos={0 0}, upload=256340]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Nov 21 11:08:52.771775644 (291h4m51.323246105s) [upload=256340]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Nov 21 11:08:52.771781364 (291h4m51.323251825s) [upload=256340]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Nov 21 11:08:53.069498780 (291h4m51.620969261s) [upload=256340]: Done uploading segments...
Nov 21 11:08:53.069524039 (291h4m51.6209945s) [upload=256340]: Upload finished. err: <nil>
Duration: 30.003748124s
core
uplink
us-select-1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [....................................................X.......] duration:
472.45152ms remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/e2bae25cf885024
FAIL (2025-11-21 11:09:06 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 472.481959ms
edge
linksharing
bgp
eu1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
472.638747ms remote:
"tcp:207.211.208.129:443" target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318 trace:
https://jaeger.dev.storj.io/trace/5285054b3a776da7
FAIL (2025-11-21 11:09:05 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318": context deadline exceeded
Duration: 472.675889ms
edge
gateway
vhost
bgp
eu1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [....................................................X.......] duration:
427.89053ms remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/41cfd75a627799ce
FAIL (2025-11-21 11:08:58 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 427.921559ms
edge
linksharing
bgp
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
11 out of 60 is failed (threshold=6) [.................................................XXXXXXXXXXX] duration:
5.000639066s trace:
https://jaeger.dev.storj.io/trace/34869364a71ee9d2
FAIL (2025-11-21 11:08:57 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 5.00067802s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)antiblip:
1 out of 15 is failed (threshold=6) [......X........] duration:
827.894277ms remote:
"tcp:[2620:cb:2000::1]:443" target:
https://www.storj.io/dcs-satellites trace:
https://jaeger.dev.storj.io/trace/3d967bc11165dc1c
FAIL (2025-11-21 11:08:38 UTC)
orbiter warning: Get "https://www.storj.io/dcs-satellites": context deadline exceeded
Duration: 827.931608ms
core
website
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
387.496061ms remote:
"tcp:207.211.208.129:443" target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace:
https://jaeger.dev.storj.io/trace/3e5c193367f9a974
FAIL (2025-11-21 11:08:29 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded
Duration: 387.523583ms
edge
gateway
vhost
bgp
us1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [.....................................................X......] duration:
450.055214ms remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/e2bae25cf885024
FAIL (2025-11-21 11:08:20 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 450.080273ms
edge
linksharing
bgp
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
10 out of 60 is failed (threshold=6) [..................................................XXXXXXXXXX] duration:
5.000377449s trace:
https://jaeger.dev.storj.io/trace/a3c7302e1936b62
FAIL (2025-11-21 11:08:19 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 5.000404109s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [.....................................................X......] duration:
389.32833ms remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/41cfd75a627799ce
FAIL (2025-11-21 11:08:07 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 389.35484ms
edge
linksharing
bgp
us1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
387.902304ms remote:
"tcp:207.211.208.129:443" target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace:
https://jaeger.dev.storj.io/trace/3e5c193367f9a974
FAIL (2025-11-21 11:07:56 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded
Duration: 387.933203ms
edge
gateway
vhost
bgp
us1
|
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)antiblip:
1 out of 15 is failed (threshold=6) [.......X.......] duration:
2.852068931s remote:
"tcp:[2620:cb:2000::1]:443" target:
https://www.storj.io/dcs-satellites trace:
https://jaeger.dev.storj.io/trace/3d967bc11165dc1c
FAIL (2025-11-21 11:07:55 UTC)
orbiter warning: Get "https://www.storj.io/dcs-satellites": context deadline exceeded
Duration: 2.852095341s
core
website
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
449.313362ms remote:
"tcp:207.211.208.129:443" target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318 trace:
https://jaeger.dev.storj.io/trace/5285054b3a776da7
FAIL (2025-11-21 11:07:52 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318": context deadline exceeded
Duration: 449.338259ms
edge
gateway
vhost
bgp
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
9 out of 60 is failed (threshold=6) [...................................................XXXXXXXXX] duration:
5.000499711s trace:
https://jaeger.dev.storj.io/trace/4bdcf20ce72cd050
FAIL (2025-11-21 11:07:52 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 5.000526243s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
5.849037576s trace:
https://jaeger.dev.storj.io/trace/3ff262bbbebe046
FAIL (2025-11-21 11:07:51 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: 4FYWPO6FR5HNS7F2KZMRW2VXGJBHJSPPO4N55SI6UG2Y4LZ5WUKQ) context deadline exceeded; piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: MARTOW2V3H5NR3SBKM6MYXJ2Q2ML7DTP6IUR4SIFZ2B77BA56NZA) context deadline exceeded
======LOGS======
Nov 21 11:06:24.838204274 (291h2m23.389674735s) [upload=256303]: Starting upload
Nov 21 11:06:24.838288034 (291h2m23.389758516s) [upload=256303]: Done starting upload
Nov 21 11:06:24.838439593 (291h2m23.389910064s) [upload=256303]: Uploading segments...
Nov 21 11:06:24.838540104 (291h2m23.390010555s) [upload=256303]: Got next segment. Inline: false
Nov 21 11:06:24.838595018 (291h2m23.390065479s) [upload=256303]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 21 11:06:24.838602894 (291h2m23.390073355s) [upload=256303]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 21 11:06:25.084425984 (291h2m23.635896455s) [seg_pos={0 0}, upload=256303]: Begin upload segment...
Nov 21 11:06:25.084486729 (291h2m23.63595719s) [seg_pos={0 0}, upload=256303]: Done begin upload segment.
Nov 21 11:06:25.084529852 (291h2m23.636000313s) [noise=true , address=45.91.116.5:4299 , piece_id=M7Y3KYROVSLXGNUJCQS3GRTNXIFJGMJAIRK6GWPAUIQJJWGAV52Q, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.084541544 (291h2m23.636011995s) [noise=true , address=145.79.227.14:4837 , piece_id=QM7PZLFXZ3MWW4LUM5UMIYKIRZK2OPHLIXHNREMCXLQ25LLHC4IA, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.084592862 (291h2m23.636063333s) [noise=true , address=77.90.31.16:4734 , piece_id=CJBGAKL77QTS73YNOH6AK4PXPNWT6GN3YM7CGXWKEFUW3P74GZOQ, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.085076642 (291h2m23.636547103s) [noise=true , address=145.79.227.20:4439 , piece_id=AMENDLG4OPVO36XYSIPJ7XH7DYUZWBCSD3W3NUKXAWYQII3LAAHQ, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.086032663 (291h2m23.637503134s) [noise=true , address=77.90.31.14:4662 , piece_id=4FYWPO6FR5HNS7F2KZMRW2VXGJBHJSPPO4N55SI6UG2Y4LZ5WUKQ, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.086505042 (291h2m23.637975503s) [noise=true , address=77.90.31.11:4553 , piece_id=MARTOW2V3H5NR3SBKM6MYXJ2Q2ML7DTP6IUR4SIFZ2B77BA56NZA, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.086961763 (291h2m23.638432224s) [noise=true , address=77.90.31.15:4699 , piece_id=NLACGDKFHVE3VNW2TMZM6TEZXJ6PAR2DD5PDWRI2YY6BGDFFWDVA, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.087417140 (291h2m23.638887611s) [noise=true , address=45.91.116.9:4263 , piece_id=ZIUOXQ7BI4DKLMJOABUZHNWKMKKN32JPJOAWLSJDTUCWDHCAMSRA, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.088855258 (291h2m23.640325719s) [noise=true , address=77.90.31.12:4589 , piece_id=4BINOYT4TRDYNDD65OP5HN2XKWKXBVG3CCB5TXW53A5QCRWK6MBA, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.088963253 (291h2m23.640433714s) [noise=true , address=145.79.227.32:4836 , piece_id=PKXA2XP3AZSLOY3IHXLDTIVEJN6IRTMAZNUCILBULSFP27OJ3T5Q, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.090121038 (291h2m23.641591489s) [noise=true , address=145.79.227.9:4365 , piece_id=A4UI5DMAVPMLF2EJAKDBEPOU5X6GAGPO3GCNBJSHY5Y3N7HW4RZA, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.090803207 (291h2m23.642273668s) [noise=true , address=145.79.227.26:4654 , piece_id=VPDD7NGUJTFPM2HMLGOY4D5GXTAMFX2JWH2RLPKLLIO4CD4PNXGA, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.093048271 (291h2m23.644518733s) [noise=true , address=145.79.227.30:4761 , piece_id=IDWVBXBWBCTZZFXHUJG2E2NV4B67EXIC6EX63XNHMZFL4XJPO5JQ, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.094970843 (291h2m23.646441304s) [noise=true , address=145.79.227.21:4475 , piece_id=NLRVDZNC5W2MZFNRQIWNWXFJ4MP62RJBLWF3HH3ATY7LVZRLKMNA, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.096762915 (291h2m23.648233376s) [noise=true , address=145.79.227.27:4690 , piece_id=6S7K5OADQRLEE3ZRLKIWNLA7JSASTIS6HTESGYWD4CISSWI3CKMQ, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.098650319 (291h2m23.65012077s) [noise=true , address=77.90.31.13:4627 , piece_id=AZ6CHK5J5M2LMIXLUKHL2WFDFCMLSFYHO3CL67NJAWQVABOYG4LQ, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:26.696581300 (291h2m25.248051771s) [upload=256303]: Next returned nil segment
Nov 21 11:06:26.696599815 (291h2m25.248070276s) [upload=256303]: Waiting for error group managing segments...
Nov 21 11:06:27.078862911 (291h2m25.630333372s) [noise=true , address=145.79.227.21:4475 , piece_id=NLRVDZNC5W2MZFNRQIWNWXFJ4MP62RJBLWF3HH3ATY7LVZRLKMNA, seg_pos={0 0}, upload=256303]: Done uploading piece. err: <nil>
Nov 21 11:06:27.081490584 (291h2m25.632961045s) [noise=true , address=145.79.227.14:4837 , piece_id=QM7PZLFXZ3MWW4LUM5UMIYKIRZK2OPHLIXHNREMCXLQ25LLHC4IA, seg_pos={0 0}, upload=256303]: Done uploading piece. err: <nil>
Nov 21 11:06:27.082286630 (291h2m25.633757091s) [noise=true , address=145.79.227.20:4439 , piece_id=AMENDLG4OPVO36XYSIPJ7XH7DYUZWBCSD3W3NUKXAWYQII3LAAHQ, seg_pos={0 0}, upload=256303]: Done uploading piece. err: <nil>
Nov 21 11:06:27.085753421 (291h2m25.637223882s) [noise=true , address=145.79.227.32:4836 , piece_id=PKXA2XP3AZSLOY3IHXLDTIVEJN6IRTMAZNUCILBULSFP27OJ3T5Q, seg_pos={0 0}, upload=256303]: Done uploading piece. err: <nil>
Nov 21 11:06:27.085843232 (291h2m25.637313694s) [noise=true , address=145.79.227.30:4761 , piece_id=IDWVBXBWBCTZZFXHUJG2E2NV4B67EXIC6EX63XNHMZFL4XJPO5JQ, seg_pos={0 0}, upload=256303]: Done uploading piece. err: <nil>
Nov 21 11:06:27.086449767 (291h2m25.637920229s) [noise=true , address=145.79.227.26:4654 , piece_id=VPDD7NGUJTFPM2HMLGOY4D5GXTAMFX2JWH2RLPKLLIO4CD4PNXGA, seg_pos={0 0}, upload=256303]: Done uploading piece. err: <nil>
Nov 21 11:06:27.087455823 (291h2m25.638926284s) [noise=true , address=145.79.227.27:4690 , piece_id=6S7K5OADQRLEE3ZRLKIWNLA7JSASTIS6HTESGYWD4CISSWI3CKMQ, seg_pos={0 0}, upload=256303]: Done uploading piece. err: <nil>
Nov 21 11:06:27.088077276 (291h2m25.639547737s) [noise=true , address=145.79.227.9:4365 , piece_id=A4UI5DMAVPMLF2EJAKDBEPOU5X6GAGPO3GCNBJSHY5Y3N7HW4RZA, seg_pos={0 0}, upload=256303]: Done uploading piece. err: <nil>
Nov 21 11:06:27.517670348 (291h2m26.069140799s) [noise=true , address=77.90.31.13:4627 , piece_id=AZ6CHK5J5M2LMIXLUKHL2WFDFCMLSFYHO3CL67NJAWQVABOYG4LQ, seg_pos={0 0}, upload=256303]: Done uploading piece. err: <nil>
Nov 21 11:06:27.624691484 (291h2m26.176161945s) [noise=true , address=77.90.31.14:4662 , piece_id=4FYWPO6FR5HNS7F2KZMRW2VXGJBHJSPPO4N55SI6UG2Y4LZ5WUKQ, seg_pos={0 0}, upload=256303]: Done uploading piece. err: <nil>
Nov 21 11:06:27.629204408 (291h2m26.180674869s) [noise=true , address=77.90.31.15:4699 , piece_id=NLACGDKFHVE3VNW2TMZM6TEZXJ6PAR2DD5PDWRI2YY6BGDFFWDVA, seg_pos={0 0}, upload=256303]: Done uploading piece. err: <nil>
Nov 21 11:06:27.642548848 (291h2m26.194019309s) [noise=true , address=77.90.31.12:4589 , piece_id=4BINOYT4TRDYNDD65OP5HN2XKWKXBVG3CCB5TXW53A5QCRWK6MBA, seg_pos={0 0}, upload=256303]: Done uploading piece. err: <nil>
Nov 21 11:06:27.658268020 (291h2m26.209738481s) [noise=true , address=77.90.31.11:4553 , piece_id=MARTOW2V3H5NR3SBKM6MYXJ2Q2ML7DTP6IUR4SIFZ2B77BA56NZA, seg_pos={0 0}, upload=256303]: Done uploading piece. err: <nil>
Nov 21 11:06:27.658291014 (291h2m26.209761475s) [seg_pos={0 0}, upload=256303]: Segment reached optimal threshold of 13 pieces.
Nov 21 11:06:27.658508819 (291h2m26.20997928s) [noise=true , address=45.91.116.9:4263 , piece_id=ZIUOXQ7BI4DKLMJOABUZHNWKMKKN32JPJOAWLSJDTUCWDHCAMSRA, seg_pos={0 0}, upload=256303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF
Nov 21 11:06:27.658515912 (291h2m26.209986363s) [noise=true , address=45.91.116.9:4263 , piece_id=ZIUOXQ7BI4DKLMJOABUZHNWKMKKN32JPJOAWLSJDTUCWDHCAMSRA, seg_pos={0 0}, upload=256303]: Long tail cancellation due to optimal threshold
Nov 21 11:06:27.658677420 (291h2m26.210147901s) [noise=true , address=45.91.116.5:4299 , piece_id=M7Y3KYROVSLXGNUJCQS3GRTNXIFJGMJAIRK6GWPAUIQJJWGAV52Q, seg_pos={0 0}, upload=256303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF
Nov 21 11:06:27.658688641 (291h2m26.210159102s) [noise=true , address=45.91.116.5:4299 , piece_id=M7Y3KYROVSLXGNUJCQS3GRTNXIFJGMJAIRK6GWPAUIQJJWGAV52Q, seg_pos={0 0}, upload=256303]: Long tail cancellation due to optimal threshold
Nov 21 11:06:27.658702157 (291h2m26.210172618s) [noise=true , address=77.90.31.16:4734 , piece_id=CJBGAKL77QTS73YNOH6AK4PXPNWT6GN3YM7CGXWKEFUW3P74GZOQ, seg_pos={0 0}, upload=256303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: expected piece hash; context canceled; EOF
Nov 21 11:06:27.658719650 (291h2m26.210190111s) [noise=true , address=77.90.31.16:4734 , piece_id=CJBGAKL77QTS73YNOH6AK4PXPNWT6GN3YM7CGXWKEFUW3P74GZOQ, seg_pos={0 0}, upload=256303]: Long tail cancellation due to optimal threshold
Nov 21 11:06:27.658735580 (291h2m26.210206031s) [seg_pos={0 0}, upload=256303]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Nov 21 11:06:27.658812387 (291h2m26.210282848s) [upload=256303]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Nov 21 11:06:27.658818058 (291h2m26.210288519s) [upload=256303]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Nov 21 11:06:27.956716981 (291h2m26.508187432s) [upload=256303]: Done uploading segments...
Nov 21 11:06:27.956741057 (291h2m26.508211518s) [upload=256303]: Upload finished. err: <nil>
Duration: 5.84906612s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
6.225556665s trace:
https://jaeger.dev.storj.io/trace/3ff262bbbebe046
FAIL (2025-11-21 11:07:35 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: 4FYWPO6FR5HNS7F2KZMRW2VXGJBHJSPPO4N55SI6UG2Y4LZ5WUKQ) context deadline exceeded; piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: MARTOW2V3H5NR3SBKM6MYXJ2Q2ML7DTP6IUR4SIFZ2B77BA56NZA) context deadline exceeded
======LOGS======
Nov 21 11:06:24.838204274 (291h2m23.389674735s) [upload=256303]: Starting upload
Nov 21 11:06:24.838288034 (291h2m23.389758516s) [upload=256303]: Done starting upload
Nov 21 11:06:24.838439593 (291h2m23.389910064s) [upload=256303]: Uploading segments...
Nov 21 11:06:24.838540104 (291h2m23.390010555s) [upload=256303]: Got next segment. Inline: false
Nov 21 11:06:24.838595018 (291h2m23.390065479s) [upload=256303]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 21 11:06:24.838602894 (291h2m23.390073355s) [upload=256303]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 21 11:06:25.084425984 (291h2m23.635896455s) [seg_pos={0 0}, upload=256303]: Begin upload segment...
Nov 21 11:06:25.084486729 (291h2m23.63595719s) [seg_pos={0 0}, upload=256303]: Done begin upload segment.
Nov 21 11:06:25.084529852 (291h2m23.636000313s) [noise=true , address=45.91.116.5:4299 , piece_id=M7Y3KYROVSLXGNUJCQS3GRTNXIFJGMJAIRK6GWPAUIQJJWGAV52Q, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.084541544 (291h2m23.636011995s) [noise=true , address=145.79.227.14:4837 , piece_id=QM7PZLFXZ3MWW4LUM5UMIYKIRZK2OPHLIXHNREMCXLQ25LLHC4IA, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.084592862 (291h2m23.636063333s) [noise=true , address=77.90.31.16:4734 , piece_id=CJBGAKL77QTS73YNOH6AK4PXPNWT6GN3YM7CGXWKEFUW3P74GZOQ, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.085076642 (291h2m23.636547103s) [noise=true , address=145.79.227.20:4439 , piece_id=AMENDLG4OPVO36XYSIPJ7XH7DYUZWBCSD3W3NUKXAWYQII3LAAHQ, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.086032663 (291h2m23.637503134s) [noise=true , address=77.90.31.14:4662 , piece_id=4FYWPO6FR5HNS7F2KZMRW2VXGJBHJSPPO4N55SI6UG2Y4LZ5WUKQ, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.086505042 (291h2m23.637975503s) [noise=true , address=77.90.31.11:4553 , piece_id=MARTOW2V3H5NR3SBKM6MYXJ2Q2ML7DTP6IUR4SIFZ2B77BA56NZA, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.086961763 (291h2m23.638432224s) [noise=true , address=77.90.31.15:4699 , piece_id=NLACGDKFHVE3VNW2TMZM6TEZXJ6PAR2DD5PDWRI2YY6BGDFFWDVA, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.087417140 (291h2m23.638887611s) [noise=true , address=45.91.116.9:4263 , piece_id=ZIUOXQ7BI4DKLMJOABUZHNWKMKKN32JPJOAWLSJDTUCWDHCAMSRA, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.088855258 (291h2m23.640325719s) [noise=true , address=77.90.31.12:4589 , piece_id=4BINOYT4TRDYNDD65OP5HN2XKWKXBVG3CCB5TXW53A5QCRWK6MBA, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.088963253 (291h2m23.640433714s) [noise=true , address=145.79.227.32:4836 , piece_id=PKXA2XP3AZSLOY3IHXLDTIVEJN6IRTMAZNUCILBULSFP27OJ3T5Q, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.090121038 (291h2m23.641591489s) [noise=true , address=145.79.227.9:4365 , piece_id=A4UI5DMAVPMLF2EJAKDBEPOU5X6GAGPO3GCNBJSHY5Y3N7HW4RZA, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.090803207 (291h2m23.642273668s) [noise=true , address=145.79.227.26:4654 , piece_id=VPDD7NGUJTFPM2HMLGOY4D5GXTAMFX2JWH2RLPKLLIO4CD4PNXGA, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.093048271 (291h2m23.644518733s) [noise=true , address=145.79.227.30:4761 , piece_id=IDWVBXBWBCTZZFXHUJG2E2NV4B67EXIC6EX63XNHMZFL4XJPO5JQ, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.094970843 (291h2m23.646441304s) [noise=true , address=145.79.227.21:4475 , piece_id=NLRVDZNC5W2MZFNRQIWNWXFJ4MP62RJBLWF3HH3ATY7LVZRLKMNA, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.096762915 (291h2m23.648233376s) [noise=true , address=145.79.227.27:4690 , piece_id=6S7K5OADQRLEE3ZRLKIWNLA7JSASTIS6HTESGYWD4CISSWI3CKMQ, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.098650319 (291h2m23.65012077s) [noise=true , address=77.90.31.13:4627 , piece_id=AZ6CHK5J5M2LMIXLUKHL2WFDFCMLSFYHO3CL67NJAWQVABOYG4LQ, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:26.696581300 (291h2m25.248051771s) [upload=256303]: Next returned nil segment
Nov 21 11:06:26.696599815 (291h2m25.248070276s) [upload=256303]: Waiting for error group managing segments...
Nov 21 11:06:27.078862911 (291h2m25.630333372s) [noise=true , address=145.79.227.21:4475 , piece_id=NLRVDZNC5W2MZFNRQIWNWXFJ4MP62RJBLWF3HH3ATY7LVZRLKMNA, seg_pos={0 0}, upload=256303]: Done uploading piece. err: <nil>
Nov 21 11:06:27.081490584 (291h2m25.632961045s) [noise=true , address=145.79.227.14:4837 , piece_id=QM7PZLFXZ3MWW4LUM5UMIYKIRZK2OPHLIXHNREMCXLQ25LLHC4IA, seg_pos={0 0}, upload=256303]: Done uploading piece. err: <nil>
Nov 21 11:06:27.082286630 (291h2m25.633757091s) [noise=true , address=145.79.227.20:4439 , piece_id=AMENDLG4OPVO36XYSIPJ7XH7DYUZWBCSD3W3NUKXAWYQII3LAAHQ, seg_pos={0 0}, upload=256303]: Done uploading piece. err: <nil>
Nov 21 11:06:27.085753421 (291h2m25.637223882s) [noise=true , address=145.79.227.32:4836 , piece_id=PKXA2XP3AZSLOY3IHXLDTIVEJN6IRTMAZNUCILBULSFP27OJ3T5Q, seg_pos={0 0}, upload=256303]: Done uploading piece. err: <nil>
Nov 21 11:06:27.085843232 (291h2m25.637313694s) [noise=true , address=145.79.227.30:4761 , piece_id=IDWVBXBWBCTZZFXHUJG2E2NV4B67EXIC6EX63XNHMZFL4XJPO5JQ, seg_pos={0 0}, upload=256303]: Done uploading piece. err: <nil>
Nov 21 11:06:27.086449767 (291h2m25.637920229s) [noise=true , address=145.79.227.26:4654 , piece_id=VPDD7NGUJTFPM2HMLGOY4D5GXTAMFX2JWH2RLPKLLIO4CD4PNXGA, seg_pos={0 0}, upload=256303]: Done uploading piece. err: <nil>
Nov 21 11:06:27.087455823 (291h2m25.638926284s) [noise=true , address=145.79.227.27:4690 , piece_id=6S7K5OADQRLEE3ZRLKIWNLA7JSASTIS6HTESGYWD4CISSWI3CKMQ, seg_pos={0 0}, upload=256303]: Done uploading piece. err: <nil>
Nov 21 11:06:27.088077276 (291h2m25.639547737s) [noise=true , address=145.79.227.9:4365 , piece_id=A4UI5DMAVPMLF2EJAKDBEPOU5X6GAGPO3GCNBJSHY5Y3N7HW4RZA, seg_pos={0 0}, upload=256303]: Done uploading piece. err: <nil>
Nov 21 11:06:27.517670348 (291h2m26.069140799s) [noise=true , address=77.90.31.13:4627 , piece_id=AZ6CHK5J5M2LMIXLUKHL2WFDFCMLSFYHO3CL67NJAWQVABOYG4LQ, seg_pos={0 0}, upload=256303]: Done uploading piece. err: <nil>
Nov 21 11:06:27.624691484 (291h2m26.176161945s) [noise=true , address=77.90.31.14:4662 , piece_id=4FYWPO6FR5HNS7F2KZMRW2VXGJBHJSPPO4N55SI6UG2Y4LZ5WUKQ, seg_pos={0 0}, upload=256303]: Done uploading piece. err: <nil>
Nov 21 11:06:27.629204408 (291h2m26.180674869s) [noise=true , address=77.90.31.15:4699 , piece_id=NLACGDKFHVE3VNW2TMZM6TEZXJ6PAR2DD5PDWRI2YY6BGDFFWDVA, seg_pos={0 0}, upload=256303]: Done uploading piece. err: <nil>
Nov 21 11:06:27.642548848 (291h2m26.194019309s) [noise=true , address=77.90.31.12:4589 , piece_id=4BINOYT4TRDYNDD65OP5HN2XKWKXBVG3CCB5TXW53A5QCRWK6MBA, seg_pos={0 0}, upload=256303]: Done uploading piece. err: <nil>
Nov 21 11:06:27.658268020 (291h2m26.209738481s) [noise=true , address=77.90.31.11:4553 , piece_id=MARTOW2V3H5NR3SBKM6MYXJ2Q2ML7DTP6IUR4SIFZ2B77BA56NZA, seg_pos={0 0}, upload=256303]: Done uploading piece. err: <nil>
Nov 21 11:06:27.658291014 (291h2m26.209761475s) [seg_pos={0 0}, upload=256303]: Segment reached optimal threshold of 13 pieces.
Nov 21 11:06:27.658508819 (291h2m26.20997928s) [noise=true , address=45.91.116.9:4263 , piece_id=ZIUOXQ7BI4DKLMJOABUZHNWKMKKN32JPJOAWLSJDTUCWDHCAMSRA, seg_pos={0 0}, upload=256303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF
Nov 21 11:06:27.658515912 (291h2m26.209986363s) [noise=true , address=45.91.116.9:4263 , piece_id=ZIUOXQ7BI4DKLMJOABUZHNWKMKKN32JPJOAWLSJDTUCWDHCAMSRA, seg_pos={0 0}, upload=256303]: Long tail cancellation due to optimal threshold
Nov 21 11:06:27.658677420 (291h2m26.210147901s) [noise=true , address=45.91.116.5:4299 , piece_id=M7Y3KYROVSLXGNUJCQS3GRTNXIFJGMJAIRK6GWPAUIQJJWGAV52Q, seg_pos={0 0}, upload=256303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF
Nov 21 11:06:27.658688641 (291h2m26.210159102s) [noise=true , address=45.91.116.5:4299 , piece_id=M7Y3KYROVSLXGNUJCQS3GRTNXIFJGMJAIRK6GWPAUIQJJWGAV52Q, seg_pos={0 0}, upload=256303]: Long tail cancellation due to optimal threshold
Nov 21 11:06:27.658702157 (291h2m26.210172618s) [noise=true , address=77.90.31.16:4734 , piece_id=CJBGAKL77QTS73YNOH6AK4PXPNWT6GN3YM7CGXWKEFUW3P74GZOQ, seg_pos={0 0}, upload=256303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: expected piece hash; context canceled; EOF
Nov 21 11:06:27.658719650 (291h2m26.210190111s) [noise=true , address=77.90.31.16:4734 , piece_id=CJBGAKL77QTS73YNOH6AK4PXPNWT6GN3YM7CGXWKEFUW3P74GZOQ, seg_pos={0 0}, upload=256303]: Long tail cancellation due to optimal threshold
Nov 21 11:06:27.658735580 (291h2m26.210206031s) [seg_pos={0 0}, upload=256303]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Nov 21 11:06:27.658812387 (291h2m26.210282848s) [upload=256303]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Nov 21 11:06:27.658818058 (291h2m26.210288519s) [upload=256303]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Nov 21 11:06:27.956716981 (291h2m26.508187432s) [upload=256303]: Done uploading segments...
Nov 21 11:06:27.956741057 (291h2m26.508211518s) [upload=256303]: Upload finished. err: <nil>
Duration: 6.225587935s
core
uplink
us-select-1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [......................................................X.....] duration:
401.422269ms remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/41cfd75a627799ce
FAIL (2025-11-21 11:07:28 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 401.444481ms
edge
linksharing
bgp
us1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [......................................................X.....] duration:
478.172685ms remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/e2bae25cf885024
FAIL (2025-11-21 11:07:22 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 478.207781ms
edge
linksharing
bgp
eu1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
380.201601ms remote:
"tcp:207.211.208.129:443" target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace:
https://jaeger.dev.storj.io/trace/3e5c193367f9a974
FAIL (2025-11-21 11:07:19 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded
Duration: 380.229595ms
edge
gateway
vhost
bgp
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
5.938586724s trace:
https://jaeger.dev.storj.io/trace/3ff262bbbebe046
FAIL (2025-11-21 11:07:16 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: 4FYWPO6FR5HNS7F2KZMRW2VXGJBHJSPPO4N55SI6UG2Y4LZ5WUKQ) context deadline exceeded; piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: MARTOW2V3H5NR3SBKM6MYXJ2Q2ML7DTP6IUR4SIFZ2B77BA56NZA) context deadline exceeded
======LOGS======
Nov 21 11:06:24.838204274 (291h2m23.389674735s) [upload=256303]: Starting upload
Nov 21 11:06:24.838288034 (291h2m23.389758516s) [upload=256303]: Done starting upload
Nov 21 11:06:24.838439593 (291h2m23.389910064s) [upload=256303]: Uploading segments...
Nov 21 11:06:24.838540104 (291h2m23.390010555s) [upload=256303]: Got next segment. Inline: false
Nov 21 11:06:24.838595018 (291h2m23.390065479s) [upload=256303]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 21 11:06:24.838602894 (291h2m23.390073355s) [upload=256303]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 21 11:06:25.084425984 (291h2m23.635896455s) [seg_pos={0 0}, upload=256303]: Begin upload segment...
Nov 21 11:06:25.084486729 (291h2m23.63595719s) [seg_pos={0 0}, upload=256303]: Done begin upload segment.
Nov 21 11:06:25.084529852 (291h2m23.636000313s) [noise=true , address=45.91.116.5:4299 , piece_id=M7Y3KYROVSLXGNUJCQS3GRTNXIFJGMJAIRK6GWPAUIQJJWGAV52Q, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.084541544 (291h2m23.636011995s) [noise=true , address=145.79.227.14:4837 , piece_id=QM7PZLFXZ3MWW4LUM5UMIYKIRZK2OPHLIXHNREMCXLQ25LLHC4IA, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.084592862 (291h2m23.636063333s) [noise=true , address=77.90.31.16:4734 , piece_id=CJBGAKL77QTS73YNOH6AK4PXPNWT6GN3YM7CGXWKEFUW3P74GZOQ, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.085076642 (291h2m23.636547103s) [noise=true , address=145.79.227.20:4439 , piece_id=AMENDLG4OPVO36XYSIPJ7XH7DYUZWBCSD3W3NUKXAWYQII3LAAHQ, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.086032663 (291h2m23.637503134s) [noise=true , address=77.90.31.14:4662 , piece_id=4FYWPO6FR5HNS7F2KZMRW2VXGJBHJSPPO4N55SI6UG2Y4LZ5WUKQ, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.086505042 (291h2m23.637975503s) [noise=true , address=77.90.31.11:4553 , piece_id=MARTOW2V3H5NR3SBKM6MYXJ2Q2ML7DTP6IUR4SIFZ2B77BA56NZA, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.086961763 (291h2m23.638432224s) [noise=true , address=77.90.31.15:4699 , piece_id=NLACGDKFHVE3VNW2TMZM6TEZXJ6PAR2DD5PDWRI2YY6BGDFFWDVA, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.087417140 (291h2m23.638887611s) [noise=true , address=45.91.116.9:4263 , piece_id=ZIUOXQ7BI4DKLMJOABUZHNWKMKKN32JPJOAWLSJDTUCWDHCAMSRA, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.088855258 (291h2m23.640325719s) [noise=true , address=77.90.31.12:4589 , piece_id=4BINOYT4TRDYNDD65OP5HN2XKWKXBVG3CCB5TXW53A5QCRWK6MBA, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.088963253 (291h2m23.640433714s) [noise=true , address=145.79.227.32:4836 , piece_id=PKXA2XP3AZSLOY3IHXLDTIVEJN6IRTMAZNUCILBULSFP27OJ3T5Q, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.090121038 (291h2m23.641591489s) [noise=true , address=145.79.227.9:4365 , piece_id=A4UI5DMAVPMLF2EJAKDBEPOU5X6GAGPO3GCNBJSHY5Y3N7HW4RZA, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.090803207 (291h2m23.642273668s) [noise=true , address=145.79.227.26:4654 , piece_id=VPDD7NGUJTFPM2HMLGOY4D5GXTAMFX2JWH2RLPKLLIO4CD4PNXGA, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.093048271 (291h2m23.644518733s) [noise=true , address=145.79.227.30:4761 , piece_id=IDWVBXBWBCTZZFXHUJG2E2NV4B67EXIC6EX63XNHMZFL4XJPO5JQ, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.094970843 (291h2m23.646441304s) [noise=true , address=145.79.227.21:4475 , piece_id=NLRVDZNC5W2MZFNRQIWNWXFJ4MP62RJBLWF3HH3ATY7LVZRLKMNA, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.096762915 (291h2m23.648233376s) [noise=true , address=145.79.227.27:4690 , piece_id=6S7K5OADQRLEE3ZRLKIWNLA7JSASTIS6HTESGYWD4CISSWI3CKMQ, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.098650319 (291h2m23.65012077s) [noise=true , address=77.90.31.13:4627 , piece_id=AZ6CHK5J5M2LMIXLUKHL2WFDFCMLSFYHO3CL67NJAWQVABOYG4LQ, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:26.696581300 (291h2m25.248051771s) [upload=256303]: Next returned nil segment
Nov 21 11:06:26.696599815 (291h2m25.248070276s) [upload=256303]: Waiting for error group managing segments...
Nov 21 11:06:27.078862911 (291h2m25.630333372s) [noise=true , address=145.79.227.21:4475 , piece_id=NLRVDZNC5W2MZFNRQIWNWXFJ4MP62RJBLWF3HH3ATY7LVZRLKMNA, seg_pos={0 0}, upload=256303]: Done uploading piece. err: <nil>
Nov 21 11:06:27.081490584 (291h2m25.632961045s) [noise=true , address=145.79.227.14:4837 , piece_id=QM7PZLFXZ3MWW4LUM5UMIYKIRZK2OPHLIXHNREMCXLQ25LLHC4IA, seg_pos={0 0}, upload=256303]: Done uploading piece. err: <nil>
Nov 21 11:06:27.082286630 (291h2m25.633757091s) [noise=true , address=145.79.227.20:4439 , piece_id=AMENDLG4OPVO36XYSIPJ7XH7DYUZWBCSD3W3NUKXAWYQII3LAAHQ, seg_pos={0 0}, upload=256303]: Done uploading piece. err: <nil>
Nov 21 11:06:27.085753421 (291h2m25.637223882s) [noise=true , address=145.79.227.32:4836 , piece_id=PKXA2XP3AZSLOY3IHXLDTIVEJN6IRTMAZNUCILBULSFP27OJ3T5Q, seg_pos={0 0}, upload=256303]: Done uploading piece. err: <nil>
Nov 21 11:06:27.085843232 (291h2m25.637313694s) [noise=true , address=145.79.227.30:4761 , piece_id=IDWVBXBWBCTZZFXHUJG2E2NV4B67EXIC6EX63XNHMZFL4XJPO5JQ, seg_pos={0 0}, upload=256303]: Done uploading piece. err: <nil>
Nov 21 11:06:27.086449767 (291h2m25.637920229s) [noise=true , address=145.79.227.26:4654 , piece_id=VPDD7NGUJTFPM2HMLGOY4D5GXTAMFX2JWH2RLPKLLIO4CD4PNXGA, seg_pos={0 0}, upload=256303]: Done uploading piece. err: <nil>
Nov 21 11:06:27.087455823 (291h2m25.638926284s) [noise=true , address=145.79.227.27:4690 , piece_id=6S7K5OADQRLEE3ZRLKIWNLA7JSASTIS6HTESGYWD4CISSWI3CKMQ, seg_pos={0 0}, upload=256303]: Done uploading piece. err: <nil>
Nov 21 11:06:27.088077276 (291h2m25.639547737s) [noise=true , address=145.79.227.9:4365 , piece_id=A4UI5DMAVPMLF2EJAKDBEPOU5X6GAGPO3GCNBJSHY5Y3N7HW4RZA, seg_pos={0 0}, upload=256303]: Done uploading piece. err: <nil>
Nov 21 11:06:27.517670348 (291h2m26.069140799s) [noise=true , address=77.90.31.13:4627 , piece_id=AZ6CHK5J5M2LMIXLUKHL2WFDFCMLSFYHO3CL67NJAWQVABOYG4LQ, seg_pos={0 0}, upload=256303]: Done uploading piece. err: <nil>
Nov 21 11:06:27.624691484 (291h2m26.176161945s) [noise=true , address=77.90.31.14:4662 , piece_id=4FYWPO6FR5HNS7F2KZMRW2VXGJBHJSPPO4N55SI6UG2Y4LZ5WUKQ, seg_pos={0 0}, upload=256303]: Done uploading piece. err: <nil>
Nov 21 11:06:27.629204408 (291h2m26.180674869s) [noise=true , address=77.90.31.15:4699 , piece_id=NLACGDKFHVE3VNW2TMZM6TEZXJ6PAR2DD5PDWRI2YY6BGDFFWDVA, seg_pos={0 0}, upload=256303]: Done uploading piece. err: <nil>
Nov 21 11:06:27.642548848 (291h2m26.194019309s) [noise=true , address=77.90.31.12:4589 , piece_id=4BINOYT4TRDYNDD65OP5HN2XKWKXBVG3CCB5TXW53A5QCRWK6MBA, seg_pos={0 0}, upload=256303]: Done uploading piece. err: <nil>
Nov 21 11:06:27.658268020 (291h2m26.209738481s) [noise=true , address=77.90.31.11:4553 , piece_id=MARTOW2V3H5NR3SBKM6MYXJ2Q2ML7DTP6IUR4SIFZ2B77BA56NZA, seg_pos={0 0}, upload=256303]: Done uploading piece. err: <nil>
Nov 21 11:06:27.658291014 (291h2m26.209761475s) [seg_pos={0 0}, upload=256303]: Segment reached optimal threshold of 13 pieces.
Nov 21 11:06:27.658508819 (291h2m26.20997928s) [noise=true , address=45.91.116.9:4263 , piece_id=ZIUOXQ7BI4DKLMJOABUZHNWKMKKN32JPJOAWLSJDTUCWDHCAMSRA, seg_pos={0 0}, upload=256303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF
Nov 21 11:06:27.658515912 (291h2m26.209986363s) [noise=true , address=45.91.116.9:4263 , piece_id=ZIUOXQ7BI4DKLMJOABUZHNWKMKKN32JPJOAWLSJDTUCWDHCAMSRA, seg_pos={0 0}, upload=256303]: Long tail cancellation due to optimal threshold
Nov 21 11:06:27.658677420 (291h2m26.210147901s) [noise=true , address=45.91.116.5:4299 , piece_id=M7Y3KYROVSLXGNUJCQS3GRTNXIFJGMJAIRK6GWPAUIQJJWGAV52Q, seg_pos={0 0}, upload=256303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF
Nov 21 11:06:27.658688641 (291h2m26.210159102s) [noise=true , address=45.91.116.5:4299 , piece_id=M7Y3KYROVSLXGNUJCQS3GRTNXIFJGMJAIRK6GWPAUIQJJWGAV52Q, seg_pos={0 0}, upload=256303]: Long tail cancellation due to optimal threshold
Nov 21 11:06:27.658702157 (291h2m26.210172618s) [noise=true , address=77.90.31.16:4734 , piece_id=CJBGAKL77QTS73YNOH6AK4PXPNWT6GN3YM7CGXWKEFUW3P74GZOQ, seg_pos={0 0}, upload=256303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: expected piece hash; context canceled; EOF
Nov 21 11:06:27.658719650 (291h2m26.210190111s) [noise=true , address=77.90.31.16:4734 , piece_id=CJBGAKL77QTS73YNOH6AK4PXPNWT6GN3YM7CGXWKEFUW3P74GZOQ, seg_pos={0 0}, upload=256303]: Long tail cancellation due to optimal threshold
Nov 21 11:06:27.658735580 (291h2m26.210206031s) [seg_pos={0 0}, upload=256303]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Nov 21 11:06:27.658812387 (291h2m26.210282848s) [upload=256303]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Nov 21 11:06:27.658818058 (291h2m26.210288519s) [upload=256303]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Nov 21 11:06:27.956716981 (291h2m26.508187432s) [upload=256303]: Done uploading segments...
Nov 21 11:06:27.956741057 (291h2m26.508211518s) [upload=256303]: Upload finished. err: <nil>
Duration: 5.938607203s
core
uplink
us-select-1
|
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)antiblip:
1 out of 15 is failed (threshold=6) [........X......] duration:
893.691836ms remote:
"tcp:[2620:cb:2000::1]:443" target:
https://www.storj.io/dcs-satellites trace:
https://jaeger.dev.storj.io/trace/3d967bc11165dc1c
FAIL (2025-11-21 11:07:10 UTC)
orbiter warning: Get "https://www.storj.io/dcs-satellites": context deadline exceeded
Duration: 893.707175ms
core
website
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [.......................................................X....] duration:
412.092126ms remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/41cfd75a627799ce
FAIL (2025-11-21 11:07:09 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 412.122023ms
edge
linksharing
bgp
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
8 out of 60 is failed (threshold=6) [....................................................XXXXXXXX] duration:
5.001141274s trace:
https://jaeger.dev.storj.io/trace/664f705c312776f4
FAIL (2025-11-21 11:07:09 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 5.001168686s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.00515788s trace:
https://jaeger.dev.storj.io/trace/3ff262bbbebe046
FAIL (2025-11-21 11:06:54 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: 4FYWPO6FR5HNS7F2KZMRW2VXGJBHJSPPO4N55SI6UG2Y4LZ5WUKQ) context deadline exceeded; piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: MARTOW2V3H5NR3SBKM6MYXJ2Q2ML7DTP6IUR4SIFZ2B77BA56NZA) context deadline exceeded
======LOGS======
Nov 21 11:06:24.838204274 (291h2m23.389674735s) [upload=256303]: Starting upload
Nov 21 11:06:24.838288034 (291h2m23.389758516s) [upload=256303]: Done starting upload
Nov 21 11:06:24.838439593 (291h2m23.389910064s) [upload=256303]: Uploading segments...
Nov 21 11:06:24.838540104 (291h2m23.390010555s) [upload=256303]: Got next segment. Inline: false
Nov 21 11:06:24.838595018 (291h2m23.390065479s) [upload=256303]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 21 11:06:24.838602894 (291h2m23.390073355s) [upload=256303]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 21 11:06:25.084425984 (291h2m23.635896455s) [seg_pos={0 0}, upload=256303]: Begin upload segment...
Nov 21 11:06:25.084486729 (291h2m23.63595719s) [seg_pos={0 0}, upload=256303]: Done begin upload segment.
Nov 21 11:06:25.084529852 (291h2m23.636000313s) [noise=true , address=45.91.116.5:4299 , piece_id=M7Y3KYROVSLXGNUJCQS3GRTNXIFJGMJAIRK6GWPAUIQJJWGAV52Q, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.084541544 (291h2m23.636011995s) [noise=true , address=145.79.227.14:4837 , piece_id=QM7PZLFXZ3MWW4LUM5UMIYKIRZK2OPHLIXHNREMCXLQ25LLHC4IA, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.084592862 (291h2m23.636063333s) [noise=true , address=77.90.31.16:4734 , piece_id=CJBGAKL77QTS73YNOH6AK4PXPNWT6GN3YM7CGXWKEFUW3P74GZOQ, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.085076642 (291h2m23.636547103s) [noise=true , address=145.79.227.20:4439 , piece_id=AMENDLG4OPVO36XYSIPJ7XH7DYUZWBCSD3W3NUKXAWYQII3LAAHQ, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.086032663 (291h2m23.637503134s) [noise=true , address=77.90.31.14:4662 , piece_id=4FYWPO6FR5HNS7F2KZMRW2VXGJBHJSPPO4N55SI6UG2Y4LZ5WUKQ, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.086505042 (291h2m23.637975503s) [noise=true , address=77.90.31.11:4553 , piece_id=MARTOW2V3H5NR3SBKM6MYXJ2Q2ML7DTP6IUR4SIFZ2B77BA56NZA, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.086961763 (291h2m23.638432224s) [noise=true , address=77.90.31.15:4699 , piece_id=NLACGDKFHVE3VNW2TMZM6TEZXJ6PAR2DD5PDWRI2YY6BGDFFWDVA, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.087417140 (291h2m23.638887611s) [noise=true , address=45.91.116.9:4263 , piece_id=ZIUOXQ7BI4DKLMJOABUZHNWKMKKN32JPJOAWLSJDTUCWDHCAMSRA, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.088855258 (291h2m23.640325719s) [noise=true , address=77.90.31.12:4589 , piece_id=4BINOYT4TRDYNDD65OP5HN2XKWKXBVG3CCB5TXW53A5QCRWK6MBA, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.088963253 (291h2m23.640433714s) [noise=true , address=145.79.227.32:4836 , piece_id=PKXA2XP3AZSLOY3IHXLDTIVEJN6IRTMAZNUCILBULSFP27OJ3T5Q, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.090121038 (291h2m23.641591489s) [noise=true , address=145.79.227.9:4365 , piece_id=A4UI5DMAVPMLF2EJAKDBEPOU5X6GAGPO3GCNBJSHY5Y3N7HW4RZA, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.090803207 (291h2m23.642273668s) [noise=true , address=145.79.227.26:4654 , piece_id=VPDD7NGUJTFPM2HMLGOY4D5GXTAMFX2JWH2RLPKLLIO4CD4PNXGA, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.093048271 (291h2m23.644518733s) [noise=true , address=145.79.227.30:4761 , piece_id=IDWVBXBWBCTZZFXHUJG2E2NV4B67EXIC6EX63XNHMZFL4XJPO5JQ, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.094970843 (291h2m23.646441304s) [noise=true , address=145.79.227.21:4475 , piece_id=NLRVDZNC5W2MZFNRQIWNWXFJ4MP62RJBLWF3HH3ATY7LVZRLKMNA, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.096762915 (291h2m23.648233376s) [noise=true , address=145.79.227.27:4690 , piece_id=6S7K5OADQRLEE3ZRLKIWNLA7JSASTIS6HTESGYWD4CISSWI3CKMQ, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:25.098650319 (291h2m23.65012077s) [noise=true , address=77.90.31.13:4627 , piece_id=AZ6CHK5J5M2LMIXLUKHL2WFDFCMLSFYHO3CL67NJAWQVABOYG4LQ, seg_pos={0 0}, upload=256303]: Uploading piece...
Nov 21 11:06:26.696581300 (291h2m25.248051771s) [upload=256303]: Next returned nil segment
Nov 21 11:06:26.696599815 (291h2m25.248070276s) [upload=256303]: Waiting for error group managing segments...
Nov 21 11:06:27.078862911 (291h2m25.630333372s) [noise=true , address=145.79.227.21:4475 , piece_id=NLRVDZNC5W2MZFNRQIWNWXFJ4MP62RJBLWF3HH3ATY7LVZRLKMNA, seg_pos={0 0}, upload=256303]: Done uploading piece. err: <nil>
Nov 21 11:06:27.081490584 (291h2m25.632961045s) [noise=true , address=145.79.227.14:4837 , piece_id=QM7PZLFXZ3MWW4LUM5UMIYKIRZK2OPHLIXHNREMCXLQ25LLHC4IA, seg_pos={0 0}, upload=256303]: Done uploading piece. err: <nil>
Nov 21 11:06:27.082286630 (291h2m25.633757091s) [noise=true , address=145.79.227.20:4439 , piece_id=AMENDLG4OPVO36XYSIPJ7XH7DYUZWBCSD3W3NUKXAWYQII3LAAHQ, seg_pos={0 0}, upload=256303]: Done uploading piece. err: <nil>
Nov 21 11:06:27.085753421 (291h2m25.637223882s) [noise=true , address=145.79.227.32:4836 , piece_id=PKXA2XP3AZSLOY3IHXLDTIVEJN6IRTMAZNUCILBULSFP27OJ3T5Q, seg_pos={0 0}, upload=256303]: Done uploading piece. err: <nil>
Nov 21 11:06:27.085843232 (291h2m25.637313694s) [noise=true , address=145.79.227.30:4761 , piece_id=IDWVBXBWBCTZZFXHUJG2E2NV4B67EXIC6EX63XNHMZFL4XJPO5JQ, seg_pos={0 0}, upload=256303]: Done uploading piece. err: <nil>
Nov 21 11:06:27.086449767 (291h2m25.637920229s) [noise=true , address=145.79.227.26:4654 , piece_id=VPDD7NGUJTFPM2HMLGOY4D5GXTAMFX2JWH2RLPKLLIO4CD4PNXGA, seg_pos={0 0}, upload=256303]: Done uploading piece. err: <nil>
Nov 21 11:06:27.087455823 (291h2m25.638926284s) [noise=true , address=145.79.227.27:4690 , piece_id=6S7K5OADQRLEE3ZRLKIWNLA7JSASTIS6HTESGYWD4CISSWI3CKMQ, seg_pos={0 0}, upload=256303]: Done uploading piece. err: <nil>
Nov 21 11:06:27.088077276 (291h2m25.639547737s) [noise=true , address=145.79.227.9:4365 , piece_id=A4UI5DMAVPMLF2EJAKDBEPOU5X6GAGPO3GCNBJSHY5Y3N7HW4RZA, seg_pos={0 0}, upload=256303]: Done uploading piece. err: <nil>
Nov 21 11:06:27.517670348 (291h2m26.069140799s) [noise=true , address=77.90.31.13:4627 , piece_id=AZ6CHK5J5M2LMIXLUKHL2WFDFCMLSFYHO3CL67NJAWQVABOYG4LQ, seg_pos={0 0}, upload=256303]: Done uploading piece. err: <nil>
Nov 21 11:06:27.624691484 (291h2m26.176161945s) [noise=true , address=77.90.31.14:4662 , piece_id=4FYWPO6FR5HNS7F2KZMRW2VXGJBHJSPPO4N55SI6UG2Y4LZ5WUKQ, seg_pos={0 0}, upload=256303]: Done uploading piece. err: <nil>
Nov 21 11:06:27.629204408 (291h2m26.180674869s) [noise=true , address=77.90.31.15:4699 , piece_id=NLACGDKFHVE3VNW2TMZM6TEZXJ6PAR2DD5PDWRI2YY6BGDFFWDVA, seg_pos={0 0}, upload=256303]: Done uploading piece. err: <nil>
Nov 21 11:06:27.642548848 (291h2m26.194019309s) [noise=true , address=77.90.31.12:4589 , piece_id=4BINOYT4TRDYNDD65OP5HN2XKWKXBVG3CCB5TXW53A5QCRWK6MBA, seg_pos={0 0}, upload=256303]: Done uploading piece. err: <nil>
Nov 21 11:06:27.658268020 (291h2m26.209738481s) [noise=true , address=77.90.31.11:4553 , piece_id=MARTOW2V3H5NR3SBKM6MYXJ2Q2ML7DTP6IUR4SIFZ2B77BA56NZA, seg_pos={0 0}, upload=256303]: Done uploading piece. err: <nil>
Nov 21 11:06:27.658291014 (291h2m26.209761475s) [seg_pos={0 0}, upload=256303]: Segment reached optimal threshold of 13 pieces.
Nov 21 11:06:27.658508819 (291h2m26.20997928s) [noise=true , address=45.91.116.9:4263 , piece_id=ZIUOXQ7BI4DKLMJOABUZHNWKMKKN32JPJOAWLSJDTUCWDHCAMSRA, seg_pos={0 0}, upload=256303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF
Nov 21 11:06:27.658515912 (291h2m26.209986363s) [noise=true , address=45.91.116.9:4263 , piece_id=ZIUOXQ7BI4DKLMJOABUZHNWKMKKN32JPJOAWLSJDTUCWDHCAMSRA, seg_pos={0 0}, upload=256303]: Long tail cancellation due to optimal threshold
Nov 21 11:06:27.658677420 (291h2m26.210147901s) [noise=true , address=45.91.116.5:4299 , piece_id=M7Y3KYROVSLXGNUJCQS3GRTNXIFJGMJAIRK6GWPAUIQJJWGAV52Q, seg_pos={0 0}, upload=256303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF
Nov 21 11:06:27.658688641 (291h2m26.210159102s) [noise=true , address=45.91.116.5:4299 , piece_id=M7Y3KYROVSLXGNUJCQS3GRTNXIFJGMJAIRK6GWPAUIQJJWGAV52Q, seg_pos={0 0}, upload=256303]: Long tail cancellation due to optimal threshold
Nov 21 11:06:27.658702157 (291h2m26.210172618s) [noise=true , address=77.90.31.16:4734 , piece_id=CJBGAKL77QTS73YNOH6AK4PXPNWT6GN3YM7CGXWKEFUW3P74GZOQ, seg_pos={0 0}, upload=256303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: expected piece hash; context canceled; EOF
Nov 21 11:06:27.658719650 (291h2m26.210190111s) [noise=true , address=77.90.31.16:4734 , piece_id=CJBGAKL77QTS73YNOH6AK4PXPNWT6GN3YM7CGXWKEFUW3P74GZOQ, seg_pos={0 0}, upload=256303]: Long tail cancellation due to optimal threshold
Nov 21 11:06:27.658735580 (291h2m26.210206031s) [seg_pos={0 0}, upload=256303]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Nov 21 11:06:27.658812387 (291h2m26.210282848s) [upload=256303]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Nov 21 11:06:27.658818058 (291h2m26.210288519s) [upload=256303]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Nov 21 11:06:27.956716981 (291h2m26.508187432s) [upload=256303]: Done uploading segments...
Nov 21 11:06:27.956741057 (291h2m26.508211518s) [upload=256303]: Upload finished. err: <nil>
Duration: 30.005185413s
core
uplink
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
467.193758ms remote:
"tcp:207.211.208.129:443" target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318 trace:
https://jaeger.dev.storj.io/trace/5285054b3a776da7
FAIL (2025-11-21 11:06:53 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318": context deadline exceeded
Duration: 467.229798ms
edge
gateway
vhost
bgp
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [.....................................................XXXXXXX] duration:
5.000960912s trace:
https://jaeger.dev.storj.io/trace/9d103e79dc13e5
FAIL (2025-11-21 11:06:52 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 5.000983012s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
488.16374ms remote:
"tcp:207.211.208.129:443" target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318 trace:
https://jaeger.dev.storj.io/trace/5285054b3a776da7
FAIL (2025-11-21 11:06:31 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318": context deadline exceeded
Duration: 488.185151ms
edge
gateway
vhost
bgp
eu1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [.......................................................X....] duration:
484.499604ms remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/e2bae25cf885024
FAIL (2025-11-21 11:06:28 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 484.531675ms
edge
linksharing
bgp
eu1
|
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)antiblip:
1 out of 15 is failed (threshold=6) [.........X.....] duration:
819.818248ms remote:
"tcp:[2620:cb:2000::1]:443" target:
https://www.storj.io/dcs-satellites trace:
https://jaeger.dev.storj.io/trace/3d967bc11165dc1c
FAIL (2025-11-21 11:06:26 UTC)
orbiter warning: Get "https://www.storj.io/dcs-satellites": context deadline exceeded
Duration: 819.845681ms
core
website
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [........................................................X...] duration:
407.529168ms remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/41cfd75a627799ce
FAIL (2025-11-21 11:06:22 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 407.679854ms
edge
linksharing
bgp
us1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
423.367636ms remote:
"tcp:207.211.208.129:443" target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace:
https://jaeger.dev.storj.io/trace/3e5c193367f9a974
FAIL (2025-11-21 11:06:14 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded
Duration: 423.399507ms
edge
gateway
vhost
bgp
us1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [........................................................X...] duration:
477.881642ms remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/e2bae25cf885024
FAIL (2025-11-21 11:06:11 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 477.907832ms
edge
linksharing
bgp
eu1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
474.900976ms remote:
"tcp:207.211.208.129:443" target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318 trace:
https://jaeger.dev.storj.io/trace/5285054b3a776da7
FAIL (2025-11-21 11:06:11 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318": context deadline exceeded
Duration: 474.921284ms
edge
gateway
vhost
bgp
eu1
|
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)antiblip:
1 out of 15 is failed (threshold=6) [..........X....] duration:
851.473876ms remote:
"tcp:[2620:cb:2000::1]:443" target:
https://www.storj.io/dcs-satellites trace:
https://jaeger.dev.storj.io/trace/3d967bc11165dc1c
FAIL (2025-11-21 11:06:00 UTC)
orbiter warning: Get "https://www.storj.io/dcs-satellites": context deadline exceeded
Duration: 851.50173ms
core
website
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
6 out of 60 is failed (threshold=6) [......................................................XXXXXX] duration:
5.00107115s trace:
https://jaeger.dev.storj.io/trace/1b3c4f72aeb9946e
FAIL (2025-11-21 11:05:59 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 5.001091168s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [.........................................................X..] duration:
407.527745ms remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/41cfd75a627799ce
FAIL (2025-11-21 11:05:56 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 407.557982ms
edge
linksharing
bgp
us1
|