Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [X..X.....................................X............X.X.XX] duration:
30.001860893s trace:
https://jaeger.dev.storj.io/trace/313baf22195be799
FAIL (2025-12-08 15:21:14 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001872705s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
8.415519855s trace:
https://jaeger.dev.storj.io/trace/153706e03a29fd88
FAIL (2025-12-08 15:21:02 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 8.4155542s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
7.457411933s trace:
https://jaeger.dev.storj.io/trace/54ba7b1e9d79a182
FAIL (2025-12-08 15:20:47 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 7.457428445s
core
uplink
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [.X..X.....................................X............X.X.X] duration:
30.001780625s trace:
https://jaeger.dev.storj.io/trace/f8b7e8719362570
FAIL (2025-12-08 15:20:43 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001797247s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
50 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XXXXX.XXX..X....X.] duration:
5.00074125s trace:
https://jaeger.dev.storj.io/trace/49ccacaae559446c
FAIL (2025-12-08 15:20:43 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000765556s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
9.06953916s trace:
https://jaeger.dev.storj.io/trace/19f19098eeec6fec
FAIL (2025-12-08 15:20:40 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, Piece ID: THMVWSNTE5SGGZZY4USNEFNEKSTCJM6CBCWLWMNFUFUZA6ATO4IA) context deadline exceeded; piecestore: (Node ID: 1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, Piece ID: XQ5BXGZMDPKFSIYMKUNGO6YMUTQYQ37OJGGKZFU5SFD2NJG45TNQ) context deadline exceeded; piecestore: (Node ID: 12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, Piece ID: RNL6C7WQFRDZ7Z5JN6J6UCHIO2WOPVTWFQC2JHROAWS3ZVEFBUEQ) context deadline exceeded
======LOGS======
Dec 8 15:19:33.770990212 (71h50m33.642073257s) [upload=62909]: Starting upload
Dec 8 15:19:33.771048684 (71h50m33.642131729s) [upload=62909]: Done starting upload
Dec 8 15:19:33.771219048 (71h50m33.642302083s) [upload=62909]: Uploading segments...
Dec 8 15:19:33.771337443 (71h50m33.642420478s) [upload=62909]: Got next segment. Inline: false
Dec 8 15:19:33.771385996 (71h50m33.642469031s) [upload=62909]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 15:19:33.771391647 (71h50m33.642474692s) [upload=62909]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 15:19:34.033637444 (71h50m33.904720489s) [seg_pos={0 0}, upload=62909]: Begin upload segment...
Dec 8 15:19:34.033686949 (71h50m33.904769984s) [seg_pos={0 0}, upload=62909]: Done begin upload segment.
Dec 8 15:19:34.033718579 (71h50m33.904801624s) [noise=true , address=145.79.227.29:4941 , piece_id=THMVWSNTE5SGGZZY4USNEFNEKSTCJM6CBCWLWMNFUFUZA6ATO4IA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.033735481 (71h50m33.904818526s) [noise=true , address=145.79.227.28:4725 , piece_id=JOTZWCSNR6HGKYEYT75SPRHTUR7R2DSCHFCRZ32KZ3FFG5EIR5PA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.035223314 (71h50m33.906306349s) [noise=true , address=145.79.227.23:4547 , piece_id=XQ5BXGZMDPKFSIYMKUNGO6YMUTQYQ37OJGGKZFU5SFD2NJG45TNQ, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.035531321 (71h50m33.906614356s) [noise=true , address=145.79.227.29:4942 , piece_id=O35KF7WMNWCV5MHYHNCXT765JS5SOWEJHDH6QL4AR23Y4AUOTFAA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.035867982 (71h50m33.906951027s) [noise=true , address=145.79.227.22:4509 , piece_id=BRV2XH7SZW2MPO53FENX2P23ICAWZQQ2LOMNQHVPWRAQJTT4JO2Q, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.037062737 (71h50m33.908145782s) [noise=true , address=145.79.227.5:4222 , piece_id=JCVF7WMKWK3JXGBTNBSIXWQPTR7LF57JTR45677SN7DUVYHP547A, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.037425418 (71h50m33.908508463s) [noise=true , address=145.79.227.38:4979 , piece_id=6CI4XIPXKSUI3ODW662O5XG63TMTJ3KIXDUYHQMQCTVHTLULWKPA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.037777228 (71h50m33.908860263s) [noise=true , address=145.79.227.22:4510 , piece_id=RNL6C7WQFRDZ7Z5JN6J6UCHIO2WOPVTWFQC2JHROAWS3ZVEFBUEQ, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.038199451 (71h50m33.909282487s) [noise=true , address=145.79.227.20:4439 , piece_id=LMKA26FPXNRAH5YZISRBSMYQZDU2BYXUGWTLG375AA36NVSFUTKQ, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.038236492 (71h50m33.909319537s) [noise=true , address=145.79.227.31:4798 , piece_id=ZWO47DBANQT6R5PVNMZTOJOTQFJNA7B2G7ZUBE2CYWS3L3E2673A, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.038277370 (71h50m33.909360405s) [noise=true , address=145.79.227.41:4087 , piece_id=QWHQGNBWKNWTMIVMKXQWSO2SHC25WB4TCKBVWZGLCWE7YEEZ6IMA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.038628238 (71h50m33.909711273s) [noise=true , address=77.90.31.14:4662 , piece_id=2CLBCTWAXRDSH67TZQJWE2XYDRDOQGHWBIBVS4N22K6N2H2TTXPQ, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.038748908 (71h50m33.909831943s) [noise=true , address=145.79.227.11:4729 , piece_id=K6DW4OM72EOSBM55TWMTGEXB6TQFHNYPSOETN3VG4B3DXKX6WHVA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.039082382 (71h50m33.910165417s) [noise=true , address=145.79.227.44:4195 , piece_id=EOB26H2KHSHMZBB6P6533V6DYZZTQ5EWLEN7UNTGRC3NHF7UW5BQ, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.039707232 (71h50m33.910790267s) [noise=true , address=145.79.227.34:4906 , piece_id=5KZQYIQ3BJDH3H2KWVVARCB7BEFWWP5TFN5M7HTSQOV3DRL2NYXA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.040053371 (71h50m33.911136406s) [noise=true , address=145.79.227.22:4511 , piece_id=2AVUFXUWWNYHK4ZOMLQZPJMOLS7QVHFNCFUOPHILP6WBWTPPWKUA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:35.870658037 (71h50m35.741741072s) [upload=62909]: Next returned nil segment
Dec 8 15:19:35.870670711 (71h50m35.741753756s) [upload=62909]: Waiting for error group managing segments...
Dec 8 15:19:36.177531846 (71h50m36.048614882s) [noise=true , address=145.79.227.20:4439 , piece_id=LMKA26FPXNRAH5YZISRBSMYQZDU2BYXUGWTLG375AA36NVSFUTKQ, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.190430637 (71h50m36.061513683s) [noise=true , address=145.79.227.22:4510 , piece_id=RNL6C7WQFRDZ7Z5JN6J6UCHIO2WOPVTWFQC2JHROAWS3ZVEFBUEQ, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.195032932 (71h50m36.066115967s) [noise=true , address=145.79.227.28:4725 , piece_id=JOTZWCSNR6HGKYEYT75SPRHTUR7R2DSCHFCRZ32KZ3FFG5EIR5PA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.204295640 (71h50m36.075378675s) [noise=true , address=145.79.227.22:4511 , piece_id=2AVUFXUWWNYHK4ZOMLQZPJMOLS7QVHFNCFUOPHILP6WBWTPPWKUA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.213119352 (71h50m36.084202387s) [noise=true , address=145.79.227.11:4729 , piece_id=K6DW4OM72EOSBM55TWMTGEXB6TQFHNYPSOETN3VG4B3DXKX6WHVA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.236502718 (71h50m36.107585753s) [noise=true , address=145.79.227.29:4942 , piece_id=O35KF7WMNWCV5MHYHNCXT765JS5SOWEJHDH6QL4AR23Y4AUOTFAA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.269573071 (71h50m36.140656116s) [noise=true , address=145.79.227.23:4547 , piece_id=XQ5BXGZMDPKFSIYMKUNGO6YMUTQYQ37OJGGKZFU5SFD2NJG45TNQ, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.274948397 (71h50m36.146031442s) [noise=true , address=145.79.227.29:4941 , piece_id=THMVWSNTE5SGGZZY4USNEFNEKSTCJM6CBCWLWMNFUFUZA6ATO4IA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.308847598 (71h50m36.179930643s) [noise=true , address=145.79.227.34:4906 , piece_id=5KZQYIQ3BJDH3H2KWVVARCB7BEFWWP5TFN5M7HTSQOV3DRL2NYXA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.352875655 (71h50m36.22395869s) [noise=true , address=77.90.31.14:4662 , piece_id=2CLBCTWAXRDSH67TZQJWE2XYDRDOQGHWBIBVS4N22K6N2H2TTXPQ, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.894699757 (71h50m36.765782792s) [noise=true , address=145.79.227.38:4979 , piece_id=6CI4XIPXKSUI3ODW662O5XG63TMTJ3KIXDUYHQMQCTVHTLULWKPA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.928926731 (71h50m36.800009776s) [noise=true , address=145.79.227.5:4222 , piece_id=JCVF7WMKWK3JXGBTNBSIXWQPTR7LF57JTR45677SN7DUVYHP547A, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.966312090 (71h50m36.837395135s) [noise=true , address=145.79.227.22:4509 , piece_id=BRV2XH7SZW2MPO53FENX2P23ICAWZQQ2LOMNQHVPWRAQJTT4JO2Q, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.966330916 (71h50m36.837413961s) [seg_pos={0 0}, upload=62909]: Segment reached optimal threshold of 13 pieces.
Dec 8 15:19:36.966494839 (71h50m36.837577884s) [noise=true , address=145.79.227.31:4798 , piece_id=ZWO47DBANQT6R5PVNMZTOJOTQFJNA7B2G7ZUBE2CYWS3L3E2673A, seg_pos={0 0}, upload=62909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ): protocol: expected piece hash; context canceled; EOF
Dec 8 15:19:36.966501081 (71h50m36.837584116s) [noise=true , address=145.79.227.31:4798 , piece_id=ZWO47DBANQT6R5PVNMZTOJOTQFJNA7B2G7ZUBE2CYWS3L3E2673A, seg_pos={0 0}, upload=62909]: Long tail cancellation due to optimal threshold
Dec 8 15:19:36.966705750 (71h50m36.837788785s) [noise=true , address=145.79.227.44:4195 , piece_id=EOB26H2KHSHMZBB6P6533V6DYZZTQ5EWLEN7UNTGRC3NHF7UW5BQ, seg_pos={0 0}, upload=62909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF
Dec 8 15:19:36.966712583 (71h50m36.837795628s) [noise=true , address=145.79.227.44:4195 , piece_id=EOB26H2KHSHMZBB6P6533V6DYZZTQ5EWLEN7UNTGRC3NHF7UW5BQ, seg_pos={0 0}, upload=62909]: Long tail cancellation due to optimal threshold
Dec 8 15:19:36.966863240 (71h50m36.837946285s) [noise=true , address=145.79.227.41:4087 , piece_id=QWHQGNBWKNWTMIVMKXQWSO2SHC25WB4TCKBVWZGLCWE7YEEZ6IMA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF
Dec 8 15:19:36.966868980 (71h50m36.837952016s) [noise=true , address=145.79.227.41:4087 , piece_id=QWHQGNBWKNWTMIVMKXQWSO2SHC25WB4TCKBVWZGLCWE7YEEZ6IMA, seg_pos={0 0}, upload=62909]: Long tail cancellation due to optimal threshold
Dec 8 15:19:36.966885362 (71h50m36.837968397s) [seg_pos={0 0}, upload=62909]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 15:19:36.966960996 (71h50m36.838044031s) [upload=62909]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:19:36.966965855 (71h50m36.8380489s) [upload=62909]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:19:37.284578473 (71h50m37.155661519s) [upload=62909]: Done uploading segments...
Dec 8 15:19:37.284599162 (71h50m37.155682207s) [upload=62909]: Upload finished. err: <nil>
Duration: 9.06958097s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
47.967077447s trace:
https://jaeger.dev.storj.io/trace/54ba7b1e9d79a182
FAIL (2025-12-08 15:20:39 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 47.967101975s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [X..........X...X.........................................X..] duration:
8.886290336s trace:
https://jaeger.dev.storj.io/trace/5cf02331713757d4
FAIL (2025-12-08 15:20:32 UTC)
orbiter warning: context deadline exceeded
Duration: 8.886370488s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
24.884613892s trace:
https://jaeger.dev.storj.io/trace/29daaac43eca6484
FAIL (2025-12-08 15:20:29 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 24.884638879s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
10.108079221s remote:
"tcp:35.212.10.183:443" target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/4a31039498bdbd4f
FAIL (2025-12-08 15:20:18 UTC)
orbiter warning: Get "https://us1.storj.io/login": EOF
Duration: 10.108110931s
core
satellite
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [..X..X.....................................X............X.X.] duration:
28.292416087s trace:
https://jaeger.dev.storj.io/trace/3c3b9a0929954eb0
FAIL (2025-12-08 15:20:13 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 28.292456245s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.016060162s trace:
https://jaeger.dev.storj.io/trace/19f19098eeec6fec
FAIL (2025-12-08 15:20:02 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, Piece ID: THMVWSNTE5SGGZZY4USNEFNEKSTCJM6CBCWLWMNFUFUZA6ATO4IA) context deadline exceeded; piecestore: (Node ID: 1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, Piece ID: XQ5BXGZMDPKFSIYMKUNGO6YMUTQYQ37OJGGKZFU5SFD2NJG45TNQ) context deadline exceeded; piecestore: (Node ID: 12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, Piece ID: RNL6C7WQFRDZ7Z5JN6J6UCHIO2WOPVTWFQC2JHROAWS3ZVEFBUEQ) context deadline exceeded
======LOGS======
Dec 8 15:19:33.770990212 (71h50m33.642073257s) [upload=62909]: Starting upload
Dec 8 15:19:33.771048684 (71h50m33.642131729s) [upload=62909]: Done starting upload
Dec 8 15:19:33.771219048 (71h50m33.642302083s) [upload=62909]: Uploading segments...
Dec 8 15:19:33.771337443 (71h50m33.642420478s) [upload=62909]: Got next segment. Inline: false
Dec 8 15:19:33.771385996 (71h50m33.642469031s) [upload=62909]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 15:19:33.771391647 (71h50m33.642474692s) [upload=62909]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 15:19:34.033637444 (71h50m33.904720489s) [seg_pos={0 0}, upload=62909]: Begin upload segment...
Dec 8 15:19:34.033686949 (71h50m33.904769984s) [seg_pos={0 0}, upload=62909]: Done begin upload segment.
Dec 8 15:19:34.033718579 (71h50m33.904801624s) [noise=true , address=145.79.227.29:4941 , piece_id=THMVWSNTE5SGGZZY4USNEFNEKSTCJM6CBCWLWMNFUFUZA6ATO4IA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.033735481 (71h50m33.904818526s) [noise=true , address=145.79.227.28:4725 , piece_id=JOTZWCSNR6HGKYEYT75SPRHTUR7R2DSCHFCRZ32KZ3FFG5EIR5PA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.035223314 (71h50m33.906306349s) [noise=true , address=145.79.227.23:4547 , piece_id=XQ5BXGZMDPKFSIYMKUNGO6YMUTQYQ37OJGGKZFU5SFD2NJG45TNQ, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.035531321 (71h50m33.906614356s) [noise=true , address=145.79.227.29:4942 , piece_id=O35KF7WMNWCV5MHYHNCXT765JS5SOWEJHDH6QL4AR23Y4AUOTFAA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.035867982 (71h50m33.906951027s) [noise=true , address=145.79.227.22:4509 , piece_id=BRV2XH7SZW2MPO53FENX2P23ICAWZQQ2LOMNQHVPWRAQJTT4JO2Q, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.037062737 (71h50m33.908145782s) [noise=true , address=145.79.227.5:4222 , piece_id=JCVF7WMKWK3JXGBTNBSIXWQPTR7LF57JTR45677SN7DUVYHP547A, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.037425418 (71h50m33.908508463s) [noise=true , address=145.79.227.38:4979 , piece_id=6CI4XIPXKSUI3ODW662O5XG63TMTJ3KIXDUYHQMQCTVHTLULWKPA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.037777228 (71h50m33.908860263s) [noise=true , address=145.79.227.22:4510 , piece_id=RNL6C7WQFRDZ7Z5JN6J6UCHIO2WOPVTWFQC2JHROAWS3ZVEFBUEQ, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.038199451 (71h50m33.909282487s) [noise=true , address=145.79.227.20:4439 , piece_id=LMKA26FPXNRAH5YZISRBSMYQZDU2BYXUGWTLG375AA36NVSFUTKQ, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.038236492 (71h50m33.909319537s) [noise=true , address=145.79.227.31:4798 , piece_id=ZWO47DBANQT6R5PVNMZTOJOTQFJNA7B2G7ZUBE2CYWS3L3E2673A, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.038277370 (71h50m33.909360405s) [noise=true , address=145.79.227.41:4087 , piece_id=QWHQGNBWKNWTMIVMKXQWSO2SHC25WB4TCKBVWZGLCWE7YEEZ6IMA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.038628238 (71h50m33.909711273s) [noise=true , address=77.90.31.14:4662 , piece_id=2CLBCTWAXRDSH67TZQJWE2XYDRDOQGHWBIBVS4N22K6N2H2TTXPQ, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.038748908 (71h50m33.909831943s) [noise=true , address=145.79.227.11:4729 , piece_id=K6DW4OM72EOSBM55TWMTGEXB6TQFHNYPSOETN3VG4B3DXKX6WHVA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.039082382 (71h50m33.910165417s) [noise=true , address=145.79.227.44:4195 , piece_id=EOB26H2KHSHMZBB6P6533V6DYZZTQ5EWLEN7UNTGRC3NHF7UW5BQ, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.039707232 (71h50m33.910790267s) [noise=true , address=145.79.227.34:4906 , piece_id=5KZQYIQ3BJDH3H2KWVVARCB7BEFWWP5TFN5M7HTSQOV3DRL2NYXA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:34.040053371 (71h50m33.911136406s) [noise=true , address=145.79.227.22:4511 , piece_id=2AVUFXUWWNYHK4ZOMLQZPJMOLS7QVHFNCFUOPHILP6WBWTPPWKUA, seg_pos={0 0}, upload=62909]: Uploading piece...
Dec 8 15:19:35.870658037 (71h50m35.741741072s) [upload=62909]: Next returned nil segment
Dec 8 15:19:35.870670711 (71h50m35.741753756s) [upload=62909]: Waiting for error group managing segments...
Dec 8 15:19:36.177531846 (71h50m36.048614882s) [noise=true , address=145.79.227.20:4439 , piece_id=LMKA26FPXNRAH5YZISRBSMYQZDU2BYXUGWTLG375AA36NVSFUTKQ, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.190430637 (71h50m36.061513683s) [noise=true , address=145.79.227.22:4510 , piece_id=RNL6C7WQFRDZ7Z5JN6J6UCHIO2WOPVTWFQC2JHROAWS3ZVEFBUEQ, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.195032932 (71h50m36.066115967s) [noise=true , address=145.79.227.28:4725 , piece_id=JOTZWCSNR6HGKYEYT75SPRHTUR7R2DSCHFCRZ32KZ3FFG5EIR5PA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.204295640 (71h50m36.075378675s) [noise=true , address=145.79.227.22:4511 , piece_id=2AVUFXUWWNYHK4ZOMLQZPJMOLS7QVHFNCFUOPHILP6WBWTPPWKUA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.213119352 (71h50m36.084202387s) [noise=true , address=145.79.227.11:4729 , piece_id=K6DW4OM72EOSBM55TWMTGEXB6TQFHNYPSOETN3VG4B3DXKX6WHVA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.236502718 (71h50m36.107585753s) [noise=true , address=145.79.227.29:4942 , piece_id=O35KF7WMNWCV5MHYHNCXT765JS5SOWEJHDH6QL4AR23Y4AUOTFAA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.269573071 (71h50m36.140656116s) [noise=true , address=145.79.227.23:4547 , piece_id=XQ5BXGZMDPKFSIYMKUNGO6YMUTQYQ37OJGGKZFU5SFD2NJG45TNQ, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.274948397 (71h50m36.146031442s) [noise=true , address=145.79.227.29:4941 , piece_id=THMVWSNTE5SGGZZY4USNEFNEKSTCJM6CBCWLWMNFUFUZA6ATO4IA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.308847598 (71h50m36.179930643s) [noise=true , address=145.79.227.34:4906 , piece_id=5KZQYIQ3BJDH3H2KWVVARCB7BEFWWP5TFN5M7HTSQOV3DRL2NYXA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.352875655 (71h50m36.22395869s) [noise=true , address=77.90.31.14:4662 , piece_id=2CLBCTWAXRDSH67TZQJWE2XYDRDOQGHWBIBVS4N22K6N2H2TTXPQ, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.894699757 (71h50m36.765782792s) [noise=true , address=145.79.227.38:4979 , piece_id=6CI4XIPXKSUI3ODW662O5XG63TMTJ3KIXDUYHQMQCTVHTLULWKPA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.928926731 (71h50m36.800009776s) [noise=true , address=145.79.227.5:4222 , piece_id=JCVF7WMKWK3JXGBTNBSIXWQPTR7LF57JTR45677SN7DUVYHP547A, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.966312090 (71h50m36.837395135s) [noise=true , address=145.79.227.22:4509 , piece_id=BRV2XH7SZW2MPO53FENX2P23ICAWZQQ2LOMNQHVPWRAQJTT4JO2Q, seg_pos={0 0}, upload=62909]: Done uploading piece. err: <nil>
Dec 8 15:19:36.966330916 (71h50m36.837413961s) [seg_pos={0 0}, upload=62909]: Segment reached optimal threshold of 13 pieces.
Dec 8 15:19:36.966494839 (71h50m36.837577884s) [noise=true , address=145.79.227.31:4798 , piece_id=ZWO47DBANQT6R5PVNMZTOJOTQFJNA7B2G7ZUBE2CYWS3L3E2673A, seg_pos={0 0}, upload=62909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ): protocol: expected piece hash; context canceled; EOF
Dec 8 15:19:36.966501081 (71h50m36.837584116s) [noise=true , address=145.79.227.31:4798 , piece_id=ZWO47DBANQT6R5PVNMZTOJOTQFJNA7B2G7ZUBE2CYWS3L3E2673A, seg_pos={0 0}, upload=62909]: Long tail cancellation due to optimal threshold
Dec 8 15:19:36.966705750 (71h50m36.837788785s) [noise=true , address=145.79.227.44:4195 , piece_id=EOB26H2KHSHMZBB6P6533V6DYZZTQ5EWLEN7UNTGRC3NHF7UW5BQ, seg_pos={0 0}, upload=62909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF
Dec 8 15:19:36.966712583 (71h50m36.837795628s) [noise=true , address=145.79.227.44:4195 , piece_id=EOB26H2KHSHMZBB6P6533V6DYZZTQ5EWLEN7UNTGRC3NHF7UW5BQ, seg_pos={0 0}, upload=62909]: Long tail cancellation due to optimal threshold
Dec 8 15:19:36.966863240 (71h50m36.837946285s) [noise=true , address=145.79.227.41:4087 , piece_id=QWHQGNBWKNWTMIVMKXQWSO2SHC25WB4TCKBVWZGLCWE7YEEZ6IMA, seg_pos={0 0}, upload=62909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF
Dec 8 15:19:36.966868980 (71h50m36.837952016s) [noise=true , address=145.79.227.41:4087 , piece_id=QWHQGNBWKNWTMIVMKXQWSO2SHC25WB4TCKBVWZGLCWE7YEEZ6IMA, seg_pos={0 0}, upload=62909]: Long tail cancellation due to optimal threshold
Dec 8 15:19:36.966885362 (71h50m36.837968397s) [seg_pos={0 0}, upload=62909]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 15:19:36.966960996 (71h50m36.838044031s) [upload=62909]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:19:36.966965855 (71h50m36.8380489s) [upload=62909]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:19:37.284578473 (71h50m37.155661519s) [upload=62909]: Done uploading segments...
Dec 8 15:19:37.284599162 (71h50m37.155682207s) [upload=62909]: Upload finished. err: <nil>
Duration: 30.016089188s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
10.918094657s trace:
https://jaeger.dev.storj.io/trace/153706e03a29fd88
FAIL (2025-12-08 15:19:59 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 10.918121268s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [.X..........X...X.........................................X.] duration:
20.2115659s trace:
https://jaeger.dev.storj.io/trace/5cf02331713757d4
FAIL (2025-12-08 15:19:33 UTC)
orbiter warning: context deadline exceeded
Duration: 20.211585418s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
51 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XXXXX.XXX..X....X] duration:
5.000840668s trace:
https://jaeger.dev.storj.io/trace/49ccacaae559446c
FAIL (2025-12-08 15:19:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.00086255s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................] duration:
15.051686661s trace:
https://jaeger.dev.storj.io/trace/29daaac43eca6484
FAIL (2025-12-08 15:19:25 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 15.051715445s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [...X..X.....................................X............X.X] duration:
30.001377223s trace:
https://jaeger.dev.storj.io/trace/3c3b9a0929954eb0
FAIL (2025-12-08 15:19:20 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001403355s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
51 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XXXXX.XXX..X....] duration:
4.3780624s trace:
https://jaeger.dev.storj.io/trace/5d8a68eca0605620
FAIL (2025-12-08 15:19:12 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.378095452s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [..X..........X...X.........................................X] duration:
1m0.001628164s trace:
https://jaeger.dev.storj.io/trace/5cf02331713757d4
FAIL (2025-12-08 15:19:12 UTC)
orbiter warning: context deadline exceeded
Duration: 1m0.001649163s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
687.786712ms remote:
"tcp:35.212.10.183:443" target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/4a31039498bdbd4f
FAIL (2025-12-08 15:19:02 UTC)
orbiter warning: Get "https://us1.storj.io/login": EOF
Duration: 687.847357ms
core
satellite
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
19.19503976s trace:
https://jaeger.dev.storj.io/trace/153706e03a29fd88
FAIL (2025-12-08 15:18:56 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 19.195064817s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
20.301404026s trace:
https://jaeger.dev.storj.io/trace/29daaac43eca6484
FAIL (2025-12-08 15:18:30 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 20.301442431s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
26.445011252s trace:
https://jaeger.dev.storj.io/trace/153706e03a29fd88
FAIL (2025-12-08 15:18:26 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 26.445053633s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
52 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XXXXX.XXX..X...] duration:
5.001214159s trace:
https://jaeger.dev.storj.io/trace/5d8a68eca0605620
FAIL (2025-12-08 15:18:09 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001243365s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
703.808803ms remote:
"tcp:35.212.10.183:443" target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/4a31039498bdbd4f
FAIL (2025-12-08 15:17:57 UTC)
orbiter warning: Get "https://us1.storj.io/login": EOF
Duration: 703.835953ms
core
satellite
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...X..........X...X.........................................] duration:
10.135752883s trace:
https://jaeger.dev.storj.io/trace/9d8451999396f06
FAIL (2025-12-08 15:17:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 10.135778403s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [....X..X.....................................X............X.] duration:
14.412629877s trace:
https://jaeger.dev.storj.io/trace/46b65737f4e7c8bb
FAIL (2025-12-08 15:17:55 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 14.412654052s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
11.643460694s trace:
https://jaeger.dev.storj.io/trace/29daaac43eca6484
FAIL (2025-12-08 15:17:13 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 11.643487295s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
53 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XXXXX.XXX..X..] duration:
4.904005076s trace:
https://jaeger.dev.storj.io/trace/5d8a68eca0605620
FAIL (2025-12-08 15:17:09 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.904032368s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [....X..........X...X........................................] duration:
22.543824799s trace:
https://jaeger.dev.storj.io/trace/9d8451999396f06
FAIL (2025-12-08 15:17:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 22.543866277s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
970.439959ms remote:
"tcp:35.212.10.183:443" target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/4a31039498bdbd4f
FAIL (2025-12-08 15:17:02 UTC)
orbiter warning: Get "https://us1.storj.io/login": EOF
Duration: 970.484484ms
core
satellite
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.....X..X.....................................X............X] duration:
30.001143886s trace:
https://jaeger.dev.storj.io/trace/46b65737f4e7c8bb
FAIL (2025-12-08 15:16:57 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001171048s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
7.532559226s trace:
https://jaeger.dev.storj.io/trace/153706e03a29fd88
FAIL (2025-12-08 15:16:54 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 7.532585476s
edge
gateway
us-select-1
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
1.004431496s remote:
"tcp:35.212.10.183:443" target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/4a31039498bdbd4f
FAIL (2025-12-08 15:16:47 UTC)
orbiter warning: Get "https://us1.storj.io/login": EOF
Duration: 1.004457025s
core
satellite
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
14.17308128s trace:
https://jaeger.dev.storj.io/trace/29daaac43eca6484
FAIL (2025-12-08 15:16:24 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 14.173100606s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [......X..X.....................................X............] duration:
20.691621656s trace:
https://jaeger.dev.storj.io/trace/dbd17afbd215569
FAIL (2025-12-08 15:16:10 UTC)
orbiter warning: context deadline exceeded
Duration: 20.691689566s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
17.479870515s trace:
https://jaeger.dev.storj.io/trace/29daaac43eca6484
FAIL (2025-12-08 15:16:09 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 17.479903158s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
21.294664748s trace:
https://jaeger.dev.storj.io/trace/153706e03a29fd88
FAIL (2025-12-08 15:16:08 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 21.294824282s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.....X..........X...X.......................................] duration:
11.608512326s trace:
https://jaeger.dev.storj.io/trace/9d8451999396f06
FAIL (2025-12-08 15:16:03 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 11.608540701s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
53 out of 60 is failed (threshold=6) [.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XXXXX.XXX..X.] duration:
3.463397414s trace:
https://jaeger.dev.storj.io/trace/5d8a68eca0605620
FAIL (2025-12-08 15:16:00 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.463428102s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
54 out of 60 is failed (threshold=6) [X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XXXXX.XXX..X] duration:
5.001169295s trace:
https://jaeger.dev.storj.io/trace/5d8a68eca0605620
FAIL (2025-12-08 15:15:43 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001190304s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
1.053875979s remote:
"tcp:35.212.10.183:443" target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/4a31039498bdbd4f
FAIL (2025-12-08 15:15:34 UTC)
orbiter warning: Get "https://us1.storj.io/login": EOF
Duration: 1.053895726s
core
satellite
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
10.751964265s trace:
https://jaeger.dev.storj.io/trace/153706e03a29fd88
FAIL (2025-12-08 15:15:30 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 10.751989766s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
54 out of 60 is failed (threshold=6) [XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XXXXX.XXX..] duration:
5.000772447s trace:
https://jaeger.dev.storj.io/trace/7708c6d0483d2c95
FAIL (2025-12-08 15:15:24 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000797626s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [X.....X..........X...X......................................] duration:
6.542927608s trace:
https://jaeger.dev.storj.io/trace/9d8451999396f06
FAIL (2025-12-08 15:15:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.542959217s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
954.562705ms remote:
"tcp:35.212.10.183:443" target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/4a31039498bdbd4f
FAIL (2025-12-08 15:15:14 UTC)
orbiter warning: Get "https://us1.storj.io/login": EOF
Duration: 954.589757ms
core
satellite
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.......X..X.....................................X...........] duration:
30.001186548s trace:
https://jaeger.dev.storj.io/trace/dbd17afbd215569
FAIL (2025-12-08 15:15:13 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001213269s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................] duration:
21.749090844s trace:
https://jaeger.dev.storj.io/trace/29daaac43eca6484
FAIL (2025-12-08 15:15:09 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 21.749119147s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [.X.....X..........X...X.....................................] duration:
11.170463735s trace:
https://jaeger.dev.storj.io/trace/9d8451999396f06
FAIL (2025-12-08 15:14:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 11.170493993s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
55 out of 60 is failed (threshold=6) [XXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XXXXX.XXX.] duration:
5.001263216s trace:
https://jaeger.dev.storj.io/trace/7708c6d0483d2c95
FAIL (2025-12-08 15:14:37 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001293603s
core
uplink
us-select-1
us1-placement-20
|