Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...........................X..............................X.] duration:
5.165331036s trace:
https://jaeger.dev.storj.io/trace/1a72c3cee1ba60f
FAIL (2025-12-09 15:06:21 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 5.165354601s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
5.851037783s trace:
https://jaeger.dev.storj.io/trace/61c71b63946688f
FAIL (2025-12-09 15:06:19 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 5.851061027s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
25 out of 60 is failed (threshold=6) [..X.....X....X..XXX.X..X...X.....X.X.XXXXXXX.X....X..X.XXXX.] duration:
5.000981662s trace:
https://jaeger.dev.storj.io/trace/d42a61cceaf6758
FAIL (2025-12-09 15:06:18 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001010097s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
2 out of 60 is failed (threshold=3) [............................X..............................X] duration:
15m3.698769976s trace:
https://jaeger.dev.storj.io/trace/1a72c3cee1ba60f
FAIL (2025-12-09 15:06:16 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 15m3.698785144s
core
uplink
eu1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
8.926636922s trace:
https://jaeger.dev.storj.io/trace/79b4d9ea215ba812
FAIL (2025-12-09 15:06:14 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 8.926671038s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [..XX...X....X........X.XX.XX.....XXX.......X......X....X..X.] duration:
24.638533625s trace:
https://jaeger.dev.storj.io/trace/b94db2c741a9aad
FAIL (2025-12-09 15:06:03 UTC)
orbiter warning: context deadline exceeded
Duration: 24.638567531s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
23.900467447s trace:
https://jaeger.dev.storj.io/trace/79b4d9ea215ba812
FAIL (2025-12-09 15:05:56 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 23.900484008s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
5.771293963s trace:
https://jaeger.dev.storj.io/trace/61c71b63946688f
FAIL (2025-12-09 15:05:35 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 5.771324392s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
25 out of 60 is failed (threshold=6) [...X.....X....X..XXX.X..X...X.....X.X.XXXXXXX.X....X..X.XXXX] duration:
5.00150554s trace:
https://jaeger.dev.storj.io/trace/d42a61cceaf6758
FAIL (2025-12-09 15:05:34 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001530336s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.................X.......................................X..] duration:
6.320672043s trace:
https://jaeger.dev.storj.io/trace/1f9f83d8cc633aae
FAIL (2025-12-09 15:05:24 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 9 14:59:32.716918810 (95h30m32.588001855s) [upload=83259]: Starting upload
Dec 9 14:59:32.716988723 (95h30m32.588071758s) [upload=83259]: Done starting upload
Dec 9 14:59:32.717137536 (95h30m32.588220581s) [upload=83259]: Uploading segments...
Dec 9 14:59:32.717286900 (95h30m32.588369935s) [upload=83259]: Got next segment. Inline: false
Dec 9 14:59:32.717341534 (95h30m32.58842458s) [upload=83259]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 9 14:59:32.717347646 (95h30m32.588430691s) [upload=83259]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 9 14:59:32.975580773 (95h30m32.846663828s) [seg_pos={0 0}, upload=83259]: Begin upload segment...
Dec 9 14:59:32.975640838 (95h30m32.846723883s) [seg_pos={0 0}, upload=83259]: Done begin upload segment.
Dec 9 14:59:32.975718646 (95h30m32.846801701s) [noise=true , address=145.79.227.16:4910 , piece_id=YZH37SYHZCT2T6LZDS2UMDMCLNPW3SUUUX6BZKEQYUKZ3H2SU3AQ, seg_pos={0 0}, upload=83259]: Uploading piece...
Dec 9 14:59:32.975774743 (95h30m32.846857788s) [noise=true , address=145.79.227.34:4906 , piece_id=FNIAMKKNCMYZEIE3VV43GAZLOMQA54NKUOL74VOJCS4WFIBRFG2A, seg_pos={0 0}, upload=83259]: Uploading piece...
Dec 9 14:59:32.976391548 (95h30m32.847474593s) [noise=true , address=145.79.227.33:4869 , piece_id=UK3COALVJUOZOSHS5WQHPZAGDP73D27ZE47MGESVOZB4JHDUUBRA, seg_pos={0 0}, upload=83259]: Uploading piece...
Dec 9 14:59:32.976648978 (95h30m32.847732023s) [noise=true , address=145.79.227.28:4726 , piece_id=ERLWH62KS663YXXYEKHY2ZLLJDYXBBTPLL7BD5UDAFH6DG2H5IUA, seg_pos={0 0}, upload=83259]: Uploading piece...
Dec 9 14:59:32.977223793 (95h30m32.848306838s) [noise=true , address=145.79.227.27:4690 , piece_id=5FAALAMRZZYSWKPEJGAQJFQGSTH5L7SSEIHZXIL57JTZJCPOR66Q, seg_pos={0 0}, upload=83259]: Uploading piece...
Dec 9 14:59:32.977895982 (95h30m32.848979057s) [noise=true , address=145.79.227.43:4157 , piece_id=ZI5ZA5MIRSWF4XPWM5D7UCMOJDIG3W3HRKGM5DYKCDJV7OONS6QA, seg_pos={0 0}, upload=83259]: Uploading piece...
Dec 9 14:59:32.978304240 (95h30m32.849387285s) [noise=true , address=77.90.31.15:4699 , piece_id=HVPBJSCEG3BFPDA6UZ65XRZKDGEZ6Z6YEREOAUHQ66Y46GX26BIA, seg_pos={0 0}, upload=83259]: Uploading piece...
Dec 9 14:59:32.978738136 (95h30m32.849821181s) [noise=true , address=145.79.227.26:4655 , piece_id=HAAP2ZXXNUBHE2Q7D5BHINZPRVAWWHPGQ4SU4I356YYAJNVZN5HA, seg_pos={0 0}, upload=83259]: Uploading piece...
Dec 9 14:59:32.978777981 (95h30m32.849861016s) [noise=true , address=145.79.227.38:4978 , piece_id=LJ4M45R4QMTU7COWVHWLC4LDJ5JXJ6HSD7HEGHQSAFNKP44R72TQ, seg_pos={0 0}, upload=83259]: Uploading piece...
Dec 9 14:59:32.979321667 (95h30m32.850404712s) [noise=true , address=145.79.227.29:4942 , piece_id=WH7PWH55KGNUBWQDHD3QRFJM6Q6PHA6JBUTTV5R5REIADMQORYNA, seg_pos={0 0}, upload=83259]: Uploading piece...
Dec 9 14:59:32.979840495 (95h30m32.85092354s) [noise=true , address=145.79.227.42:4121 , piece_id=KAPV7DTW5H7ETUCRMVMJUCGTA5ILRBJDB5AOOUAWONAUSWNWXSSQ, seg_pos={0 0}, upload=83259]: Uploading piece...
Dec 9 14:59:32.979932580 (95h30m32.851015625s) [noise=true , address=145.79.227.21:4474 , piece_id=FCSLF4NIRCI6QVRESZP5UY2FK2FNEH3TJUJZTI5XKYBCGDPCGTDQ, seg_pos={0 0}, upload=83259]: Uploading piece...
Dec 9 14:59:32.980394961 (95h30m32.851477996s) [noise=true , address=145.79.227.16:4912 , piece_id=XOZ5J3SLEFI7H3AZRFHC5HK2OBVROEOREMY57WJEWYBGVL6QADYA, seg_pos={0 0}, upload=83259]: Uploading piece...
Dec 9 14:59:32.980744797 (95h30m32.851827832s) [noise=true , address=145.79.227.24:4582 , piece_id=UGNKCJAA6PJBE3CWC46ZVVSBZIRTJ4C3L327LLAROK4WMU72MMXQ, seg_pos={0 0}, upload=83259]: Uploading piece...
Dec 9 14:59:32.982809799 (95h30m32.853892844s) [noise=true , address=45.91.116.13:4153 , piece_id=TPZSU5MDFBQSXFM7WRAWFTRHVOLI3TXTZ5MBZDZGIBHSCTOJJEZQ, seg_pos={0 0}, upload=83259]: Uploading piece...
Dec 9 14:59:32.983317917 (95h30m32.854400992s) [noise=true , address=145.79.227.39:4014 , piece_id=MRO2KIUUPJGX6NGQNVY6VDMCIFECHHL3FFJ7OALNM2TFE4YUCMXQ, seg_pos={0 0}, upload=83259]: Uploading piece...
Dec 9 14:59:34.680921451 (95h30m34.552004486s) [upload=83259]: Next returned nil segment
Dec 9 14:59:34.680943153 (95h30m34.552026198s) [upload=83259]: Waiting for error group managing segments...
Dec 9 14:59:34.985677890 (95h30m34.856760925s) [noise=true , address=145.79.227.21:4474 , piece_id=FCSLF4NIRCI6QVRESZP5UY2FK2FNEH3TJUJZTI5XKYBCGDPCGTDQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil>
Dec 9 14:59:34.986532547 (95h30m34.857615592s) [noise=true , address=145.79.227.42:4121 , piece_id=KAPV7DTW5H7ETUCRMVMJUCGTA5ILRBJDB5AOOUAWONAUSWNWXSSQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil>
Dec 9 14:59:34.991365901 (95h30m34.862448936s) [noise=true , address=145.79.227.16:4912 , piece_id=XOZ5J3SLEFI7H3AZRFHC5HK2OBVROEOREMY57WJEWYBGVL6QADYA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil>
Dec 9 14:59:34.996089796 (95h30m34.867172831s) [noise=true , address=145.79.227.43:4157 , piece_id=ZI5ZA5MIRSWF4XPWM5D7UCMOJDIG3W3HRKGM5DYKCDJV7OONS6QA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil>
Dec 9 14:59:34.999561678 (95h30m34.870644713s) [noise=true , address=145.79.227.27:4690 , piece_id=5FAALAMRZZYSWKPEJGAQJFQGSTH5L7SSEIHZXIL57JTZJCPOR66Q, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil>
Dec 9 14:59:35.001337950 (95h30m34.872420985s) [noise=true , address=145.79.227.26:4655 , piece_id=HAAP2ZXXNUBHE2Q7D5BHINZPRVAWWHPGQ4SU4I356YYAJNVZN5HA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil>
Dec 9 14:59:35.001503816 (95h30m34.872586881s) [noise=true , address=145.79.227.28:4726 , piece_id=ERLWH62KS663YXXYEKHY2ZLLJDYXBBTPLL7BD5UDAFH6DG2H5IUA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil>
Dec 9 14:59:35.012234980 (95h30m34.883318025s) [noise=true , address=145.79.227.38:4978 , piece_id=LJ4M45R4QMTU7COWVHWLC4LDJ5JXJ6HSD7HEGHQSAFNKP44R72TQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil>
Dec 9 14:59:35.012262382 (95h30m34.883345437s) [noise=true , address=145.79.227.16:4910 , piece_id=YZH37SYHZCT2T6LZDS2UMDMCLNPW3SUUUX6BZKEQYUKZ3H2SU3AQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil>
Dec 9 14:59:35.015363096 (95h30m34.886446131s) [noise=true , address=145.79.227.33:4869 , piece_id=UK3COALVJUOZOSHS5WQHPZAGDP73D27ZE47MGESVOZB4JHDUUBRA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil>
Dec 9 14:59:35.016246058 (95h30m34.887329103s) [noise=true , address=45.91.116.13:4153 , piece_id=TPZSU5MDFBQSXFM7WRAWFTRHVOLI3TXTZ5MBZDZGIBHSCTOJJEZQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil>
Dec 9 14:59:35.076907590 (95h30m34.947990635s) [noise=true , address=145.79.227.24:4582 , piece_id=UGNKCJAA6PJBE3CWC46ZVVSBZIRTJ4C3L327LLAROK4WMU72MMXQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil>
Dec 9 14:59:35.085585104 (95h30m34.956668149s) [noise=true , address=145.79.227.39:4014 , piece_id=MRO2KIUUPJGX6NGQNVY6VDMCIFECHHL3FFJ7OALNM2TFE4YUCMXQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil>
Dec 9 14:59:35.085609681 (95h30m34.956692726s) [seg_pos={0 0}, upload=83259]: Segment reached optimal threshold of 13 pieces.
Dec 9 14:59:35.085857443 (95h30m34.956940488s) [noise=true , address=145.79.227.29:4942 , piece_id=WH7PWH55KGNUBWQDHD3QRFJM6Q6PHA6JBUTTV5R5REIADMQORYNA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF
Dec 9 14:59:35.085865989 (95h30m34.956949034s) [noise=true , address=145.79.227.29:4942 , piece_id=WH7PWH55KGNUBWQDHD3QRFJM6Q6PHA6JBUTTV5R5REIADMQORYNA, seg_pos={0 0}, upload=83259]: Long tail cancellation due to optimal threshold
Dec 9 14:59:35.086179496 (95h30m34.957262581s) [noise=true , address=145.79.227.34:4906 , piece_id=FNIAMKKNCMYZEIE3VV43GAZLOMQA54NKUOL74VOJCS4WFIBRFG2A, seg_pos={0 0}, upload=83259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF
Dec 9 14:59:35.086188963 (95h30m34.957271999s) [noise=true , address=145.79.227.34:4906 , piece_id=FNIAMKKNCMYZEIE3VV43GAZLOMQA54NKUOL74VOJCS4WFIBRFG2A, seg_pos={0 0}, upload=83259]: Long tail cancellation due to optimal threshold
Dec 9 14:59:35.086341084 (95h30m34.957424129s) [noise=true , address=77.90.31.15:4699 , piece_id=HVPBJSCEG3BFPDA6UZ65XRZKDGEZ6Z6YEREOAUHQ66Y46GX26BIA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: write tcp 188.166.225.170:45542->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:45542->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45542->77.90.31.15:4699: use of closed network connection
Dec 9 14:59:35.086348858 (95h30m34.957431903s) [noise=true , address=77.90.31.15:4699 , piece_id=HVPBJSCEG3BFPDA6UZ65XRZKDGEZ6Z6YEREOAUHQ66Y46GX26BIA, seg_pos={0 0}, upload=83259]: Long tail cancellation due to optimal threshold
Dec 9 14:59:35.086370269 (95h30m34.957453315s) [seg_pos={0 0}, upload=83259]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 9 14:59:35.086452486 (95h30m34.957535531s) [upload=83259]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 9 14:59:35.086459168 (95h30m34.957542203s) [upload=83259]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 9 15:04:20.015922420 (95h35m19.887005475s) [upload=83259]: Done uploading segments...
Dec 9 15:04:20.015991150 (95h35m19.887074196s) [upload=83259]: Upload finished. err: metaclient: context deadline exceeded
Duration: 6.320698523s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [X..XX...X....X........X.XX.XX.....XXX.......X......X....X..X] duration:
30.000963926s trace:
https://jaeger.dev.storj.io/trace/b94db2c741a9aad
FAIL (2025-12-09 15:05:24 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001000295s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
24 out of 60 is failed (threshold=6) [....X.....X....X..XXX.X..X...X.....X.X.XXXXXXX.X....X..X.XXX] duration:
5.001275697s trace:
https://jaeger.dev.storj.io/trace/6d15355a89aeb090
FAIL (2025-12-09 15:05:07 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001300725s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
13.998680666s trace:
https://jaeger.dev.storj.io/trace/79b4d9ea215ba812
FAIL (2025-12-09 15:05:02 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 13.998748205s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
2 out of 60 is failed (threshold=3) [X...........X...............................................] duration:
5.735174695s trace:
https://jaeger.dev.storj.io/trace/61c71b63946688f
FAIL (2025-12-09 15:05:01 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 5.735206957s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
23 out of 60 is failed (threshold=6) [.....X.....X....X..XXX.X..X...X.....X.X.XXXXXXX.X....X..X.XX] duration:
5.001065191s trace:
https://jaeger.dev.storj.io/trace/6ed62da010cb292b
FAIL (2025-12-09 15:04:38 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001158819s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..................X.......................................X.] duration:
6.139038001s trace:
https://jaeger.dev.storj.io/trace/1f9f83d8cc633aae
FAIL (2025-12-09 15:04:26 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 9 14:59:32.716918810 (95h30m32.588001855s) [upload=83259]: Starting upload
Dec 9 14:59:32.716988723 (95h30m32.588071758s) [upload=83259]: Done starting upload
Dec 9 14:59:32.717137536 (95h30m32.588220581s) [upload=83259]: Uploading segments...
Dec 9 14:59:32.717286900 (95h30m32.588369935s) [upload=83259]: Got next segment. Inline: false
Dec 9 14:59:32.717341534 (95h30m32.58842458s) [upload=83259]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 9 14:59:32.717347646 (95h30m32.588430691s) [upload=83259]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 9 14:59:32.975580773 (95h30m32.846663828s) [seg_pos={0 0}, upload=83259]: Begin upload segment...
Dec 9 14:59:32.975640838 (95h30m32.846723883s) [seg_pos={0 0}, upload=83259]: Done begin upload segment.
Dec 9 14:59:32.975718646 (95h30m32.846801701s) [noise=true , address=145.79.227.16:4910 , piece_id=YZH37SYHZCT2T6LZDS2UMDMCLNPW3SUUUX6BZKEQYUKZ3H2SU3AQ, seg_pos={0 0}, upload=83259]: Uploading piece...
Dec 9 14:59:32.975774743 (95h30m32.846857788s) [noise=true , address=145.79.227.34:4906 , piece_id=FNIAMKKNCMYZEIE3VV43GAZLOMQA54NKUOL74VOJCS4WFIBRFG2A, seg_pos={0 0}, upload=83259]: Uploading piece...
Dec 9 14:59:32.976391548 (95h30m32.847474593s) [noise=true , address=145.79.227.33:4869 , piece_id=UK3COALVJUOZOSHS5WQHPZAGDP73D27ZE47MGESVOZB4JHDUUBRA, seg_pos={0 0}, upload=83259]: Uploading piece...
Dec 9 14:59:32.976648978 (95h30m32.847732023s) [noise=true , address=145.79.227.28:4726 , piece_id=ERLWH62KS663YXXYEKHY2ZLLJDYXBBTPLL7BD5UDAFH6DG2H5IUA, seg_pos={0 0}, upload=83259]: Uploading piece...
Dec 9 14:59:32.977223793 (95h30m32.848306838s) [noise=true , address=145.79.227.27:4690 , piece_id=5FAALAMRZZYSWKPEJGAQJFQGSTH5L7SSEIHZXIL57JTZJCPOR66Q, seg_pos={0 0}, upload=83259]: Uploading piece...
Dec 9 14:59:32.977895982 (95h30m32.848979057s) [noise=true , address=145.79.227.43:4157 , piece_id=ZI5ZA5MIRSWF4XPWM5D7UCMOJDIG3W3HRKGM5DYKCDJV7OONS6QA, seg_pos={0 0}, upload=83259]: Uploading piece...
Dec 9 14:59:32.978304240 (95h30m32.849387285s) [noise=true , address=77.90.31.15:4699 , piece_id=HVPBJSCEG3BFPDA6UZ65XRZKDGEZ6Z6YEREOAUHQ66Y46GX26BIA, seg_pos={0 0}, upload=83259]: Uploading piece...
Dec 9 14:59:32.978738136 (95h30m32.849821181s) [noise=true , address=145.79.227.26:4655 , piece_id=HAAP2ZXXNUBHE2Q7D5BHINZPRVAWWHPGQ4SU4I356YYAJNVZN5HA, seg_pos={0 0}, upload=83259]: Uploading piece...
Dec 9 14:59:32.978777981 (95h30m32.849861016s) [noise=true , address=145.79.227.38:4978 , piece_id=LJ4M45R4QMTU7COWVHWLC4LDJ5JXJ6HSD7HEGHQSAFNKP44R72TQ, seg_pos={0 0}, upload=83259]: Uploading piece...
Dec 9 14:59:32.979321667 (95h30m32.850404712s) [noise=true , address=145.79.227.29:4942 , piece_id=WH7PWH55KGNUBWQDHD3QRFJM6Q6PHA6JBUTTV5R5REIADMQORYNA, seg_pos={0 0}, upload=83259]: Uploading piece...
Dec 9 14:59:32.979840495 (95h30m32.85092354s) [noise=true , address=145.79.227.42:4121 , piece_id=KAPV7DTW5H7ETUCRMVMJUCGTA5ILRBJDB5AOOUAWONAUSWNWXSSQ, seg_pos={0 0}, upload=83259]: Uploading piece...
Dec 9 14:59:32.979932580 (95h30m32.851015625s) [noise=true , address=145.79.227.21:4474 , piece_id=FCSLF4NIRCI6QVRESZP5UY2FK2FNEH3TJUJZTI5XKYBCGDPCGTDQ, seg_pos={0 0}, upload=83259]: Uploading piece...
Dec 9 14:59:32.980394961 (95h30m32.851477996s) [noise=true , address=145.79.227.16:4912 , piece_id=XOZ5J3SLEFI7H3AZRFHC5HK2OBVROEOREMY57WJEWYBGVL6QADYA, seg_pos={0 0}, upload=83259]: Uploading piece...
Dec 9 14:59:32.980744797 (95h30m32.851827832s) [noise=true , address=145.79.227.24:4582 , piece_id=UGNKCJAA6PJBE3CWC46ZVVSBZIRTJ4C3L327LLAROK4WMU72MMXQ, seg_pos={0 0}, upload=83259]: Uploading piece...
Dec 9 14:59:32.982809799 (95h30m32.853892844s) [noise=true , address=45.91.116.13:4153 , piece_id=TPZSU5MDFBQSXFM7WRAWFTRHVOLI3TXTZ5MBZDZGIBHSCTOJJEZQ, seg_pos={0 0}, upload=83259]: Uploading piece...
Dec 9 14:59:32.983317917 (95h30m32.854400992s) [noise=true , address=145.79.227.39:4014 , piece_id=MRO2KIUUPJGX6NGQNVY6VDMCIFECHHL3FFJ7OALNM2TFE4YUCMXQ, seg_pos={0 0}, upload=83259]: Uploading piece...
Dec 9 14:59:34.680921451 (95h30m34.552004486s) [upload=83259]: Next returned nil segment
Dec 9 14:59:34.680943153 (95h30m34.552026198s) [upload=83259]: Waiting for error group managing segments...
Dec 9 14:59:34.985677890 (95h30m34.856760925s) [noise=true , address=145.79.227.21:4474 , piece_id=FCSLF4NIRCI6QVRESZP5UY2FK2FNEH3TJUJZTI5XKYBCGDPCGTDQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil>
Dec 9 14:59:34.986532547 (95h30m34.857615592s) [noise=true , address=145.79.227.42:4121 , piece_id=KAPV7DTW5H7ETUCRMVMJUCGTA5ILRBJDB5AOOUAWONAUSWNWXSSQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil>
Dec 9 14:59:34.991365901 (95h30m34.862448936s) [noise=true , address=145.79.227.16:4912 , piece_id=XOZ5J3SLEFI7H3AZRFHC5HK2OBVROEOREMY57WJEWYBGVL6QADYA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil>
Dec 9 14:59:34.996089796 (95h30m34.867172831s) [noise=true , address=145.79.227.43:4157 , piece_id=ZI5ZA5MIRSWF4XPWM5D7UCMOJDIG3W3HRKGM5DYKCDJV7OONS6QA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil>
Dec 9 14:59:34.999561678 (95h30m34.870644713s) [noise=true , address=145.79.227.27:4690 , piece_id=5FAALAMRZZYSWKPEJGAQJFQGSTH5L7SSEIHZXIL57JTZJCPOR66Q, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil>
Dec 9 14:59:35.001337950 (95h30m34.872420985s) [noise=true , address=145.79.227.26:4655 , piece_id=HAAP2ZXXNUBHE2Q7D5BHINZPRVAWWHPGQ4SU4I356YYAJNVZN5HA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil>
Dec 9 14:59:35.001503816 (95h30m34.872586881s) [noise=true , address=145.79.227.28:4726 , piece_id=ERLWH62KS663YXXYEKHY2ZLLJDYXBBTPLL7BD5UDAFH6DG2H5IUA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil>
Dec 9 14:59:35.012234980 (95h30m34.883318025s) [noise=true , address=145.79.227.38:4978 , piece_id=LJ4M45R4QMTU7COWVHWLC4LDJ5JXJ6HSD7HEGHQSAFNKP44R72TQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil>
Dec 9 14:59:35.012262382 (95h30m34.883345437s) [noise=true , address=145.79.227.16:4910 , piece_id=YZH37SYHZCT2T6LZDS2UMDMCLNPW3SUUUX6BZKEQYUKZ3H2SU3AQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil>
Dec 9 14:59:35.015363096 (95h30m34.886446131s) [noise=true , address=145.79.227.33:4869 , piece_id=UK3COALVJUOZOSHS5WQHPZAGDP73D27ZE47MGESVOZB4JHDUUBRA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil>
Dec 9 14:59:35.016246058 (95h30m34.887329103s) [noise=true , address=45.91.116.13:4153 , piece_id=TPZSU5MDFBQSXFM7WRAWFTRHVOLI3TXTZ5MBZDZGIBHSCTOJJEZQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil>
Dec 9 14:59:35.076907590 (95h30m34.947990635s) [noise=true , address=145.79.227.24:4582 , piece_id=UGNKCJAA6PJBE3CWC46ZVVSBZIRTJ4C3L327LLAROK4WMU72MMXQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil>
Dec 9 14:59:35.085585104 (95h30m34.956668149s) [noise=true , address=145.79.227.39:4014 , piece_id=MRO2KIUUPJGX6NGQNVY6VDMCIFECHHL3FFJ7OALNM2TFE4YUCMXQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil>
Dec 9 14:59:35.085609681 (95h30m34.956692726s) [seg_pos={0 0}, upload=83259]: Segment reached optimal threshold of 13 pieces.
Dec 9 14:59:35.085857443 (95h30m34.956940488s) [noise=true , address=145.79.227.29:4942 , piece_id=WH7PWH55KGNUBWQDHD3QRFJM6Q6PHA6JBUTTV5R5REIADMQORYNA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF
Dec 9 14:59:35.085865989 (95h30m34.956949034s) [noise=true , address=145.79.227.29:4942 , piece_id=WH7PWH55KGNUBWQDHD3QRFJM6Q6PHA6JBUTTV5R5REIADMQORYNA, seg_pos={0 0}, upload=83259]: Long tail cancellation due to optimal threshold
Dec 9 14:59:35.086179496 (95h30m34.957262581s) [noise=true , address=145.79.227.34:4906 , piece_id=FNIAMKKNCMYZEIE3VV43GAZLOMQA54NKUOL74VOJCS4WFIBRFG2A, seg_pos={0 0}, upload=83259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF
Dec 9 14:59:35.086188963 (95h30m34.957271999s) [noise=true , address=145.79.227.34:4906 , piece_id=FNIAMKKNCMYZEIE3VV43GAZLOMQA54NKUOL74VOJCS4WFIBRFG2A, seg_pos={0 0}, upload=83259]: Long tail cancellation due to optimal threshold
Dec 9 14:59:35.086341084 (95h30m34.957424129s) [noise=true , address=77.90.31.15:4699 , piece_id=HVPBJSCEG3BFPDA6UZ65XRZKDGEZ6Z6YEREOAUHQ66Y46GX26BIA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: write tcp 188.166.225.170:45542->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:45542->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45542->77.90.31.15:4699: use of closed network connection
Dec 9 14:59:35.086348858 (95h30m34.957431903s) [noise=true , address=77.90.31.15:4699 , piece_id=HVPBJSCEG3BFPDA6UZ65XRZKDGEZ6Z6YEREOAUHQ66Y46GX26BIA, seg_pos={0 0}, upload=83259]: Long tail cancellation due to optimal threshold
Dec 9 14:59:35.086370269 (95h30m34.957453315s) [seg_pos={0 0}, upload=83259]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 9 14:59:35.086452486 (95h30m34.957535531s) [upload=83259]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 9 14:59:35.086459168 (95h30m34.957542203s) [upload=83259]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 9 15:04:20.015922420 (95h35m19.887005475s) [upload=83259]: Done uploading segments...
Dec 9 15:04:20.015991150 (95h35m19.887074196s) [upload=83259]: Upload finished. err: metaclient: context deadline exceeded
Duration: 6.139054552s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [XX..XX...X....X........X.XX.XX.....XXX.......X......X....X..] duration:
13.274772606s trace:
https://jaeger.dev.storj.io/trace/883a30cbeaf6a17
FAIL (2025-12-09 15:04:24 UTC)
orbiter warning: context deadline exceeded
Duration: 13.274802543s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...................X.......................................X] duration:
4m48.957717613s trace:
https://jaeger.dev.storj.io/trace/1f9f83d8cc633aae
FAIL (2025-12-09 15:04:20 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 9 14:59:32.716918810 (95h30m32.588001855s) [upload=83259]: Starting upload
Dec 9 14:59:32.716988723 (95h30m32.588071758s) [upload=83259]: Done starting upload
Dec 9 14:59:32.717137536 (95h30m32.588220581s) [upload=83259]: Uploading segments...
Dec 9 14:59:32.717286900 (95h30m32.588369935s) [upload=83259]: Got next segment. Inline: false
Dec 9 14:59:32.717341534 (95h30m32.58842458s) [upload=83259]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 9 14:59:32.717347646 (95h30m32.588430691s) [upload=83259]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 9 14:59:32.975580773 (95h30m32.846663828s) [seg_pos={0 0}, upload=83259]: Begin upload segment...
Dec 9 14:59:32.975640838 (95h30m32.846723883s) [seg_pos={0 0}, upload=83259]: Done begin upload segment.
Dec 9 14:59:32.975718646 (95h30m32.846801701s) [noise=true , address=145.79.227.16:4910 , piece_id=YZH37SYHZCT2T6LZDS2UMDMCLNPW3SUUUX6BZKEQYUKZ3H2SU3AQ, seg_pos={0 0}, upload=83259]: Uploading piece...
Dec 9 14:59:32.975774743 (95h30m32.846857788s) [noise=true , address=145.79.227.34:4906 , piece_id=FNIAMKKNCMYZEIE3VV43GAZLOMQA54NKUOL74VOJCS4WFIBRFG2A, seg_pos={0 0}, upload=83259]: Uploading piece...
Dec 9 14:59:32.976391548 (95h30m32.847474593s) [noise=true , address=145.79.227.33:4869 , piece_id=UK3COALVJUOZOSHS5WQHPZAGDP73D27ZE47MGESVOZB4JHDUUBRA, seg_pos={0 0}, upload=83259]: Uploading piece...
Dec 9 14:59:32.976648978 (95h30m32.847732023s) [noise=true , address=145.79.227.28:4726 , piece_id=ERLWH62KS663YXXYEKHY2ZLLJDYXBBTPLL7BD5UDAFH6DG2H5IUA, seg_pos={0 0}, upload=83259]: Uploading piece...
Dec 9 14:59:32.977223793 (95h30m32.848306838s) [noise=true , address=145.79.227.27:4690 , piece_id=5FAALAMRZZYSWKPEJGAQJFQGSTH5L7SSEIHZXIL57JTZJCPOR66Q, seg_pos={0 0}, upload=83259]: Uploading piece...
Dec 9 14:59:32.977895982 (95h30m32.848979057s) [noise=true , address=145.79.227.43:4157 , piece_id=ZI5ZA5MIRSWF4XPWM5D7UCMOJDIG3W3HRKGM5DYKCDJV7OONS6QA, seg_pos={0 0}, upload=83259]: Uploading piece...
Dec 9 14:59:32.978304240 (95h30m32.849387285s) [noise=true , address=77.90.31.15:4699 , piece_id=HVPBJSCEG3BFPDA6UZ65XRZKDGEZ6Z6YEREOAUHQ66Y46GX26BIA, seg_pos={0 0}, upload=83259]: Uploading piece...
Dec 9 14:59:32.978738136 (95h30m32.849821181s) [noise=true , address=145.79.227.26:4655 , piece_id=HAAP2ZXXNUBHE2Q7D5BHINZPRVAWWHPGQ4SU4I356YYAJNVZN5HA, seg_pos={0 0}, upload=83259]: Uploading piece...
Dec 9 14:59:32.978777981 (95h30m32.849861016s) [noise=true , address=145.79.227.38:4978 , piece_id=LJ4M45R4QMTU7COWVHWLC4LDJ5JXJ6HSD7HEGHQSAFNKP44R72TQ, seg_pos={0 0}, upload=83259]: Uploading piece...
Dec 9 14:59:32.979321667 (95h30m32.850404712s) [noise=true , address=145.79.227.29:4942 , piece_id=WH7PWH55KGNUBWQDHD3QRFJM6Q6PHA6JBUTTV5R5REIADMQORYNA, seg_pos={0 0}, upload=83259]: Uploading piece...
Dec 9 14:59:32.979840495 (95h30m32.85092354s) [noise=true , address=145.79.227.42:4121 , piece_id=KAPV7DTW5H7ETUCRMVMJUCGTA5ILRBJDB5AOOUAWONAUSWNWXSSQ, seg_pos={0 0}, upload=83259]: Uploading piece...
Dec 9 14:59:32.979932580 (95h30m32.851015625s) [noise=true , address=145.79.227.21:4474 , piece_id=FCSLF4NIRCI6QVRESZP5UY2FK2FNEH3TJUJZTI5XKYBCGDPCGTDQ, seg_pos={0 0}, upload=83259]: Uploading piece...
Dec 9 14:59:32.980394961 (95h30m32.851477996s) [noise=true , address=145.79.227.16:4912 , piece_id=XOZ5J3SLEFI7H3AZRFHC5HK2OBVROEOREMY57WJEWYBGVL6QADYA, seg_pos={0 0}, upload=83259]: Uploading piece...
Dec 9 14:59:32.980744797 (95h30m32.851827832s) [noise=true , address=145.79.227.24:4582 , piece_id=UGNKCJAA6PJBE3CWC46ZVVSBZIRTJ4C3L327LLAROK4WMU72MMXQ, seg_pos={0 0}, upload=83259]: Uploading piece...
Dec 9 14:59:32.982809799 (95h30m32.853892844s) [noise=true , address=45.91.116.13:4153 , piece_id=TPZSU5MDFBQSXFM7WRAWFTRHVOLI3TXTZ5MBZDZGIBHSCTOJJEZQ, seg_pos={0 0}, upload=83259]: Uploading piece...
Dec 9 14:59:32.983317917 (95h30m32.854400992s) [noise=true , address=145.79.227.39:4014 , piece_id=MRO2KIUUPJGX6NGQNVY6VDMCIFECHHL3FFJ7OALNM2TFE4YUCMXQ, seg_pos={0 0}, upload=83259]: Uploading piece...
Dec 9 14:59:34.680921451 (95h30m34.552004486s) [upload=83259]: Next returned nil segment
Dec 9 14:59:34.680943153 (95h30m34.552026198s) [upload=83259]: Waiting for error group managing segments...
Dec 9 14:59:34.985677890 (95h30m34.856760925s) [noise=true , address=145.79.227.21:4474 , piece_id=FCSLF4NIRCI6QVRESZP5UY2FK2FNEH3TJUJZTI5XKYBCGDPCGTDQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil>
Dec 9 14:59:34.986532547 (95h30m34.857615592s) [noise=true , address=145.79.227.42:4121 , piece_id=KAPV7DTW5H7ETUCRMVMJUCGTA5ILRBJDB5AOOUAWONAUSWNWXSSQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil>
Dec 9 14:59:34.991365901 (95h30m34.862448936s) [noise=true , address=145.79.227.16:4912 , piece_id=XOZ5J3SLEFI7H3AZRFHC5HK2OBVROEOREMY57WJEWYBGVL6QADYA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil>
Dec 9 14:59:34.996089796 (95h30m34.867172831s) [noise=true , address=145.79.227.43:4157 , piece_id=ZI5ZA5MIRSWF4XPWM5D7UCMOJDIG3W3HRKGM5DYKCDJV7OONS6QA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil>
Dec 9 14:59:34.999561678 (95h30m34.870644713s) [noise=true , address=145.79.227.27:4690 , piece_id=5FAALAMRZZYSWKPEJGAQJFQGSTH5L7SSEIHZXIL57JTZJCPOR66Q, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil>
Dec 9 14:59:35.001337950 (95h30m34.872420985s) [noise=true , address=145.79.227.26:4655 , piece_id=HAAP2ZXXNUBHE2Q7D5BHINZPRVAWWHPGQ4SU4I356YYAJNVZN5HA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil>
Dec 9 14:59:35.001503816 (95h30m34.872586881s) [noise=true , address=145.79.227.28:4726 , piece_id=ERLWH62KS663YXXYEKHY2ZLLJDYXBBTPLL7BD5UDAFH6DG2H5IUA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil>
Dec 9 14:59:35.012234980 (95h30m34.883318025s) [noise=true , address=145.79.227.38:4978 , piece_id=LJ4M45R4QMTU7COWVHWLC4LDJ5JXJ6HSD7HEGHQSAFNKP44R72TQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil>
Dec 9 14:59:35.012262382 (95h30m34.883345437s) [noise=true , address=145.79.227.16:4910 , piece_id=YZH37SYHZCT2T6LZDS2UMDMCLNPW3SUUUX6BZKEQYUKZ3H2SU3AQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil>
Dec 9 14:59:35.015363096 (95h30m34.886446131s) [noise=true , address=145.79.227.33:4869 , piece_id=UK3COALVJUOZOSHS5WQHPZAGDP73D27ZE47MGESVOZB4JHDUUBRA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil>
Dec 9 14:59:35.016246058 (95h30m34.887329103s) [noise=true , address=45.91.116.13:4153 , piece_id=TPZSU5MDFBQSXFM7WRAWFTRHVOLI3TXTZ5MBZDZGIBHSCTOJJEZQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil>
Dec 9 14:59:35.076907590 (95h30m34.947990635s) [noise=true , address=145.79.227.24:4582 , piece_id=UGNKCJAA6PJBE3CWC46ZVVSBZIRTJ4C3L327LLAROK4WMU72MMXQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil>
Dec 9 14:59:35.085585104 (95h30m34.956668149s) [noise=true , address=145.79.227.39:4014 , piece_id=MRO2KIUUPJGX6NGQNVY6VDMCIFECHHL3FFJ7OALNM2TFE4YUCMXQ, seg_pos={0 0}, upload=83259]: Done uploading piece. err: <nil>
Dec 9 14:59:35.085609681 (95h30m34.956692726s) [seg_pos={0 0}, upload=83259]: Segment reached optimal threshold of 13 pieces.
Dec 9 14:59:35.085857443 (95h30m34.956940488s) [noise=true , address=145.79.227.29:4942 , piece_id=WH7PWH55KGNUBWQDHD3QRFJM6Q6PHA6JBUTTV5R5REIADMQORYNA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF
Dec 9 14:59:35.085865989 (95h30m34.956949034s) [noise=true , address=145.79.227.29:4942 , piece_id=WH7PWH55KGNUBWQDHD3QRFJM6Q6PHA6JBUTTV5R5REIADMQORYNA, seg_pos={0 0}, upload=83259]: Long tail cancellation due to optimal threshold
Dec 9 14:59:35.086179496 (95h30m34.957262581s) [noise=true , address=145.79.227.34:4906 , piece_id=FNIAMKKNCMYZEIE3VV43GAZLOMQA54NKUOL74VOJCS4WFIBRFG2A, seg_pos={0 0}, upload=83259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF
Dec 9 14:59:35.086188963 (95h30m34.957271999s) [noise=true , address=145.79.227.34:4906 , piece_id=FNIAMKKNCMYZEIE3VV43GAZLOMQA54NKUOL74VOJCS4WFIBRFG2A, seg_pos={0 0}, upload=83259]: Long tail cancellation due to optimal threshold
Dec 9 14:59:35.086341084 (95h30m34.957424129s) [noise=true , address=77.90.31.15:4699 , piece_id=HVPBJSCEG3BFPDA6UZ65XRZKDGEZ6Z6YEREOAUHQ66Y46GX26BIA, seg_pos={0 0}, upload=83259]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: write tcp 188.166.225.170:45542->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:45542->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45542->77.90.31.15:4699: use of closed network connection
Dec 9 14:59:35.086348858 (95h30m34.957431903s) [noise=true , address=77.90.31.15:4699 , piece_id=HVPBJSCEG3BFPDA6UZ65XRZKDGEZ6Z6YEREOAUHQ66Y46GX26BIA, seg_pos={0 0}, upload=83259]: Long tail cancellation due to optimal threshold
Dec 9 14:59:35.086370269 (95h30m34.957453315s) [seg_pos={0 0}, upload=83259]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 9 14:59:35.086452486 (95h30m34.957535531s) [upload=83259]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 9 14:59:35.086459168 (95h30m34.957542203s) [upload=83259]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 9 15:04:20.015922420 (95h35m19.887005475s) [upload=83259]: Done uploading segments...
Dec 9 15:04:20.015991150 (95h35m19.887074196s) [upload=83259]: Upload finished. err: metaclient: context deadline exceeded
Duration: 4m48.957742599s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
23 out of 60 is failed (threshold=6) [X.....X.....X....X..XXX.X..X...X.....X.X.XXXXXXX.X....X..X.X] duration:
5.001845684s trace:
https://jaeger.dev.storj.io/trace/2f1e4eb562eacc7c
FAIL (2025-12-09 15:04:19 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001871384s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.X...........X..............................................] duration:
6.109003492s trace:
https://jaeger.dev.storj.io/trace/61c71b63946688f
FAIL (2025-12-09 15:04:03 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 6.109039881s
core
uplink
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
3.552453518s trace:
https://jaeger.dev.storj.io/trace/79b4d9ea215ba812
FAIL (2025-12-09 15:03:48 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.552482071s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
22 out of 60 is failed (threshold=6) [.X.....X.....X....X..XXX.X..X...X.....X.X.XXXXXXX.X....X..X.] duration:
5.001199587s trace:
https://jaeger.dev.storj.io/trace/102db962348e4331
FAIL (2025-12-09 15:03:36 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001229685s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [.XX..XX...X....X........X.XX.XX.....XXX.......X......X....X.] duration:
22.827607155s trace:
https://jaeger.dev.storj.io/trace/883a30cbeaf6a17
FAIL (2025-12-09 15:03:30 UTC)
orbiter warning: context deadline exceeded
Duration: 22.827640721s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
22 out of 60 is failed (threshold=6) [..X.....X.....X....X..XXX.X..X...X.....X.X.XXXXXXX.X....X..X] duration:
5.001482447s trace:
https://jaeger.dev.storj.io/trace/102db962348e4331
FAIL (2025-12-09 15:03:10 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001508216s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..X...........X.............................................] duration:
5.931358363s trace:
https://jaeger.dev.storj.io/trace/61c71b63946688f
FAIL (2025-12-09 15:03:04 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 5.931392168s
core
uplink
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
12.678979067s trace:
https://jaeger.dev.storj.io/trace/79b4d9ea215ba812
FAIL (2025-12-09 15:02:53 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 12.679008764s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...X...........X............................................] duration:
5.761424143s trace:
https://jaeger.dev.storj.io/trace/61c71b63946688f
FAIL (2025-12-09 15:02:39 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 5.761455223s
core
uplink
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [..XX..XX...X....X........X.XX.XX.....XXX.......X......X....X] duration:
30.00125314s trace:
https://jaeger.dev.storj.io/trace/883a30cbeaf6a17
FAIL (2025-12-09 15:02:33 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001280422s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
2 out of 60 is failed (threshold=3) [....X...........X...........................................] duration:
5.709484587s trace:
https://jaeger.dev.storj.io/trace/61c71b63946688f
FAIL (2025-12-09 15:02:00 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 5.709514564s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
22 out of 60 is failed (threshold=6) [X..X.....X.....X....X..XXX.X..X...X.....X.X.XXXXXXX.X....X..] duration:
5.001381893s trace:
https://jaeger.dev.storj.io/trace/246fd772e90ac91
FAIL (2025-12-09 15:01:56 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.00141709s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
5.720812219s trace:
https://jaeger.dev.storj.io/trace/79b4d9ea215ba812
FAIL (2025-12-09 15:01:41 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.720841665s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....X...........X..........................................] duration:
5.713101465s trace:
https://jaeger.dev.storj.io/trace/61c71b63946688f
FAIL (2025-12-09 15:01:40 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 5.71322496s
core
uplink
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [X..XX..XX...X....X........X.XX.XX.....XXX.......X......X....] duration:
10.998543438s trace:
https://jaeger.dev.storj.io/trace/778dd0c5ac23e6be
FAIL (2025-12-09 15:01:38 UTC)
orbiter warning: context deadline exceeded
Duration: 10.998576001s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [.X..XX..XX...X....X........X.XX.XX.....XXX.......X......X...] duration:
21.546275905s trace:
https://jaeger.dev.storj.io/trace/778dd0c5ac23e6be
FAIL (2025-12-09 15:01:18 UTC)
orbiter warning: context deadline exceeded
Duration: 21.546321491s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
2 out of 60 is failed (threshold=3) [......X...........X.........................................] duration:
5.889102677s trace:
https://jaeger.dev.storj.io/trace/61c71b63946688f
FAIL (2025-12-09 15:01:17 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 5.889135899s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
2.021961278s trace:
https://jaeger.dev.storj.io/trace/ab517902cf7571b
FAIL (2025-12-09 15:01:07 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 14:14:03.293481138 (94h45m3.164564193s) [upload=82602]: Starting upload Dec 9 14:14:03.293630602 (94h45m3.164713667s) [upload=82602]: Done starting upload Dec 9 14:14:03.293745661 (94h45m3.164828716s) [upload=82602]: Uploading segments... Dec 9 14:14:03.293890728 (94h45m3.164973783s) [upload=82602]: Got next segment. Inline: true Dec 9 14:14:03.293897561 (94h45m3.164980616s) [upload=82602]: Waiting for error group managing segments... Dec 9 14:14:03.294051774 (94h45m3.165134809s) [upload=82602]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Dec 9 14:14:03.294059880 (94h45m3.165142925s) [upload=82602]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Dec 9 14:14:03.294063066 (94h45m3.165146111s) [upload=82602]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Dec 9 14:15:01.261001342 (94h46m1.132084387s) [upload=82602]: Done uploading segments... Dec 9 14:15:01.261048982 (94h46m1.132132017s) [upload=82602]: Upload finished. err: metaclient: context deadline exceeded
Duration: 2.02199368s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [..X..XX..XX...X....X........X.XX.XX.....XXX.......X......X..] duration:
26.624422608s trace:
https://jaeger.dev.storj.io/trace/778dd0c5ac23e6be
FAIL (2025-12-09 15:00:56 UTC)
orbiter warning: context deadline exceeded
Duration: 26.624451444s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......X...........X........................................] duration:
6.017594313s trace:
https://jaeger.dev.storj.io/trace/61c71b63946688f
FAIL (2025-12-09 15:00:48 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 6.017648757s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
23 out of 60 is failed (threshold=6) [XX..X.....X.....X....X..XXX.X..X...X.....X.X.XXXXXXX.X....X.] duration:
4.761198606s trace:
https://jaeger.dev.storj.io/trace/246fd772e90ac91
FAIL (2025-12-09 15:00:45 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.761227722s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
6.156643761s trace:
https://jaeger.dev.storj.io/trace/79b4d9ea215ba812
FAIL (2025-12-09 15:00:30 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.156676054s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
24 out of 60 is failed (threshold=6) [XXX..X.....X.....X....X..XXX.X..X...X.....X.X.XXXXXXX.X....X] duration:
5.002225972s trace:
https://jaeger.dev.storj.io/trace/246fd772e90ac91
FAIL (2025-12-09 15:00:01 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.002251453s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
2.038098396s trace:
https://jaeger.dev.storj.io/trace/ab517902cf7571b
FAIL (2025-12-09 14:59:57 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 14:14:03.293481138 (94h45m3.164564193s) [upload=82602]: Starting upload Dec 9 14:14:03.293630602 (94h45m3.164713667s) [upload=82602]: Done starting upload Dec 9 14:14:03.293745661 (94h45m3.164828716s) [upload=82602]: Uploading segments... Dec 9 14:14:03.293890728 (94h45m3.164973783s) [upload=82602]: Got next segment. Inline: true Dec 9 14:14:03.293897561 (94h45m3.164980616s) [upload=82602]: Waiting for error group managing segments... Dec 9 14:14:03.294051774 (94h45m3.165134809s) [upload=82602]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Dec 9 14:14:03.294059880 (94h45m3.165142925s) [upload=82602]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Dec 9 14:14:03.294063066 (94h45m3.165146111s) [upload=82602]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Dec 9 14:15:01.261001342 (94h46m1.132084387s) [upload=82602]: Done uploading segments... Dec 9 14:15:01.261048982 (94h46m1.132132017s) [upload=82602]: Upload finished. err: metaclient: context deadline exceeded
Duration: 2.038124987s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [...X..XX..XX...X....X........X.XX.XX.....XXX.......X......X.] duration:
20.380692487s trace:
https://jaeger.dev.storj.io/trace/778dd0c5ac23e6be
FAIL (2025-12-09 14:59:51 UTC)
orbiter warning: context deadline exceeded
Duration: 20.380708729s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
2 out of 60 is failed (threshold=3) [........X...........X.......................................] duration:
5.645733922s trace:
https://jaeger.dev.storj.io/trace/61c71b63946688f
FAIL (2025-12-09 14:59:40 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 5.645767896s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
2.094951376s trace:
https://jaeger.dev.storj.io/trace/ab517902cf7571b
FAIL (2025-12-09 14:59:39 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 14:14:03.293481138 (94h45m3.164564193s) [upload=82602]: Starting upload Dec 9 14:14:03.293630602 (94h45m3.164713667s) [upload=82602]: Done starting upload Dec 9 14:14:03.293745661 (94h45m3.164828716s) [upload=82602]: Uploading segments... Dec 9 14:14:03.293890728 (94h45m3.164973783s) [upload=82602]: Got next segment. Inline: true Dec 9 14:14:03.293897561 (94h45m3.164980616s) [upload=82602]: Waiting for error group managing segments... Dec 9 14:14:03.294051774 (94h45m3.165134809s) [upload=82602]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Dec 9 14:14:03.294059880 (94h45m3.165142925s) [upload=82602]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Dec 9 14:14:03.294063066 (94h45m3.165146111s) [upload=82602]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Dec 9 14:15:01.261001342 (94h46m1.132084387s) [upload=82602]: Done uploading segments... Dec 9 14:15:01.261048982 (94h46m1.132132017s) [upload=82602]: Upload finished. err: metaclient: context deadline exceeded
Duration: 2.094982936s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
2.924221023s trace:
https://jaeger.dev.storj.io/trace/79b4d9ea215ba812
FAIL (2025-12-09 14:59:33 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.924249567s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [....X..XX..XX...X....X........X.XX.XX.....XXX.......X......X] duration:
30.000782492s trace:
https://jaeger.dev.storj.io/trace/778dd0c5ac23e6be
FAIL (2025-12-09 14:59:31 UTC)
orbiter warning: context deadline exceeded
Duration: 30.000887512s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
23 out of 60 is failed (threshold=6) [.XXX..X.....X.....X....X..XXX.X..X...X.....X.X.XXXXXXX.X....] duration:
4.611073699s trace:
https://jaeger.dev.storj.io/trace/60fc70d527e05171
FAIL (2025-12-09 14:59:10 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.611105128s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
2.019099711s trace:
https://jaeger.dev.storj.io/trace/ab517902cf7571b
FAIL (2025-12-09 14:58:56 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 14:14:03.293481138 (94h45m3.164564193s) [upload=82602]: Starting upload Dec 9 14:14:03.293630602 (94h45m3.164713667s) [upload=82602]: Done starting upload Dec 9 14:14:03.293745661 (94h45m3.164828716s) [upload=82602]: Uploading segments... Dec 9 14:14:03.293890728 (94h45m3.164973783s) [upload=82602]: Got next segment. Inline: true Dec 9 14:14:03.293897561 (94h45m3.164980616s) [upload=82602]: Waiting for error group managing segments... Dec 9 14:14:03.294051774 (94h45m3.165134809s) [upload=82602]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Dec 9 14:14:03.294059880 (94h45m3.165142925s) [upload=82602]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Dec 9 14:14:03.294063066 (94h45m3.165146111s) [upload=82602]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Dec 9 14:15:01.261001342 (94h46m1.132084387s) [upload=82602]: Done uploading segments... Dec 9 14:15:01.261048982 (94h46m1.132132017s) [upload=82602]: Upload finished. err: metaclient: context deadline exceeded
Duration: 2.019128285s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
9.820158363s trace:
https://jaeger.dev.storj.io/trace/79b4d9ea215ba812
FAIL (2025-12-09 14:58:33 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 9.820180794s
edge
gateway
us-select-1
|