Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
8.840018595s trace:
https://jaeger.dev.storj.io/trace/406d13f34dbd7740
FAIL (2025-12-08 14:28:24 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 8.840054754s
core
uplink
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
25 out of 60 is failed (threshold=3) [...X.X.............X.X..X.XXXX...XXXXXXX.X.X....X.XXX.X..XX.] duration:
14.148757578s trace:
https://jaeger.dev.storj.io/trace/5da19c55cb606228
FAIL (2025-12-08 14:28:11 UTC)
orbiter warning: context deadline exceeded
Duration: 14.148773399s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................] duration:
2.966949535s trace:
https://jaeger.dev.storj.io/trace/29e54bf3415d9356
FAIL (2025-12-08 14:28:06 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 2.966984933s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
24 out of 60 is failed (threshold=3) [.XX....X.X..X.X..XX....X..........X.XX.XXX.X..XXX.X.XXX.X...] duration:
17.928607649s trace:
https://jaeger.dev.storj.io/trace/2c2d60706ffaf08c
FAIL (2025-12-08 14:27:58 UTC)
orbiter warning: context deadline exceeded
Duration: 17.928636564s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
25 out of 60 is failed (threshold=3) [....X.X.............X.X..X.XXXX...XXXXXXX.X.X....X.XXX.X..XX] duration:
30.00129861s trace:
https://jaeger.dev.storj.io/trace/5da19c55cb606228
FAIL (2025-12-08 14:27:56 UTC)
orbiter warning: context deadline exceeded
Duration: 30.00131381s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [............................................X..........X....] duration:
12.652031582s trace:
https://jaeger.dev.storj.io/trace/5ce839c1456bbf8a
FAIL (2025-12-08 14:27:54 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
25:20.182936263 (70h56m20.054019298s) [upload=62126]: Starting upload
Dec 8 14:25:20.183037416 (70h56m20.054120461s) [upload=62126]: Done starting upload
Dec 8 14:25:20.183136214 (70h56m20.054219259s) [upload=62126]: Uploading segments...
Dec 8 14:25:20.183291900 (70h56m20.054374945s) [upload=62126]: Got next segment. Inline: false
Dec 8 14:25:20.183343058 (70h56m20.054426103s) [upload=62126]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 14:25:20.183349700 (70h56m20.054432745s) [upload=62126]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 14:25:20.560968522 (70h56m20.432051567s) [seg_pos={0 0}, upload=62126]: Begin upload segment...
Dec 8 14:25:20.561021322 (70h56m20.432104357s) [seg_pos={0 0}, upload=62126]: Done begin upload segment.
Dec 8 14:25:20.561050337 (70h56m20.432133383s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.562561536 (70h56m20.433644581s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.563077198 (70h56m20.434160243s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.563560729 (70h56m20.434643774s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.564043677 (70h56m20.435126732s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.564505988 (70h56m20.435589043s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.564975802 (70h56m20.436058848s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.566789025 (70h56m20.43787207s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.568723158 (70h56m20.439806204s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.570985847 (70h56m20.442068892s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.573462843 (70h56m20.444545889s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.573987674 (70h56m20.445070719s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.575740802 (70h56m20.446823837s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.577275113 (70h56m20.448358158s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.578540883 (70h56m20.449623928s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.580403731 (70h56m20.451486776s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:22.334236681 (70h56m22.205319726s) [upload=62126]: Next returned nil segment
Dec 8 14:25:22.334251700 (70h56m22.205334735s) [upload=62126]: Waiting for error group managing segments...
Dec 8 14:25:22.746429339 (70h56m22.617512384s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:22.747165611 (70h56m22.618248646s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:22.753178531 (70h56m22.624261566s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:22.922057260 (70h56m22.793140305s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:22.951038517 (70h56m22.822121562s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:23.651296259 (70h56m23.522379304s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:23.653766143 (70h56m23.524849178s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:23.680671276 (70h56m23.551754321s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:23.700290564 (70h56m23.571373609s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:23.707126822 (70h56m23.578209868s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:23.721842694 (70h56m23.592925739s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:23.730554192 (70h56m23.601637237s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:24.670621120 (70h56m24.541704185s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:24.670647519 (70h56m24.541730564s) [seg_pos={0 0}, upload=62126]: Segment reached optimal threshold of 13 pieces.
Dec 8 14:25:24.670967409 (70h56m24.542050454s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection
Dec 8 14:25:24.670977318 (70h56m24.542060363s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold
Dec 8 14:25:24.671144947 (70h56m24.542227982s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: read tcp 188.166.225.170:39576->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 8 14:25:24.671168231 (70h56m24.542251306s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold
Dec 8 14:25:24.671490626 (70h56m24.542573661s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF
Dec 8 14:25:24.671499091 (70h56m24.542582126s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold
Dec 8 14:25:24.671551481 (70h56m24.542634526s) [seg_pos={0 0}, upload=62126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 14:25:24.671640871 (70h56m24.542723906s) [upload=62126]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 14:25:24.671647103 (70h56m24.542730138s) [upload=62126]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 14:25:25.114013719 (70h56m24.985096764s) [upload=62126]: Done uploading segments...
Dec 8 14:25:25.114042193 (70h56m24.985125258s) [upload=62126]: Upload finished. err: <nil>
Duration: 12.652060588s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
8.213147291s trace:
https://jaeger.dev.storj.io/trace/406d13f34dbd7740
FAIL (2025-12-08 14:27:41 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 8.213177079s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
59 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX] duration:
5.0058052s trace:
https://jaeger.dev.storj.io/trace/60ebef8f11bcb4dd
FAIL (2025-12-08 14:27:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.005820449s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............................................X..........X...] duration:
24.925503998s trace:
https://jaeger.dev.storj.io/trace/5ce839c1456bbf8a
FAIL (2025-12-08 14:27:33 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
25:20.182936263 (70h56m20.054019298s) [upload=62126]: Starting upload
Dec 8 14:25:20.183037416 (70h56m20.054120461s) [upload=62126]: Done starting upload
Dec 8 14:25:20.183136214 (70h56m20.054219259s) [upload=62126]: Uploading segments...
Dec 8 14:25:20.183291900 (70h56m20.054374945s) [upload=62126]: Got next segment. Inline: false
Dec 8 14:25:20.183343058 (70h56m20.054426103s) [upload=62126]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 14:25:20.183349700 (70h56m20.054432745s) [upload=62126]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 14:25:20.560968522 (70h56m20.432051567s) [seg_pos={0 0}, upload=62126]: Begin upload segment...
Dec 8 14:25:20.561021322 (70h56m20.432104357s) [seg_pos={0 0}, upload=62126]: Done begin upload segment.
Dec 8 14:25:20.561050337 (70h56m20.432133383s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.562561536 (70h56m20.433644581s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.563077198 (70h56m20.434160243s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.563560729 (70h56m20.434643774s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.564043677 (70h56m20.435126732s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.564505988 (70h56m20.435589043s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.564975802 (70h56m20.436058848s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.566789025 (70h56m20.43787207s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.568723158 (70h56m20.439806204s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.570985847 (70h56m20.442068892s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.573462843 (70h56m20.444545889s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.573987674 (70h56m20.445070719s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.575740802 (70h56m20.446823837s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.577275113 (70h56m20.448358158s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.578540883 (70h56m20.449623928s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.580403731 (70h56m20.451486776s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:22.334236681 (70h56m22.205319726s) [upload=62126]: Next returned nil segment
Dec 8 14:25:22.334251700 (70h56m22.205334735s) [upload=62126]: Waiting for error group managing segments...
Dec 8 14:25:22.746429339 (70h56m22.617512384s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:22.747165611 (70h56m22.618248646s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:22.753178531 (70h56m22.624261566s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:22.922057260 (70h56m22.793140305s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:22.951038517 (70h56m22.822121562s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:23.651296259 (70h56m23.522379304s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:23.653766143 (70h56m23.524849178s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:23.680671276 (70h56m23.551754321s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:23.700290564 (70h56m23.571373609s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:23.707126822 (70h56m23.578209868s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:23.721842694 (70h56m23.592925739s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:23.730554192 (70h56m23.601637237s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:24.670621120 (70h56m24.541704185s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:24.670647519 (70h56m24.541730564s) [seg_pos={0 0}, upload=62126]: Segment reached optimal threshold of 13 pieces.
Dec 8 14:25:24.670967409 (70h56m24.542050454s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection
Dec 8 14:25:24.670977318 (70h56m24.542060363s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold
Dec 8 14:25:24.671144947 (70h56m24.542227982s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: read tcp 188.166.225.170:39576->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 8 14:25:24.671168231 (70h56m24.542251306s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold
Dec 8 14:25:24.671490626 (70h56m24.542573661s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF
Dec 8 14:25:24.671499091 (70h56m24.542582126s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold
Dec 8 14:25:24.671551481 (70h56m24.542634526s) [seg_pos={0 0}, upload=62126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 14:25:24.671640871 (70h56m24.542723906s) [upload=62126]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 14:25:24.671647103 (70h56m24.542730138s) [upload=62126]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 14:25:25.114013719 (70h56m24.985096764s) [upload=62126]: Done uploading segments...
Dec 8 14:25:25.114042193 (70h56m24.985125258s) [upload=62126]: Upload finished. err: <nil>
Duration: 24.925533854s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
24 out of 60 is failed (threshold=3) [..XX....X.X..X.X..XX....X..........X.XX.XXX.X..XXX.X.XXX.X..] duration:
21.563280131s trace:
https://jaeger.dev.storj.io/trace/2c2d60706ffaf08c
FAIL (2025-12-08 14:27:30 UTC)
orbiter warning: context deadline exceeded
Duration: 21.563301161s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
25 out of 60 is failed (threshold=3) [X....X.X.............X.X..X.XXXX...XXXXXXX.X.X....X.XXX.X..X] duration:
30.001867792s trace:
https://jaeger.dev.storj.io/trace/3be2b7bcf9f07eec
FAIL (2025-12-08 14:27:26 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001891918s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
9.955850432s trace:
https://jaeger.dev.storj.io/trace/406d13f34dbd7740
FAIL (2025-12-08 14:27:14 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 9.955881052s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
3.104464389s trace:
https://jaeger.dev.storj.io/trace/29e54bf3415d9356
FAIL (2025-12-08 14:27:00 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 3.104494686s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
59 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX] duration:
5.000795618s trace:
https://jaeger.dev.storj.io/trace/3ab61d7eb93bdac
FAIL (2025-12-08 14:26:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000920045s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
25 out of 60 is failed (threshold=3) [X..XX....X.X..X.X..XX....X..........X.XX.XXX.X..XXX.X.XXX.X.] duration:
19.141744684s trace:
https://jaeger.dev.storj.io/trace/2c2d60706ffaf08c
FAIL (2025-12-08 14:26:47 UTC)
orbiter warning: context deadline exceeded
Duration: 19.141761956s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..............................................X..........X..] duration:
10.59774896s trace:
https://jaeger.dev.storj.io/trace/5ce839c1456bbf8a
FAIL (2025-12-08 14:26:41 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
25:20.182936263 (70h56m20.054019298s) [upload=62126]: Starting upload
Dec 8 14:25:20.183037416 (70h56m20.054120461s) [upload=62126]: Done starting upload
Dec 8 14:25:20.183136214 (70h56m20.054219259s) [upload=62126]: Uploading segments...
Dec 8 14:25:20.183291900 (70h56m20.054374945s) [upload=62126]: Got next segment. Inline: false
Dec 8 14:25:20.183343058 (70h56m20.054426103s) [upload=62126]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 14:25:20.183349700 (70h56m20.054432745s) [upload=62126]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 14:25:20.560968522 (70h56m20.432051567s) [seg_pos={0 0}, upload=62126]: Begin upload segment...
Dec 8 14:25:20.561021322 (70h56m20.432104357s) [seg_pos={0 0}, upload=62126]: Done begin upload segment.
Dec 8 14:25:20.561050337 (70h56m20.432133383s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.562561536 (70h56m20.433644581s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.563077198 (70h56m20.434160243s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.563560729 (70h56m20.434643774s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.564043677 (70h56m20.435126732s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.564505988 (70h56m20.435589043s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.564975802 (70h56m20.436058848s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.566789025 (70h56m20.43787207s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.568723158 (70h56m20.439806204s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.570985847 (70h56m20.442068892s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.573462843 (70h56m20.444545889s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.573987674 (70h56m20.445070719s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.575740802 (70h56m20.446823837s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.577275113 (70h56m20.448358158s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.578540883 (70h56m20.449623928s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.580403731 (70h56m20.451486776s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:22.334236681 (70h56m22.205319726s) [upload=62126]: Next returned nil segment
Dec 8 14:25:22.334251700 (70h56m22.205334735s) [upload=62126]: Waiting for error group managing segments...
Dec 8 14:25:22.746429339 (70h56m22.617512384s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:22.747165611 (70h56m22.618248646s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:22.753178531 (70h56m22.624261566s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:22.922057260 (70h56m22.793140305s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:22.951038517 (70h56m22.822121562s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:23.651296259 (70h56m23.522379304s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:23.653766143 (70h56m23.524849178s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:23.680671276 (70h56m23.551754321s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:23.700290564 (70h56m23.571373609s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:23.707126822 (70h56m23.578209868s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:23.721842694 (70h56m23.592925739s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:23.730554192 (70h56m23.601637237s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:24.670621120 (70h56m24.541704185s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:24.670647519 (70h56m24.541730564s) [seg_pos={0 0}, upload=62126]: Segment reached optimal threshold of 13 pieces.
Dec 8 14:25:24.670967409 (70h56m24.542050454s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection
Dec 8 14:25:24.670977318 (70h56m24.542060363s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold
Dec 8 14:25:24.671144947 (70h56m24.542227982s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: read tcp 188.166.225.170:39576->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 8 14:25:24.671168231 (70h56m24.542251306s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold
Dec 8 14:25:24.671490626 (70h56m24.542573661s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF
Dec 8 14:25:24.671499091 (70h56m24.542582126s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold
Dec 8 14:25:24.671551481 (70h56m24.542634526s) [seg_pos={0 0}, upload=62126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 14:25:24.671640871 (70h56m24.542723906s) [upload=62126]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 14:25:24.671647103 (70h56m24.542730138s) [upload=62126]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 14:25:25.114013719 (70h56m24.985096764s) [upload=62126]: Done uploading segments...
Dec 8 14:25:25.114042193 (70h56m24.985125258s) [upload=62126]: Upload finished. err: <nil>
Duration: 10.597774428s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
24 out of 60 is failed (threshold=3) [.X....X.X.............X.X..X.XXXX...XXXXXXX.X.X....X.XXX.X..] duration:
28.897717754s trace:
https://jaeger.dev.storj.io/trace/da20230466dec99
FAIL (2025-12-08 14:26:37 UTC)
orbiter warning: context deadline exceeded
Duration: 28.897747561s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
26 out of 60 is failed (threshold=3) [XX..XX....X.X..X.X..XX....X..........X.XX.XXX.X..XXX.X.XXX.X] duration:
1m0.001043124s trace:
https://jaeger.dev.storj.io/trace/2c2d60706ffaf08c
FAIL (2025-12-08 14:26:28 UTC)
orbiter warning: context deadline exceeded
Duration: 1m0.001059295s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
7.659399994s trace:
https://jaeger.dev.storj.io/trace/406d13f34dbd7740
FAIL (2025-12-08 14:26:15 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 7.659437144s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...............................................X..........X.] duration:
15.034626699s trace:
https://jaeger.dev.storj.io/trace/5ce839c1456bbf8a
FAIL (2025-12-08 14:26:10 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
25:20.182936263 (70h56m20.054019298s) [upload=62126]: Starting upload
Dec 8 14:25:20.183037416 (70h56m20.054120461s) [upload=62126]: Done starting upload
Dec 8 14:25:20.183136214 (70h56m20.054219259s) [upload=62126]: Uploading segments...
Dec 8 14:25:20.183291900 (70h56m20.054374945s) [upload=62126]: Got next segment. Inline: false
Dec 8 14:25:20.183343058 (70h56m20.054426103s) [upload=62126]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 14:25:20.183349700 (70h56m20.054432745s) [upload=62126]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 14:25:20.560968522 (70h56m20.432051567s) [seg_pos={0 0}, upload=62126]: Begin upload segment...
Dec 8 14:25:20.561021322 (70h56m20.432104357s) [seg_pos={0 0}, upload=62126]: Done begin upload segment.
Dec 8 14:25:20.561050337 (70h56m20.432133383s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.562561536 (70h56m20.433644581s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.563077198 (70h56m20.434160243s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.563560729 (70h56m20.434643774s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.564043677 (70h56m20.435126732s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.564505988 (70h56m20.435589043s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.564975802 (70h56m20.436058848s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.566789025 (70h56m20.43787207s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.568723158 (70h56m20.439806204s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.570985847 (70h56m20.442068892s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.573462843 (70h56m20.444545889s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.573987674 (70h56m20.445070719s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.575740802 (70h56m20.446823837s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.577275113 (70h56m20.448358158s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.578540883 (70h56m20.449623928s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.580403731 (70h56m20.451486776s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:22.334236681 (70h56m22.205319726s) [upload=62126]: Next returned nil segment
Dec 8 14:25:22.334251700 (70h56m22.205334735s) [upload=62126]: Waiting for error group managing segments...
Dec 8 14:25:22.746429339 (70h56m22.617512384s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:22.747165611 (70h56m22.618248646s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:22.753178531 (70h56m22.624261566s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:22.922057260 (70h56m22.793140305s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:22.951038517 (70h56m22.822121562s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:23.651296259 (70h56m23.522379304s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:23.653766143 (70h56m23.524849178s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:23.680671276 (70h56m23.551754321s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:23.700290564 (70h56m23.571373609s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:23.707126822 (70h56m23.578209868s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:23.721842694 (70h56m23.592925739s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:23.730554192 (70h56m23.601637237s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:24.670621120 (70h56m24.541704185s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:24.670647519 (70h56m24.541730564s) [seg_pos={0 0}, upload=62126]: Segment reached optimal threshold of 13 pieces.
Dec 8 14:25:24.670967409 (70h56m24.542050454s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection
Dec 8 14:25:24.670977318 (70h56m24.542060363s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold
Dec 8 14:25:24.671144947 (70h56m24.542227982s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: read tcp 188.166.225.170:39576->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 8 14:25:24.671168231 (70h56m24.542251306s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold
Dec 8 14:25:24.671490626 (70h56m24.542573661s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF
Dec 8 14:25:24.671499091 (70h56m24.542582126s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold
Dec 8 14:25:24.671551481 (70h56m24.542634526s) [seg_pos={0 0}, upload=62126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 14:25:24.671640871 (70h56m24.542723906s) [upload=62126]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 14:25:24.671647103 (70h56m24.542730138s) [upload=62126]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 14:25:25.114013719 (70h56m24.985096764s) [upload=62126]: Done uploading segments...
Dec 8 14:25:25.114042193 (70h56m24.985125258s) [upload=62126]: Upload finished. err: <nil>
Duration: 15.034653862s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
3.02389293s trace:
https://jaeger.dev.storj.io/trace/29e54bf3415d9356
FAIL (2025-12-08 14:26:06 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 3.0239615s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
59 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X] duration:
5.001210678s trace:
https://jaeger.dev.storj.io/trace/28f9da17c8181142
FAIL (2025-12-08 14:25:56 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001274699s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [................................................X..........X] duration:
30.002775264s trace:
https://jaeger.dev.storj.io/trace/5ce839c1456bbf8a
FAIL (2025-12-08 14:25:49 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
25:20.182936263 (70h56m20.054019298s) [upload=62126]: Starting upload
Dec 8 14:25:20.183037416 (70h56m20.054120461s) [upload=62126]: Done starting upload
Dec 8 14:25:20.183136214 (70h56m20.054219259s) [upload=62126]: Uploading segments...
Dec 8 14:25:20.183291900 (70h56m20.054374945s) [upload=62126]: Got next segment. Inline: false
Dec 8 14:25:20.183343058 (70h56m20.054426103s) [upload=62126]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 14:25:20.183349700 (70h56m20.054432745s) [upload=62126]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 14:25:20.560968522 (70h56m20.432051567s) [seg_pos={0 0}, upload=62126]: Begin upload segment...
Dec 8 14:25:20.561021322 (70h56m20.432104357s) [seg_pos={0 0}, upload=62126]: Done begin upload segment.
Dec 8 14:25:20.561050337 (70h56m20.432133383s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.562561536 (70h56m20.433644581s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.563077198 (70h56m20.434160243s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.563560729 (70h56m20.434643774s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.564043677 (70h56m20.435126732s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.564505988 (70h56m20.435589043s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.564975802 (70h56m20.436058848s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.566789025 (70h56m20.43787207s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.568723158 (70h56m20.439806204s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.570985847 (70h56m20.442068892s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.573462843 (70h56m20.444545889s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.573987674 (70h56m20.445070719s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.575740802 (70h56m20.446823837s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.577275113 (70h56m20.448358158s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.578540883 (70h56m20.449623928s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:20.580403731 (70h56m20.451486776s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Uploading piece...
Dec 8 14:25:22.334236681 (70h56m22.205319726s) [upload=62126]: Next returned nil segment
Dec 8 14:25:22.334251700 (70h56m22.205334735s) [upload=62126]: Waiting for error group managing segments...
Dec 8 14:25:22.746429339 (70h56m22.617512384s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:22.747165611 (70h56m22.618248646s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:22.753178531 (70h56m22.624261566s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:22.922057260 (70h56m22.793140305s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:22.951038517 (70h56m22.822121562s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:23.651296259 (70h56m23.522379304s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:23.653766143 (70h56m23.524849178s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:23.680671276 (70h56m23.551754321s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:23.700290564 (70h56m23.571373609s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:23.707126822 (70h56m23.578209868s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:23.721842694 (70h56m23.592925739s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:23.730554192 (70h56m23.601637237s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:24.670621120 (70h56m24.541704185s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil>
Dec 8 14:25:24.670647519 (70h56m24.541730564s) [seg_pos={0 0}, upload=62126]: Segment reached optimal threshold of 13 pieces.
Dec 8 14:25:24.670967409 (70h56m24.542050454s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection
Dec 8 14:25:24.670977318 (70h56m24.542060363s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold
Dec 8 14:25:24.671144947 (70h56m24.542227982s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: read tcp 188.166.225.170:39576->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 8 14:25:24.671168231 (70h56m24.542251306s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold
Dec 8 14:25:24.671490626 (70h56m24.542573661s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF
Dec 8 14:25:24.671499091 (70h56m24.542582126s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold
Dec 8 14:25:24.671551481 (70h56m24.542634526s) [seg_pos={0 0}, upload=62126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 14:25:24.671640871 (70h56m24.542723906s) [upload=62126]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 14:25:24.671647103 (70h56m24.542730138s) [upload=62126]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 14:25:25.114013719 (70h56m24.985096764s) [upload=62126]: Done uploading segments...
Dec 8 14:25:25.114042193 (70h56m24.985125258s) [upload=62126]: Upload finished. err: <nil>
Duration: 30.002802695s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
7.926477288s trace:
https://jaeger.dev.storj.io/trace/406d13f34dbd7740
FAIL (2025-12-08 14:25:49 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 7.926513568s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
14.881928728s trace:
https://jaeger.dev.storj.io/trace/29e54bf3415d9356
FAIL (2025-12-08 14:25:29 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 14.881958213s
core
uplink
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
24 out of 60 is failed (threshold=3) [..X....X.X.............X.X..X.XXXX...XXXXXXX.X.X....X.XXX.X.] duration:
22.092311131s trace:
https://jaeger.dev.storj.io/trace/da20230466dec99
FAIL (2025-12-08 14:25:21 UTC)
orbiter warning: context deadline exceeded
Duration: 22.09233182s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
24 out of 60 is failed (threshold=3) [...X....X.X.............X.X..X.XXXX...XXXXXXX.X.X....X.XXX.X] duration:
30.001648792s trace:
https://jaeger.dev.storj.io/trace/da20230466dec99
FAIL (2025-12-08 14:24:59 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001668649s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
59 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.] duration:
3.714144496s trace:
https://jaeger.dev.storj.io/trace/14079119cd1f278d
FAIL (2025-12-08 14:24:57 UTC)
orbiter warning: context deadline exceeded
Duration: 3.71417207s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
26 out of 60 is failed (threshold=3) [XXX..XX....X.X..X.X..XX....X..........X.XX.XXX.X..XXX.X.XXX.] duration:
20.694092966s trace:
https://jaeger.dev.storj.io/trace/fe07c3139e7e198
FAIL (2025-12-08 14:24:54 UTC)
orbiter warning: context deadline exceeded
Duration: 20.694112203s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
8.507163617s trace:
https://jaeger.dev.storj.io/trace/79da76e828604af6
FAIL (2025-12-08 14:24:52 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 8 14:15:00.792576635 (70h46m0.66365968s) [upload=61969]: Starting upload
Dec 8 14:15:00.792651899 (70h46m0.663734975s) [upload=61969]: Done starting upload
Dec 8 14:15:00.792689761 (70h46m0.663772797s) [upload=61969]: Uploading segments...
Dec 8 14:15:00.792980154 (70h46m0.664063209s) [upload=61969]: Got next segment. Inline: false
Dec 8 14:15:00.793097006 (70h46m0.664180051s) [upload=61969]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 14:15:00.793104832 (70h46m0.664187887s) [upload=61969]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 14:15:02.526579968 (70h46m2.397663013s) [seg_pos={0 0}, upload=61969]: Begin upload segment...
Dec 8 14:15:02.526648298 (70h46m2.397731353s) [seg_pos={0 0}, upload=61969]: Done begin upload segment.
Dec 8 14:15:02.526688614 (70h46m2.397771659s) [noise=true , address=145.79.227.32:4835 , piece_id=HPWNW5AC354GTIRDSCEU3NN3ULKBIUUMPRHLVNCECVKGYA5TR2DQ, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.529494027 (70h46m2.400577102s) [noise=true , address=145.79.227.41:4087 , piece_id=OZA2MCHUMIJYWXHS3KAVJRVTJ6LWAW5XR257OGAZRNPIOGMETZRA, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.531073364 (70h46m2.402156419s) [noise=true , address=145.79.227.20:4439 , piece_id=ELOM2UKEDSKKNWILB2I325A5OTSQAKIILQHP345RSTC2LXECKTVQ, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.531556935 (70h46m2.40263998s) [noise=true , address=145.79.227.33:4871 , piece_id=F7NB65E364EI6Y4WDKXERQD6ECGCKWA6NX3TBE73WDYJSXUX2BKQ, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.533589044 (70h46m2.404672089s) [noise=true , address=145.79.227.23:4547 , piece_id=4366WJBATQGLNJXM2V63Z6WD42YZNCYA2DMXT7D77TSTYSBVX7RA, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.534232690 (70h46m2.405315735s) [noise=true , address=145.79.227.43:4157 , piece_id=7KTRATOFVKLDJQPDZL35YERZZTNWRQS334T73AQVC55RXQQ4JWTQ, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.534327270 (70h46m2.405410315s) [noise=true , address=145.79.227.21:4474 , piece_id=BZZX3YTKJKQT3KAWDEUF2V46J7B4COD7CZPZ2H4SAFNEMC3NHVCQ, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.534749855 (70h46m2.4058329s) [noise=true , address=77.90.31.18:4806 , piece_id=M5ENXW4OVQOS7RUIG62OIAA4DXJERWCLHAA6JOANRHRJGBN6ADNQ, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.534905431 (70h46m2.405988476s) [noise=true , address=145.79.227.24:4581 , piece_id=EUHJEAD536QMDG7UGB2MD2NXABD5U4OXSF2REQKTHEZWA7KEO72A, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.535348254 (70h46m2.406431299s) [noise=true , address=145.79.227.16:4910 , piece_id=HKGXMBLAD2XDZIBRL3OVJPUD4EEZ2R4RPD4V2MZ7V34KI2WUW5QQ, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.535719391 (70h46m2.406802436s) [noise=true , address=145.79.227.32:4836 , piece_id=4O6OJM23CPVD4DUJZECFF2LVM5DCLBMCVOUFU7V2LRVC2B7LHSLQ, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.535932487 (70h46m2.407015562s) [noise=true , address=77.90.31.14:4663 , piece_id=4NPSCMGDDR6J7YPQPJDZ3RXWX4V7PD5BR7IBJVZPOZSJXVXRKXMQ, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.536686312 (70h46m2.407769357s) [noise=true , address=145.79.227.34:4905 , piece_id=43NXHL7VBVKVFSDKOPBUOZJXOHL3HMYANOOBKPDACNUBOAQ4SBMA, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.537242150 (70h46m2.408325195s) [noise=true , address=145.79.227.22:4511 , piece_id=K37NXW5QF4PSP4HCVD6XMJIOSDEPTLDY3URHKBYGZFLICD44AXNA, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.538005784 (70h46m2.409088829s) [noise=true , address=145.79.227.39:4013 , piece_id=AIIRUZSHTXIJ3Q4HKSTJ6LTHCAQKLI3CIY4KJQW3GKL3IL6ZR6GQ, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.538636735 (70h46m2.409719791s) [noise=true , address=145.79.227.28:4726 , piece_id=CB6RMZTHGV5CK4HCAHVH3BLKI545Q2V4QRWXS5OA4S2IUB4LFBEQ, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:04.349584554 (70h46m4.220667599s) [upload=61969]: Next returned nil segment
Dec 8 14:15:04.349605775 (70h46m4.22068882s) [upload=61969]: Waiting for error group managing segments...
Dec 8 14:15:04.784251928 (70h46m4.655334973s) [noise=true , address=77.90.31.14:4663 , piece_id=4NPSCMGDDR6J7YPQPJDZ3RXWX4V7PD5BR7IBJVZPOZSJXVXRKXMQ, seg_pos={0 0}, upload=61969]: Done uploading piece. err: <nil>
Dec 8 14:15:04.837322415 (70h46m4.70840546s) [noise=true , address=77.90.31.18:4806 , piece_id=M5ENXW4OVQOS7RUIG62OIAA4DXJERWCLHAA6JOANRHRJGBN6ADNQ, seg_pos={0 0}, upload=61969]: Done uploading piece. err: <nil>
Dec 8 14:15:05.587735312 (70h46m5.458818347s) [noise=true , address=145.79.227.32:4836 , piece_id=4O6OJM23CPVD4DUJZECFF2LVM5DCLBMCVOUFU7V2LRVC2B7LHSLQ, seg_pos={0 0}, upload=61969]: Done uploading piece. err: <nil>
Dec 8 14:15:05.588371883 (70h46m5.459454929s) [noise=true , address=145.79.227.22:4511 , piece_id=K37NXW5QF4PSP4HCVD6XMJIOSDEPTLDY3URHKBYGZFLICD44AXNA, seg_pos={0 0}, upload=61969]: Done uploading piece. err: <nil>
Dec 8 14:15:05.607297883 (70h46m5.478380928s) [noise=true , address=145.79.227.24:4581 , piece_id=EUHJEAD536QMDG7UGB2MD2NXABD5U4OXSF2REQKTHEZWA7KEO72A, seg_pos={0 0}, upload=61969]: Done uploading piece. err: <nil>
Dec 8 14:15:05.630124479 (70h46m5.501207524s) [noise=true , address=145.79.227.33:4871 , piece_id=F7NB65E364EI6Y4WDKXERQD6ECGCKWA6NX3TBE73WDYJSXUX2BKQ, seg_pos={0 0}, upload=61969]: Done uploading piece. err: <nil>
Dec 8 14:15:05.630590277 (70h46m5.501673322s) [noise=true , address=145.79.227.43:4157 , piece_id=7KTRATOFVKLDJQPDZL35YERZZTNWRQS334T73AQVC55RXQQ4JWTQ, seg_pos={0 0}, upload=61969]: Done uploading piece. err: <nil>
Dec 8 14:15:05.630817319 (70h46m5.501900364s) [noise=true , address=145.79.227.20:4439 , piece_id=ELOM2UKEDSKKNWILB2I325A5OTSQAKIILQHP345RSTC2LXECKTVQ, seg_pos={0 0}, upload=61969]: Done uploading piece. err: <nil>
Dec 8 14:15:05.672033613 (70h46m5.543116668s) [noise=true , address=145.79.227.21:4474 , piece_id=BZZX3YTKJKQT3KAWDEUF2V46J7B4COD7CZPZ2H4SAFNEMC3NHVCQ, seg_pos={0 0}, upload=61969]: Done uploading piece. err: <nil>
Dec 8 14:15:05.735071049 (70h46m5.606154094s) [noise=true , address=145.79.227.32:4835 , piece_id=HPWNW5AC354GTIRDSCEU3NN3ULKBIUUMPRHLVNCECVKGYA5TR2DQ, seg_pos={0 0}, upload=61969]: Done uploading piece. err: <nil>
Dec 8 14:15:05.817333182 (70h46m5.688416237s) [noise=true , address=145.79.227.34:4905 , piece_id=43NXHL7VBVKVFSDKOPBUOZJXOHL3HMYANOOBKPDACNUBOAQ4SBMA, seg_pos={0 0}, upload=61969]: Done uploading piece. err: <nil>
Dec 8 14:15:05.822913809 (70h46m5.693996854s) [noise=true , address=145.79.227.41:4087 , piece_id=OZA2MCHUMIJYWXHS3KAVJRVTJ6LWAW5XR257OGAZRNPIOGMETZRA, seg_pos={0 0}, upload=61969]: Done uploading piece. err: <nil>
Dec 8 14:15:05.823926878 (70h46m5.695009923s) [noise=true , address=145.79.227.23:4547 , piece_id=4366WJBATQGLNJXM2V63Z6WD42YZNCYA2DMXT7D77TSTYSBVX7RA, seg_pos={0 0}, upload=61969]: Done uploading piece. err: <nil>
Dec 8 14:15:05.823938099 (70h46m5.695021154s) [seg_pos={0 0}, upload=61969]: Segment reached optimal threshold of 13 pieces.
Dec 8 14:15:05.824198556 (70h46m5.695281601s) [noise=true , address=145.79.227.39:4013 , piece_id=AIIRUZSHTXIJ3Q4HKSTJ6LTHCAQKLI3CIY4KJQW3GKL3IL6ZR6GQ, seg_pos={0 0}, upload=61969]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: expected piece hash; context canceled; EOF
Dec 8 14:15:05.824210698 (70h46m5.695293743s) [noise=true , address=145.79.227.39:4013 , piece_id=AIIRUZSHTXIJ3Q4HKSTJ6LTHCAQKLI3CIY4KJQW3GKL3IL6ZR6GQ, seg_pos={0 0}, upload=61969]: Long tail cancellation due to optimal threshold
Dec 8 14:15:05.824416240 (70h46m5.695499285s) [noise=true , address=145.79.227.28:4726 , piece_id=CB6RMZTHGV5CK4HCAHVH3BLKI545Q2V4QRWXS5OA4S2IUB4LFBEQ, seg_pos={0 0}, upload=61969]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: expected piece hash; context canceled; EOF
Dec 8 14:15:05.824448682 (70h46m5.695531717s) [noise=true , address=145.79.227.28:4726 , piece_id=CB6RMZTHGV5CK4HCAHVH3BLKI545Q2V4QRWXS5OA4S2IUB4LFBEQ, seg_pos={0 0}, upload=61969]: Long tail cancellation due to optimal threshold
Dec 8 14:15:05.824649695 (70h46m5.69573274s) [noise=true , address=145.79.227.16:4910 , piece_id=HKGXMBLAD2XDZIBRL3OVJPUD4EEZ2R4RPD4V2MZ7V34KI2WUW5QQ, seg_pos={0 0}, upload=61969]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: expected piece hash; context canceled; EOF
Dec 8 14:15:05.824662118 (70h46m5.695745163s) [noise=true , address=145.79.227.16:4910 , piece_id=HKGXMBLAD2XDZIBRL3OVJPUD4EEZ2R4RPD4V2MZ7V34KI2WUW5QQ, seg_pos={0 0}, upload=61969]: Long tail cancellation due to optimal threshold
Dec 8 14:15:05.824696944 (70h46m5.695779989s) [seg_pos={0 0}, upload=61969]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 14:15:05.824818956 (70h46m5.695902001s) [upload=61969]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 14:15:05.824827082 (70h46m5.695910127s) [upload=61969]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 14:17:06.241593111 (70h48m6.112676156s) [upload=61969]: Done uploading segments...
Dec 8 14:17:06.241713471 (70h48m6.112796516s) [upload=61969]: Upload finished. err: metaclient: context deadline exceeded
Duration: 8.507194106s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
2.984361662s trace:
https://jaeger.dev.storj.io/trace/29e54bf3415d9356
FAIL (2025-12-08 14:24:49 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 2.984392062s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................] duration:
7.897078292s trace:
https://jaeger.dev.storj.io/trace/406d13f34dbd7740
FAIL (2025-12-08 14:24:38 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 7.897099201s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
26 out of 60 is failed (threshold=3) [.XXX..XX....X.X..X.X..XX....X..........X.XX.XXX.X..XXX.X.XXX] duration:
1m0.000772287s trace:
https://jaeger.dev.storj.io/trace/fe07c3139e7e198
FAIL (2025-12-08 14:24:33 UTC)
orbiter warning: context deadline exceeded
Duration: 1m0.000811732s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
14.634858625s trace:
https://jaeger.dev.storj.io/trace/79da76e828604af6
FAIL (2025-12-08 14:24:30 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 8 14:15:00.792576635 (70h46m0.66365968s) [upload=61969]: Starting upload
Dec 8 14:15:00.792651899 (70h46m0.663734975s) [upload=61969]: Done starting upload
Dec 8 14:15:00.792689761 (70h46m0.663772797s) [upload=61969]: Uploading segments...
Dec 8 14:15:00.792980154 (70h46m0.664063209s) [upload=61969]: Got next segment. Inline: false
Dec 8 14:15:00.793097006 (70h46m0.664180051s) [upload=61969]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 14:15:00.793104832 (70h46m0.664187887s) [upload=61969]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 14:15:02.526579968 (70h46m2.397663013s) [seg_pos={0 0}, upload=61969]: Begin upload segment...
Dec 8 14:15:02.526648298 (70h46m2.397731353s) [seg_pos={0 0}, upload=61969]: Done begin upload segment.
Dec 8 14:15:02.526688614 (70h46m2.397771659s) [noise=true , address=145.79.227.32:4835 , piece_id=HPWNW5AC354GTIRDSCEU3NN3ULKBIUUMPRHLVNCECVKGYA5TR2DQ, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.529494027 (70h46m2.400577102s) [noise=true , address=145.79.227.41:4087 , piece_id=OZA2MCHUMIJYWXHS3KAVJRVTJ6LWAW5XR257OGAZRNPIOGMETZRA, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.531073364 (70h46m2.402156419s) [noise=true , address=145.79.227.20:4439 , piece_id=ELOM2UKEDSKKNWILB2I325A5OTSQAKIILQHP345RSTC2LXECKTVQ, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.531556935 (70h46m2.40263998s) [noise=true , address=145.79.227.33:4871 , piece_id=F7NB65E364EI6Y4WDKXERQD6ECGCKWA6NX3TBE73WDYJSXUX2BKQ, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.533589044 (70h46m2.404672089s) [noise=true , address=145.79.227.23:4547 , piece_id=4366WJBATQGLNJXM2V63Z6WD42YZNCYA2DMXT7D77TSTYSBVX7RA, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.534232690 (70h46m2.405315735s) [noise=true , address=145.79.227.43:4157 , piece_id=7KTRATOFVKLDJQPDZL35YERZZTNWRQS334T73AQVC55RXQQ4JWTQ, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.534327270 (70h46m2.405410315s) [noise=true , address=145.79.227.21:4474 , piece_id=BZZX3YTKJKQT3KAWDEUF2V46J7B4COD7CZPZ2H4SAFNEMC3NHVCQ, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.534749855 (70h46m2.4058329s) [noise=true , address=77.90.31.18:4806 , piece_id=M5ENXW4OVQOS7RUIG62OIAA4DXJERWCLHAA6JOANRHRJGBN6ADNQ, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.534905431 (70h46m2.405988476s) [noise=true , address=145.79.227.24:4581 , piece_id=EUHJEAD536QMDG7UGB2MD2NXABD5U4OXSF2REQKTHEZWA7KEO72A, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.535348254 (70h46m2.406431299s) [noise=true , address=145.79.227.16:4910 , piece_id=HKGXMBLAD2XDZIBRL3OVJPUD4EEZ2R4RPD4V2MZ7V34KI2WUW5QQ, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.535719391 (70h46m2.406802436s) [noise=true , address=145.79.227.32:4836 , piece_id=4O6OJM23CPVD4DUJZECFF2LVM5DCLBMCVOUFU7V2LRVC2B7LHSLQ, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.535932487 (70h46m2.407015562s) [noise=true , address=77.90.31.14:4663 , piece_id=4NPSCMGDDR6J7YPQPJDZ3RXWX4V7PD5BR7IBJVZPOZSJXVXRKXMQ, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.536686312 (70h46m2.407769357s) [noise=true , address=145.79.227.34:4905 , piece_id=43NXHL7VBVKVFSDKOPBUOZJXOHL3HMYANOOBKPDACNUBOAQ4SBMA, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.537242150 (70h46m2.408325195s) [noise=true , address=145.79.227.22:4511 , piece_id=K37NXW5QF4PSP4HCVD6XMJIOSDEPTLDY3URHKBYGZFLICD44AXNA, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.538005784 (70h46m2.409088829s) [noise=true , address=145.79.227.39:4013 , piece_id=AIIRUZSHTXIJ3Q4HKSTJ6LTHCAQKLI3CIY4KJQW3GKL3IL6ZR6GQ, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.538636735 (70h46m2.409719791s) [noise=true , address=145.79.227.28:4726 , piece_id=CB6RMZTHGV5CK4HCAHVH3BLKI545Q2V4QRWXS5OA4S2IUB4LFBEQ, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:04.349584554 (70h46m4.220667599s) [upload=61969]: Next returned nil segment
Dec 8 14:15:04.349605775 (70h46m4.22068882s) [upload=61969]: Waiting for error group managing segments...
Dec 8 14:15:04.784251928 (70h46m4.655334973s) [noise=true , address=77.90.31.14:4663 , piece_id=4NPSCMGDDR6J7YPQPJDZ3RXWX4V7PD5BR7IBJVZPOZSJXVXRKXMQ, seg_pos={0 0}, upload=61969]: Done uploading piece. err: <nil>
Dec 8 14:15:04.837322415 (70h46m4.70840546s) [noise=true , address=77.90.31.18:4806 , piece_id=M5ENXW4OVQOS7RUIG62OIAA4DXJERWCLHAA6JOANRHRJGBN6ADNQ, seg_pos={0 0}, upload=61969]: Done uploading piece. err: <nil>
Dec 8 14:15:05.587735312 (70h46m5.458818347s) [noise=true , address=145.79.227.32:4836 , piece_id=4O6OJM23CPVD4DUJZECFF2LVM5DCLBMCVOUFU7V2LRVC2B7LHSLQ, seg_pos={0 0}, upload=61969]: Done uploading piece. err: <nil>
Dec 8 14:15:05.588371883 (70h46m5.459454929s) [noise=true , address=145.79.227.22:4511 , piece_id=K37NXW5QF4PSP4HCVD6XMJIOSDEPTLDY3URHKBYGZFLICD44AXNA, seg_pos={0 0}, upload=61969]: Done uploading piece. err: <nil>
Dec 8 14:15:05.607297883 (70h46m5.478380928s) [noise=true , address=145.79.227.24:4581 , piece_id=EUHJEAD536QMDG7UGB2MD2NXABD5U4OXSF2REQKTHEZWA7KEO72A, seg_pos={0 0}, upload=61969]: Done uploading piece. err: <nil>
Dec 8 14:15:05.630124479 (70h46m5.501207524s) [noise=true , address=145.79.227.33:4871 , piece_id=F7NB65E364EI6Y4WDKXERQD6ECGCKWA6NX3TBE73WDYJSXUX2BKQ, seg_pos={0 0}, upload=61969]: Done uploading piece. err: <nil>
Dec 8 14:15:05.630590277 (70h46m5.501673322s) [noise=true , address=145.79.227.43:4157 , piece_id=7KTRATOFVKLDJQPDZL35YERZZTNWRQS334T73AQVC55RXQQ4JWTQ, seg_pos={0 0}, upload=61969]: Done uploading piece. err: <nil>
Dec 8 14:15:05.630817319 (70h46m5.501900364s) [noise=true , address=145.79.227.20:4439 , piece_id=ELOM2UKEDSKKNWILB2I325A5OTSQAKIILQHP345RSTC2LXECKTVQ, seg_pos={0 0}, upload=61969]: Done uploading piece. err: <nil>
Dec 8 14:15:05.672033613 (70h46m5.543116668s) [noise=true , address=145.79.227.21:4474 , piece_id=BZZX3YTKJKQT3KAWDEUF2V46J7B4COD7CZPZ2H4SAFNEMC3NHVCQ, seg_pos={0 0}, upload=61969]: Done uploading piece. err: <nil>
Dec 8 14:15:05.735071049 (70h46m5.606154094s) [noise=true , address=145.79.227.32:4835 , piece_id=HPWNW5AC354GTIRDSCEU3NN3ULKBIUUMPRHLVNCECVKGYA5TR2DQ, seg_pos={0 0}, upload=61969]: Done uploading piece. err: <nil>
Dec 8 14:15:05.817333182 (70h46m5.688416237s) [noise=true , address=145.79.227.34:4905 , piece_id=43NXHL7VBVKVFSDKOPBUOZJXOHL3HMYANOOBKPDACNUBOAQ4SBMA, seg_pos={0 0}, upload=61969]: Done uploading piece. err: <nil>
Dec 8 14:15:05.822913809 (70h46m5.693996854s) [noise=true , address=145.79.227.41:4087 , piece_id=OZA2MCHUMIJYWXHS3KAVJRVTJ6LWAW5XR257OGAZRNPIOGMETZRA, seg_pos={0 0}, upload=61969]: Done uploading piece. err: <nil>
Dec 8 14:15:05.823926878 (70h46m5.695009923s) [noise=true , address=145.79.227.23:4547 , piece_id=4366WJBATQGLNJXM2V63Z6WD42YZNCYA2DMXT7D77TSTYSBVX7RA, seg_pos={0 0}, upload=61969]: Done uploading piece. err: <nil>
Dec 8 14:15:05.823938099 (70h46m5.695021154s) [seg_pos={0 0}, upload=61969]: Segment reached optimal threshold of 13 pieces.
Dec 8 14:15:05.824198556 (70h46m5.695281601s) [noise=true , address=145.79.227.39:4013 , piece_id=AIIRUZSHTXIJ3Q4HKSTJ6LTHCAQKLI3CIY4KJQW3GKL3IL6ZR6GQ, seg_pos={0 0}, upload=61969]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: expected piece hash; context canceled; EOF
Dec 8 14:15:05.824210698 (70h46m5.695293743s) [noise=true , address=145.79.227.39:4013 , piece_id=AIIRUZSHTXIJ3Q4HKSTJ6LTHCAQKLI3CIY4KJQW3GKL3IL6ZR6GQ, seg_pos={0 0}, upload=61969]: Long tail cancellation due to optimal threshold
Dec 8 14:15:05.824416240 (70h46m5.695499285s) [noise=true , address=145.79.227.28:4726 , piece_id=CB6RMZTHGV5CK4HCAHVH3BLKI545Q2V4QRWXS5OA4S2IUB4LFBEQ, seg_pos={0 0}, upload=61969]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: expected piece hash; context canceled; EOF
Dec 8 14:15:05.824448682 (70h46m5.695531717s) [noise=true , address=145.79.227.28:4726 , piece_id=CB6RMZTHGV5CK4HCAHVH3BLKI545Q2V4QRWXS5OA4S2IUB4LFBEQ, seg_pos={0 0}, upload=61969]: Long tail cancellation due to optimal threshold
Dec 8 14:15:05.824649695 (70h46m5.69573274s) [noise=true , address=145.79.227.16:4910 , piece_id=HKGXMBLAD2XDZIBRL3OVJPUD4EEZ2R4RPD4V2MZ7V34KI2WUW5QQ, seg_pos={0 0}, upload=61969]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: expected piece hash; context canceled; EOF
Dec 8 14:15:05.824662118 (70h46m5.695745163s) [noise=true , address=145.79.227.16:4910 , piece_id=HKGXMBLAD2XDZIBRL3OVJPUD4EEZ2R4RPD4V2MZ7V34KI2WUW5QQ, seg_pos={0 0}, upload=61969]: Long tail cancellation due to optimal threshold
Dec 8 14:15:05.824696944 (70h46m5.695779989s) [seg_pos={0 0}, upload=61969]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 14:15:05.824818956 (70h46m5.695902001s) [upload=61969]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 14:15:05.824827082 (70h46m5.695910127s) [upload=61969]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 14:17:06.241593111 (70h48m6.112676156s) [upload=61969]: Done uploading segments...
Dec 8 14:17:06.241713471 (70h48m6.112796516s) [upload=61969]: Upload finished. err: metaclient: context deadline exceeded
Duration: 14.634883933s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
60 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.000912883s trace:
https://jaeger.dev.storj.io/trace/14079119cd1f278d
FAIL (2025-12-08 14:24:30 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000941006s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
23 out of 60 is failed (threshold=3) [....X....X.X.............X.X..X.XXXX...XXXXXXX.X.X....X.XXX.] duration:
23.487515233s trace:
https://jaeger.dev.storj.io/trace/69d5a5cd4545e425
FAIL (2025-12-08 14:24:29 UTC)
orbiter warning: context deadline exceeded
Duration: 23.487541534s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
8.202051163s trace:
https://jaeger.dev.storj.io/trace/406d13f34dbd7740
FAIL (2025-12-08 14:24:12 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 8.202091801s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
60 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.0025509s trace:
https://jaeger.dev.storj.io/trace/469e3c022ebd8bbf
FAIL (2025-12-08 14:24:08 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.002656501s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
60 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001288678s trace:
https://jaeger.dev.storj.io/trace/3fba1a7b67403279
FAIL (2025-12-08 14:23:53 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001335668s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
23 out of 60 is failed (threshold=3) [.....X....X.X.............X.X..X.XXXX...XXXXXXX.X.X....X.XXX] duration:
30.001496132s trace:
https://jaeger.dev.storj.io/trace/69d5a5cd4545e425
FAIL (2025-12-08 14:23:52 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001523865s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
2 out of 60 is failed (threshold=3) [X...........................X...............................] duration:
2.929920708s trace:
https://jaeger.dev.storj.io/trace/29e54bf3415d9356
FAIL (2025-12-08 14:23:39 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 2.929945285s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
7.96013964s trace:
https://jaeger.dev.storj.io/trace/406d13f34dbd7740
FAIL (2025-12-08 14:23:36 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 7.960170569s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
25 out of 60 is failed (threshold=3) [..XXX..XX....X.X..X.X..XX....X..........X.XX.XXX.X..XXX.X.XX] duration:
1m0.001184082s trace:
https://jaeger.dev.storj.io/trace/2b7b05dce2c6fe58
FAIL (2025-12-08 14:23:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001199942s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
22 out of 60 is failed (threshold=3) [......X....X.X.............X.X..X.XXXX...XXXXXXX.X.X....X.XX] duration:
30.001328392s trace:
https://jaeger.dev.storj.io/trace/220e797a1cb72904
FAIL (2025-12-08 14:23:15 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001383005s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
9.62365905s trace:
https://jaeger.dev.storj.io/trace/79da76e828604af6
FAIL (2025-12-08 14:23:14 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 8 14:15:00.792576635 (70h46m0.66365968s) [upload=61969]: Starting upload
Dec 8 14:15:00.792651899 (70h46m0.663734975s) [upload=61969]: Done starting upload
Dec 8 14:15:00.792689761 (70h46m0.663772797s) [upload=61969]: Uploading segments...
Dec 8 14:15:00.792980154 (70h46m0.664063209s) [upload=61969]: Got next segment. Inline: false
Dec 8 14:15:00.793097006 (70h46m0.664180051s) [upload=61969]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 14:15:00.793104832 (70h46m0.664187887s) [upload=61969]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 14:15:02.526579968 (70h46m2.397663013s) [seg_pos={0 0}, upload=61969]: Begin upload segment...
Dec 8 14:15:02.526648298 (70h46m2.397731353s) [seg_pos={0 0}, upload=61969]: Done begin upload segment.
Dec 8 14:15:02.526688614 (70h46m2.397771659s) [noise=true , address=145.79.227.32:4835 , piece_id=HPWNW5AC354GTIRDSCEU3NN3ULKBIUUMPRHLVNCECVKGYA5TR2DQ, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.529494027 (70h46m2.400577102s) [noise=true , address=145.79.227.41:4087 , piece_id=OZA2MCHUMIJYWXHS3KAVJRVTJ6LWAW5XR257OGAZRNPIOGMETZRA, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.531073364 (70h46m2.402156419s) [noise=true , address=145.79.227.20:4439 , piece_id=ELOM2UKEDSKKNWILB2I325A5OTSQAKIILQHP345RSTC2LXECKTVQ, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.531556935 (70h46m2.40263998s) [noise=true , address=145.79.227.33:4871 , piece_id=F7NB65E364EI6Y4WDKXERQD6ECGCKWA6NX3TBE73WDYJSXUX2BKQ, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.533589044 (70h46m2.404672089s) [noise=true , address=145.79.227.23:4547 , piece_id=4366WJBATQGLNJXM2V63Z6WD42YZNCYA2DMXT7D77TSTYSBVX7RA, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.534232690 (70h46m2.405315735s) [noise=true , address=145.79.227.43:4157 , piece_id=7KTRATOFVKLDJQPDZL35YERZZTNWRQS334T73AQVC55RXQQ4JWTQ, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.534327270 (70h46m2.405410315s) [noise=true , address=145.79.227.21:4474 , piece_id=BZZX3YTKJKQT3KAWDEUF2V46J7B4COD7CZPZ2H4SAFNEMC3NHVCQ, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.534749855 (70h46m2.4058329s) [noise=true , address=77.90.31.18:4806 , piece_id=M5ENXW4OVQOS7RUIG62OIAA4DXJERWCLHAA6JOANRHRJGBN6ADNQ, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.534905431 (70h46m2.405988476s) [noise=true , address=145.79.227.24:4581 , piece_id=EUHJEAD536QMDG7UGB2MD2NXABD5U4OXSF2REQKTHEZWA7KEO72A, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.535348254 (70h46m2.406431299s) [noise=true , address=145.79.227.16:4910 , piece_id=HKGXMBLAD2XDZIBRL3OVJPUD4EEZ2R4RPD4V2MZ7V34KI2WUW5QQ, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.535719391 (70h46m2.406802436s) [noise=true , address=145.79.227.32:4836 , piece_id=4O6OJM23CPVD4DUJZECFF2LVM5DCLBMCVOUFU7V2LRVC2B7LHSLQ, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.535932487 (70h46m2.407015562s) [noise=true , address=77.90.31.14:4663 , piece_id=4NPSCMGDDR6J7YPQPJDZ3RXWX4V7PD5BR7IBJVZPOZSJXVXRKXMQ, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.536686312 (70h46m2.407769357s) [noise=true , address=145.79.227.34:4905 , piece_id=43NXHL7VBVKVFSDKOPBUOZJXOHL3HMYANOOBKPDACNUBOAQ4SBMA, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.537242150 (70h46m2.408325195s) [noise=true , address=145.79.227.22:4511 , piece_id=K37NXW5QF4PSP4HCVD6XMJIOSDEPTLDY3URHKBYGZFLICD44AXNA, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.538005784 (70h46m2.409088829s) [noise=true , address=145.79.227.39:4013 , piece_id=AIIRUZSHTXIJ3Q4HKSTJ6LTHCAQKLI3CIY4KJQW3GKL3IL6ZR6GQ, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.538636735 (70h46m2.409719791s) [noise=true , address=145.79.227.28:4726 , piece_id=CB6RMZTHGV5CK4HCAHVH3BLKI545Q2V4QRWXS5OA4S2IUB4LFBEQ, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:04.349584554 (70h46m4.220667599s) [upload=61969]: Next returned nil segment
Dec 8 14:15:04.349605775 (70h46m4.22068882s) [upload=61969]: Waiting for error group managing segments...
Dec 8 14:15:04.784251928 (70h46m4.655334973s) [noise=true , address=77.90.31.14:4663 , piece_id=4NPSCMGDDR6J7YPQPJDZ3RXWX4V7PD5BR7IBJVZPOZSJXVXRKXMQ, seg_pos={0 0}, upload=61969]: Done uploading piece. err: <nil>
Dec 8 14:15:04.837322415 (70h46m4.70840546s) [noise=true , address=77.90.31.18:4806 , piece_id=M5ENXW4OVQOS7RUIG62OIAA4DXJERWCLHAA6JOANRHRJGBN6ADNQ, seg_pos={0 0}, upload=61969]: Done uploading piece. err: <nil>
Dec 8 14:15:05.587735312 (70h46m5.458818347s) [noise=true , address=145.79.227.32:4836 , piece_id=4O6OJM23CPVD4DUJZECFF2LVM5DCLBMCVOUFU7V2LRVC2B7LHSLQ, seg_pos={0 0}, upload=61969]: Done uploading piece. err: <nil>
Dec 8 14:15:05.588371883 (70h46m5.459454929s) [noise=true , address=145.79.227.22:4511 , piece_id=K37NXW5QF4PSP4HCVD6XMJIOSDEPTLDY3URHKBYGZFLICD44AXNA, seg_pos={0 0}, upload=61969]: Done uploading piece. err: <nil>
Dec 8 14:15:05.607297883 (70h46m5.478380928s) [noise=true , address=145.79.227.24:4581 , piece_id=EUHJEAD536QMDG7UGB2MD2NXABD5U4OXSF2REQKTHEZWA7KEO72A, seg_pos={0 0}, upload=61969]: Done uploading piece. err: <nil>
Dec 8 14:15:05.630124479 (70h46m5.501207524s) [noise=true , address=145.79.227.33:4871 , piece_id=F7NB65E364EI6Y4WDKXERQD6ECGCKWA6NX3TBE73WDYJSXUX2BKQ, seg_pos={0 0}, upload=61969]: Done uploading piece. err: <nil>
Dec 8 14:15:05.630590277 (70h46m5.501673322s) [noise=true , address=145.79.227.43:4157 , piece_id=7KTRATOFVKLDJQPDZL35YERZZTNWRQS334T73AQVC55RXQQ4JWTQ, seg_pos={0 0}, upload=61969]: Done uploading piece. err: <nil>
Dec 8 14:15:05.630817319 (70h46m5.501900364s) [noise=true , address=145.79.227.20:4439 , piece_id=ELOM2UKEDSKKNWILB2I325A5OTSQAKIILQHP345RSTC2LXECKTVQ, seg_pos={0 0}, upload=61969]: Done uploading piece. err: <nil>
Dec 8 14:15:05.672033613 (70h46m5.543116668s) [noise=true , address=145.79.227.21:4474 , piece_id=BZZX3YTKJKQT3KAWDEUF2V46J7B4COD7CZPZ2H4SAFNEMC3NHVCQ, seg_pos={0 0}, upload=61969]: Done uploading piece. err: <nil>
Dec 8 14:15:05.735071049 (70h46m5.606154094s) [noise=true , address=145.79.227.32:4835 , piece_id=HPWNW5AC354GTIRDSCEU3NN3ULKBIUUMPRHLVNCECVKGYA5TR2DQ, seg_pos={0 0}, upload=61969]: Done uploading piece. err: <nil>
Dec 8 14:15:05.817333182 (70h46m5.688416237s) [noise=true , address=145.79.227.34:4905 , piece_id=43NXHL7VBVKVFSDKOPBUOZJXOHL3HMYANOOBKPDACNUBOAQ4SBMA, seg_pos={0 0}, upload=61969]: Done uploading piece. err: <nil>
Dec 8 14:15:05.822913809 (70h46m5.693996854s) [noise=true , address=145.79.227.41:4087 , piece_id=OZA2MCHUMIJYWXHS3KAVJRVTJ6LWAW5XR257OGAZRNPIOGMETZRA, seg_pos={0 0}, upload=61969]: Done uploading piece. err: <nil>
Dec 8 14:15:05.823926878 (70h46m5.695009923s) [noise=true , address=145.79.227.23:4547 , piece_id=4366WJBATQGLNJXM2V63Z6WD42YZNCYA2DMXT7D77TSTYSBVX7RA, seg_pos={0 0}, upload=61969]: Done uploading piece. err: <nil>
Dec 8 14:15:05.823938099 (70h46m5.695021154s) [seg_pos={0 0}, upload=61969]: Segment reached optimal threshold of 13 pieces.
Dec 8 14:15:05.824198556 (70h46m5.695281601s) [noise=true , address=145.79.227.39:4013 , piece_id=AIIRUZSHTXIJ3Q4HKSTJ6LTHCAQKLI3CIY4KJQW3GKL3IL6ZR6GQ, seg_pos={0 0}, upload=61969]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: expected piece hash; context canceled; EOF
Dec 8 14:15:05.824210698 (70h46m5.695293743s) [noise=true , address=145.79.227.39:4013 , piece_id=AIIRUZSHTXIJ3Q4HKSTJ6LTHCAQKLI3CIY4KJQW3GKL3IL6ZR6GQ, seg_pos={0 0}, upload=61969]: Long tail cancellation due to optimal threshold
Dec 8 14:15:05.824416240 (70h46m5.695499285s) [noise=true , address=145.79.227.28:4726 , piece_id=CB6RMZTHGV5CK4HCAHVH3BLKI545Q2V4QRWXS5OA4S2IUB4LFBEQ, seg_pos={0 0}, upload=61969]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: expected piece hash; context canceled; EOF
Dec 8 14:15:05.824448682 (70h46m5.695531717s) [noise=true , address=145.79.227.28:4726 , piece_id=CB6RMZTHGV5CK4HCAHVH3BLKI545Q2V4QRWXS5OA4S2IUB4LFBEQ, seg_pos={0 0}, upload=61969]: Long tail cancellation due to optimal threshold
Dec 8 14:15:05.824649695 (70h46m5.69573274s) [noise=true , address=145.79.227.16:4910 , piece_id=HKGXMBLAD2XDZIBRL3OVJPUD4EEZ2R4RPD4V2MZ7V34KI2WUW5QQ, seg_pos={0 0}, upload=61969]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: expected piece hash; context canceled; EOF
Dec 8 14:15:05.824662118 (70h46m5.695745163s) [noise=true , address=145.79.227.16:4910 , piece_id=HKGXMBLAD2XDZIBRL3OVJPUD4EEZ2R4RPD4V2MZ7V34KI2WUW5QQ, seg_pos={0 0}, upload=61969]: Long tail cancellation due to optimal threshold
Dec 8 14:15:05.824696944 (70h46m5.695779989s) [seg_pos={0 0}, upload=61969]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 14:15:05.824818956 (70h46m5.695902001s) [upload=61969]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 14:15:05.824827082 (70h46m5.695910127s) [upload=61969]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 14:17:06.241593111 (70h48m6.112676156s) [upload=61969]: Done uploading segments...
Dec 8 14:17:06.241713471 (70h48m6.112796516s) [upload=61969]: Upload finished. err: metaclient: context deadline exceeded
Duration: 9.623783477s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
2 out of 60 is failed (threshold=3) [.X...........................X..............................] duration:
3.074322086s trace:
https://jaeger.dev.storj.io/trace/29e54bf3415d9356
FAIL (2025-12-08 14:23:13 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 3.074346463s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
60 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001934289s trace:
https://jaeger.dev.storj.io/trace/5c711c0af2063353
FAIL (2025-12-08 14:22:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001958426s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
24 out of 60 is failed (threshold=3) [...XXX..XX....X.X..X.X..XX....X..........X.XX.XXX.X..XXX.X.X] duration:
1m0.00183817s trace:
https://jaeger.dev.storj.io/trace/6e6966afcab3418e
FAIL (2025-12-08 14:22:32 UTC)
orbiter warning: context deadline exceeded
Duration: 1m0.001854701s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
9.90430779s trace:
https://jaeger.dev.storj.io/trace/79da76e828604af6
FAIL (2025-12-08 14:22:31 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 8 14:15:00.792576635 (70h46m0.66365968s) [upload=61969]: Starting upload
Dec 8 14:15:00.792651899 (70h46m0.663734975s) [upload=61969]: Done starting upload
Dec 8 14:15:00.792689761 (70h46m0.663772797s) [upload=61969]: Uploading segments...
Dec 8 14:15:00.792980154 (70h46m0.664063209s) [upload=61969]: Got next segment. Inline: false
Dec 8 14:15:00.793097006 (70h46m0.664180051s) [upload=61969]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 14:15:00.793104832 (70h46m0.664187887s) [upload=61969]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 14:15:02.526579968 (70h46m2.397663013s) [seg_pos={0 0}, upload=61969]: Begin upload segment...
Dec 8 14:15:02.526648298 (70h46m2.397731353s) [seg_pos={0 0}, upload=61969]: Done begin upload segment.
Dec 8 14:15:02.526688614 (70h46m2.397771659s) [noise=true , address=145.79.227.32:4835 , piece_id=HPWNW5AC354GTIRDSCEU3NN3ULKBIUUMPRHLVNCECVKGYA5TR2DQ, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.529494027 (70h46m2.400577102s) [noise=true , address=145.79.227.41:4087 , piece_id=OZA2MCHUMIJYWXHS3KAVJRVTJ6LWAW5XR257OGAZRNPIOGMETZRA, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.531073364 (70h46m2.402156419s) [noise=true , address=145.79.227.20:4439 , piece_id=ELOM2UKEDSKKNWILB2I325A5OTSQAKIILQHP345RSTC2LXECKTVQ, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.531556935 (70h46m2.40263998s) [noise=true , address=145.79.227.33:4871 , piece_id=F7NB65E364EI6Y4WDKXERQD6ECGCKWA6NX3TBE73WDYJSXUX2BKQ, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.533589044 (70h46m2.404672089s) [noise=true , address=145.79.227.23:4547 , piece_id=4366WJBATQGLNJXM2V63Z6WD42YZNCYA2DMXT7D77TSTYSBVX7RA, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.534232690 (70h46m2.405315735s) [noise=true , address=145.79.227.43:4157 , piece_id=7KTRATOFVKLDJQPDZL35YERZZTNWRQS334T73AQVC55RXQQ4JWTQ, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.534327270 (70h46m2.405410315s) [noise=true , address=145.79.227.21:4474 , piece_id=BZZX3YTKJKQT3KAWDEUF2V46J7B4COD7CZPZ2H4SAFNEMC3NHVCQ, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.534749855 (70h46m2.4058329s) [noise=true , address=77.90.31.18:4806 , piece_id=M5ENXW4OVQOS7RUIG62OIAA4DXJERWCLHAA6JOANRHRJGBN6ADNQ, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.534905431 (70h46m2.405988476s) [noise=true , address=145.79.227.24:4581 , piece_id=EUHJEAD536QMDG7UGB2MD2NXABD5U4OXSF2REQKTHEZWA7KEO72A, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.535348254 (70h46m2.406431299s) [noise=true , address=145.79.227.16:4910 , piece_id=HKGXMBLAD2XDZIBRL3OVJPUD4EEZ2R4RPD4V2MZ7V34KI2WUW5QQ, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.535719391 (70h46m2.406802436s) [noise=true , address=145.79.227.32:4836 , piece_id=4O6OJM23CPVD4DUJZECFF2LVM5DCLBMCVOUFU7V2LRVC2B7LHSLQ, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.535932487 (70h46m2.407015562s) [noise=true , address=77.90.31.14:4663 , piece_id=4NPSCMGDDR6J7YPQPJDZ3RXWX4V7PD5BR7IBJVZPOZSJXVXRKXMQ, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.536686312 (70h46m2.407769357s) [noise=true , address=145.79.227.34:4905 , piece_id=43NXHL7VBVKVFSDKOPBUOZJXOHL3HMYANOOBKPDACNUBOAQ4SBMA, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.537242150 (70h46m2.408325195s) [noise=true , address=145.79.227.22:4511 , piece_id=K37NXW5QF4PSP4HCVD6XMJIOSDEPTLDY3URHKBYGZFLICD44AXNA, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.538005784 (70h46m2.409088829s) [noise=true , address=145.79.227.39:4013 , piece_id=AIIRUZSHTXIJ3Q4HKSTJ6LTHCAQKLI3CIY4KJQW3GKL3IL6ZR6GQ, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:02.538636735 (70h46m2.409719791s) [noise=true , address=145.79.227.28:4726 , piece_id=CB6RMZTHGV5CK4HCAHVH3BLKI545Q2V4QRWXS5OA4S2IUB4LFBEQ, seg_pos={0 0}, upload=61969]: Uploading piece...
Dec 8 14:15:04.349584554 (70h46m4.220667599s) [upload=61969]: Next returned nil segment
Dec 8 14:15:04.349605775 (70h46m4.22068882s) [upload=61969]: Waiting for error group managing segments...
Dec 8 14:15:04.784251928 (70h46m4.655334973s) [noise=true , address=77.90.31.14:4663 , piece_id=4NPSCMGDDR6J7YPQPJDZ3RXWX4V7PD5BR7IBJVZPOZSJXVXRKXMQ, seg_pos={0 0}, upload=61969]: Done uploading piece. err: <nil>
Dec 8 14:15:04.837322415 (70h46m4.70840546s) [noise=true , address=77.90.31.18:4806 , piece_id=M5ENXW4OVQOS7RUIG62OIAA4DXJERWCLHAA6JOANRHRJGBN6ADNQ, seg_pos={0 0}, upload=61969]: Done uploading piece. err: <nil>
Dec 8 14:15:05.587735312 (70h46m5.458818347s) [noise=true , address=145.79.227.32:4836 , piece_id=4O6OJM23CPVD4DUJZECFF2LVM5DCLBMCVOUFU7V2LRVC2B7LHSLQ, seg_pos={0 0}, upload=61969]: Done uploading piece. err: <nil>
Dec 8 14:15:05.588371883 (70h46m5.459454929s) [noise=true , address=145.79.227.22:4511 , piece_id=K37NXW5QF4PSP4HCVD6XMJIOSDEPTLDY3URHKBYGZFLICD44AXNA, seg_pos={0 0}, upload=61969]: Done uploading piece. err: <nil>
Dec 8 14:15:05.607297883 (70h46m5.478380928s) [noise=true , address=145.79.227.24:4581 , piece_id=EUHJEAD536QMDG7UGB2MD2NXABD5U4OXSF2REQKTHEZWA7KEO72A, seg_pos={0 0}, upload=61969]: Done uploading piece. err: <nil>
Dec 8 14:15:05.630124479 (70h46m5.501207524s) [noise=true , address=145.79.227.33:4871 , piece_id=F7NB65E364EI6Y4WDKXERQD6ECGCKWA6NX3TBE73WDYJSXUX2BKQ, seg_pos={0 0}, upload=61969]: Done uploading piece. err: <nil>
Dec 8 14:15:05.630590277 (70h46m5.501673322s) [noise=true , address=145.79.227.43:4157 , piece_id=7KTRATOFVKLDJQPDZL35YERZZTNWRQS334T73AQVC55RXQQ4JWTQ, seg_pos={0 0}, upload=61969]: Done uploading piece. err: <nil>
Dec 8 14:15:05.630817319 (70h46m5.501900364s) [noise=true , address=145.79.227.20:4439 , piece_id=ELOM2UKEDSKKNWILB2I325A5OTSQAKIILQHP345RSTC2LXECKTVQ, seg_pos={0 0}, upload=61969]: Done uploading piece. err: <nil>
Dec 8 14:15:05.672033613 (70h46m5.543116668s) [noise=true , address=145.79.227.21:4474 , piece_id=BZZX3YTKJKQT3KAWDEUF2V46J7B4COD7CZPZ2H4SAFNEMC3NHVCQ, seg_pos={0 0}, upload=61969]: Done uploading piece. err: <nil>
Dec 8 14:15:05.735071049 (70h46m5.606154094s) [noise=true , address=145.79.227.32:4835 , piece_id=HPWNW5AC354GTIRDSCEU3NN3ULKBIUUMPRHLVNCECVKGYA5TR2DQ, seg_pos={0 0}, upload=61969]: Done uploading piece. err: <nil>
Dec 8 14:15:05.817333182 (70h46m5.688416237s) [noise=true , address=145.79.227.34:4905 , piece_id=43NXHL7VBVKVFSDKOPBUOZJXOHL3HMYANOOBKPDACNUBOAQ4SBMA, seg_pos={0 0}, upload=61969]: Done uploading piece. err: <nil>
Dec 8 14:15:05.822913809 (70h46m5.693996854s) [noise=true , address=145.79.227.41:4087 , piece_id=OZA2MCHUMIJYWXHS3KAVJRVTJ6LWAW5XR257OGAZRNPIOGMETZRA, seg_pos={0 0}, upload=61969]: Done uploading piece. err: <nil>
Dec 8 14:15:05.823926878 (70h46m5.695009923s) [noise=true , address=145.79.227.23:4547 , piece_id=4366WJBATQGLNJXM2V63Z6WD42YZNCYA2DMXT7D77TSTYSBVX7RA, seg_pos={0 0}, upload=61969]: Done uploading piece. err: <nil>
Dec 8 14:15:05.823938099 (70h46m5.695021154s) [seg_pos={0 0}, upload=61969]: Segment reached optimal threshold of 13 pieces.
Dec 8 14:15:05.824198556 (70h46m5.695281601s) [noise=true , address=145.79.227.39:4013 , piece_id=AIIRUZSHTXIJ3Q4HKSTJ6LTHCAQKLI3CIY4KJQW3GKL3IL6ZR6GQ, seg_pos={0 0}, upload=61969]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: expected piece hash; context canceled; EOF
Dec 8 14:15:05.824210698 (70h46m5.695293743s) [noise=true , address=145.79.227.39:4013 , piece_id=AIIRUZSHTXIJ3Q4HKSTJ6LTHCAQKLI3CIY4KJQW3GKL3IL6ZR6GQ, seg_pos={0 0}, upload=61969]: Long tail cancellation due to optimal threshold
Dec 8 14:15:05.824416240 (70h46m5.695499285s) [noise=true , address=145.79.227.28:4726 , piece_id=CB6RMZTHGV5CK4HCAHVH3BLKI545Q2V4QRWXS5OA4S2IUB4LFBEQ, seg_pos={0 0}, upload=61969]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: expected piece hash; context canceled; EOF
Dec 8 14:15:05.824448682 (70h46m5.695531717s) [noise=true , address=145.79.227.28:4726 , piece_id=CB6RMZTHGV5CK4HCAHVH3BLKI545Q2V4QRWXS5OA4S2IUB4LFBEQ, seg_pos={0 0}, upload=61969]: Long tail cancellation due to optimal threshold
Dec 8 14:15:05.824649695 (70h46m5.69573274s) [noise=true , address=145.79.227.16:4910 , piece_id=HKGXMBLAD2XDZIBRL3OVJPUD4EEZ2R4RPD4V2MZ7V34KI2WUW5QQ, seg_pos={0 0}, upload=61969]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: expected piece hash; context canceled; EOF
Dec 8 14:15:05.824662118 (70h46m5.695745163s) [noise=true , address=145.79.227.16:4910 , piece_id=HKGXMBLAD2XDZIBRL3OVJPUD4EEZ2R4RPD4V2MZ7V34KI2WUW5QQ, seg_pos={0 0}, upload=61969]: Long tail cancellation due to optimal threshold
Dec 8 14:15:05.824696944 (70h46m5.695779989s) [seg_pos={0 0}, upload=61969]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 14:15:05.824818956 (70h46m5.695902001s) [upload=61969]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 14:15:05.824827082 (70h46m5.695910127s) [upload=61969]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 14:17:06.241593111 (70h48m6.112676156s) [upload=61969]: Done uploading segments...
Dec 8 14:17:06.241713471 (70h48m6.112796516s) [upload=61969]: Upload finished. err: metaclient: context deadline exceeded
Duration: 9.904337136s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
8.575600342s trace:
https://jaeger.dev.storj.io/trace/406d13f34dbd7740
FAIL (2025-12-08 14:22:24 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 8.575629116s
core
uplink
us1
|