Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 864097 reports (Page 11712 of 17282)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 23.501365885s trace: https://jaeger.dev.storj.io/trace/29daaac43eca6484 FAIL (2025-12-08 15:14:33 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 23.501406122s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 1.019459563s remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4a31039498bdbd4f FAIL (2025-12-08 15:14:33 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 1.019478809s core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 56 out of 60 is failed (threshold=6) [XXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XXXXX.XXX] duration: 5.001495097s trace: https://jaeger.dev.storj.io/trace/7708c6d0483d2c95 FAIL (2025-12-08 15:14:17 UTC) orbiter warning: context deadline exceeded Duration: 5.001520154s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X.......X..X.....................................X..........] duration: 23.254691882s trace: https://jaeger.dev.storj.io/trace/dbd17afbd215569 FAIL (2025-12-08 15:14:01 UTC) orbiter warning: context deadline exceeded Duration: 23.254720518s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 18.578673361s trace: https://jaeger.dev.storj.io/trace/29daaac43eca6484 FAIL (2025-12-08 15:13:48 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 18.578709239s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..X.....X..........X...X....................................] duration: 6.123156074s trace: https://jaeger.dev.storj.io/trace/9d8451999396f06 FAIL (2025-12-08 15:13:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.123182003s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 11.914819978s trace: https://jaeger.dev.storj.io/trace/5ce839c1456bbf8a FAIL (2025-12-08 15:13:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 25:20.182936263 (70h56m20.054019298s) [upload=62126]: Starting upload Dec 8 14:25:20.183037416 (70h56m20.054120461s) [upload=62126]: Done starting upload Dec 8 14:25:20.183136214 (70h56m20.054219259s) [upload=62126]: Uploading segments... Dec 8 14:25:20.183291900 (70h56m20.054374945s) [upload=62126]: Got next segment. Inline: false Dec 8 14:25:20.183343058 (70h56m20.054426103s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 14:25:20.183349700 (70h56m20.054432745s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 14:25:20.560968522 (70h56m20.432051567s) [seg_pos={0 0}, upload=62126]: Begin upload segment... Dec 8 14:25:20.561021322 (70h56m20.432104357s) [seg_pos={0 0}, upload=62126]: Done begin upload segment. Dec 8 14:25:20.561050337 (70h56m20.432133383s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.562561536 (70h56m20.433644581s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563077198 (70h56m20.434160243s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563560729 (70h56m20.434643774s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564043677 (70h56m20.435126732s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564505988 (70h56m20.435589043s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564975802 (70h56m20.436058848s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.566789025 (70h56m20.43787207s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.568723158 (70h56m20.439806204s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.570985847 (70h56m20.442068892s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573462843 (70h56m20.444545889s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573987674 (70h56m20.445070719s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.575740802 (70h56m20.446823837s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.577275113 (70h56m20.448358158s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.578540883 (70h56m20.449623928s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.580403731 (70h56m20.451486776s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:22.334236681 (70h56m22.205319726s) [upload=62126]: Next returned nil segment Dec 8 14:25:22.334251700 (70h56m22.205334735s) [upload=62126]: Waiting for error group managing segments... Dec 8 14:25:22.746429339 (70h56m22.617512384s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.747165611 (70h56m22.618248646s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.753178531 (70h56m22.624261566s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.922057260 (70h56m22.793140305s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.951038517 (70h56m22.822121562s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.651296259 (70h56m23.522379304s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.653766143 (70h56m23.524849178s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.680671276 (70h56m23.551754321s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.700290564 (70h56m23.571373609s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.707126822 (70h56m23.578209868s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.721842694 (70h56m23.592925739s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.730554192 (70h56m23.601637237s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670621120 (70h56m24.541704185s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670647519 (70h56m24.541730564s) [seg_pos={0 0}, upload=62126]: Segment reached optimal threshold of 13 pieces. Dec 8 14:25:24.670967409 (70h56m24.542050454s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection Dec 8 14:25:24.670977318 (70h56m24.542060363s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671144947 (70h56m24.542227982s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: read tcp 188.166.225.170:39576->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 14:25:24.671168231 (70h56m24.542251306s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671490626 (70h56m24.542573661s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 8 14:25:24.671499091 (70h56m24.542582126s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671551481 (70h56m24.542634526s) [seg_pos={0 0}, upload=62126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 14:25:24.671640871 (70h56m24.542723906s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 14:25:24.671647103 (70h56m24.542730138s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 14:25:25.114013719 (70h56m24.985096764s) [upload=62126]: Done uploading segments... Dec 8 14:25:25.114042193 (70h56m24.985125258s) [upload=62126]: Upload finished. err: <nil> Duration: 11.914849785s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 18.012639389s trace: https://jaeger.dev.storj.io/trace/29daaac43eca6484 FAIL (2025-12-08 15:13:28 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 18.01267183s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 977.127955ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4a31039498bdbd4f FAIL (2025-12-08 15:13:28 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 977.151479ms core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 56 out of 60 is failed (threshold=6) [XXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XXXXX.XX] duration: 5.00128361s trace: https://jaeger.dev.storj.io/trace/4650e90f04f564d5 FAIL (2025-12-08 15:13:25 UTC) orbiter warning: context deadline exceeded Duration: 5.001302305s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X.......X..X.....................................X.........] duration: 26.451888558s trace: https://jaeger.dev.storj.io/trace/dbd17afbd215569 FAIL (2025-12-08 15:13:03 UTC) orbiter warning: context deadline exceeded Duration: 26.451913616s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...X.....X..........X...X...................................] duration: 9.723648583s trace: https://jaeger.dev.storj.io/trace/9d8451999396f06 FAIL (2025-12-08 15:13:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.72367892s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 15.703241239s trace: https://jaeger.dev.storj.io/trace/29daaac43eca6484 FAIL (2025-12-08 15:12:47 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 15.703274132s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 1.014505158s remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4a31039498bdbd4f FAIL (2025-12-08 15:12:44 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 1.014533352s core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 8.492365016s trace: https://jaeger.dev.storj.io/trace/5ce839c1456bbf8a FAIL (2025-12-08 15:12:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 25:20.182936263 (70h56m20.054019298s) [upload=62126]: Starting upload Dec 8 14:25:20.183037416 (70h56m20.054120461s) [upload=62126]: Done starting upload Dec 8 14:25:20.183136214 (70h56m20.054219259s) [upload=62126]: Uploading segments... Dec 8 14:25:20.183291900 (70h56m20.054374945s) [upload=62126]: Got next segment. Inline: false Dec 8 14:25:20.183343058 (70h56m20.054426103s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 14:25:20.183349700 (70h56m20.054432745s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 14:25:20.560968522 (70h56m20.432051567s) [seg_pos={0 0}, upload=62126]: Begin upload segment... Dec 8 14:25:20.561021322 (70h56m20.432104357s) [seg_pos={0 0}, upload=62126]: Done begin upload segment. Dec 8 14:25:20.561050337 (70h56m20.432133383s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.562561536 (70h56m20.433644581s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563077198 (70h56m20.434160243s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563560729 (70h56m20.434643774s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564043677 (70h56m20.435126732s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564505988 (70h56m20.435589043s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564975802 (70h56m20.436058848s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.566789025 (70h56m20.43787207s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.568723158 (70h56m20.439806204s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.570985847 (70h56m20.442068892s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573462843 (70h56m20.444545889s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573987674 (70h56m20.445070719s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.575740802 (70h56m20.446823837s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.577275113 (70h56m20.448358158s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.578540883 (70h56m20.449623928s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.580403731 (70h56m20.451486776s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:22.334236681 (70h56m22.205319726s) [upload=62126]: Next returned nil segment Dec 8 14:25:22.334251700 (70h56m22.205334735s) [upload=62126]: Waiting for error group managing segments... Dec 8 14:25:22.746429339 (70h56m22.617512384s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.747165611 (70h56m22.618248646s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.753178531 (70h56m22.624261566s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.922057260 (70h56m22.793140305s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.951038517 (70h56m22.822121562s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.651296259 (70h56m23.522379304s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.653766143 (70h56m23.524849178s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.680671276 (70h56m23.551754321s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.700290564 (70h56m23.571373609s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.707126822 (70h56m23.578209868s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.721842694 (70h56m23.592925739s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.730554192 (70h56m23.601637237s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670621120 (70h56m24.541704185s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670647519 (70h56m24.541730564s) [seg_pos={0 0}, upload=62126]: Segment reached optimal threshold of 13 pieces. Dec 8 14:25:24.670967409 (70h56m24.542050454s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection Dec 8 14:25:24.670977318 (70h56m24.542060363s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671144947 (70h56m24.542227982s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: read tcp 188.166.225.170:39576->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 14:25:24.671168231 (70h56m24.542251306s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671490626 (70h56m24.542573661s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 8 14:25:24.671499091 (70h56m24.542582126s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671551481 (70h56m24.542634526s) [seg_pos={0 0}, upload=62126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 14:25:24.671640871 (70h56m24.542723906s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 14:25:24.671647103 (70h56m24.542730138s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 14:25:25.114013719 (70h56m24.985096764s) [upload=62126]: Done uploading segments... Dec 8 14:25:25.114042193 (70h56m24.985125258s) [upload=62126]: Upload finished. err: <nil> Duration: 8.492396024s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X.......X..X.....................................X........] duration: 24.898479066s trace: https://jaeger.dev.storj.io/trace/dbd17afbd215569 FAIL (2025-12-08 15:12:29 UTC) orbiter warning: context deadline exceeded Duration: 24.898594034s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 56 out of 60 is failed (threshold=6) [XXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XXXXX.X] duration: 5.00139338s trace: https://jaeger.dev.storj.io/trace/4ce47e9fbd3f6852 FAIL (2025-12-08 15:12:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001418549s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [X...X.....X..........X...X..................................] duration: 18.44362054s trace: https://jaeger.dev.storj.io/trace/9d8451999396f06 FAIL (2025-12-08 15:12:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.443651308s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 13.210740399s trace: https://jaeger.dev.storj.io/trace/29daaac43eca6484 FAIL (2025-12-08 15:12:02 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 13.210774444s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 27.691737069s trace: https://jaeger.dev.storj.io/trace/5ce839c1456bbf8a FAIL (2025-12-08 15:11:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 25:20.182936263 (70h56m20.054019298s) [upload=62126]: Starting upload Dec 8 14:25:20.183037416 (70h56m20.054120461s) [upload=62126]: Done starting upload Dec 8 14:25:20.183136214 (70h56m20.054219259s) [upload=62126]: Uploading segments... Dec 8 14:25:20.183291900 (70h56m20.054374945s) [upload=62126]: Got next segment. Inline: false Dec 8 14:25:20.183343058 (70h56m20.054426103s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 14:25:20.183349700 (70h56m20.054432745s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 14:25:20.560968522 (70h56m20.432051567s) [seg_pos={0 0}, upload=62126]: Begin upload segment... Dec 8 14:25:20.561021322 (70h56m20.432104357s) [seg_pos={0 0}, upload=62126]: Done begin upload segment. Dec 8 14:25:20.561050337 (70h56m20.432133383s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.562561536 (70h56m20.433644581s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563077198 (70h56m20.434160243s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563560729 (70h56m20.434643774s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564043677 (70h56m20.435126732s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564505988 (70h56m20.435589043s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564975802 (70h56m20.436058848s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.566789025 (70h56m20.43787207s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.568723158 (70h56m20.439806204s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.570985847 (70h56m20.442068892s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573462843 (70h56m20.444545889s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573987674 (70h56m20.445070719s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.575740802 (70h56m20.446823837s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.577275113 (70h56m20.448358158s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.578540883 (70h56m20.449623928s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.580403731 (70h56m20.451486776s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:22.334236681 (70h56m22.205319726s) [upload=62126]: Next returned nil segment Dec 8 14:25:22.334251700 (70h56m22.205334735s) [upload=62126]: Waiting for error group managing segments... Dec 8 14:25:22.746429339 (70h56m22.617512384s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.747165611 (70h56m22.618248646s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.753178531 (70h56m22.624261566s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.922057260 (70h56m22.793140305s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.951038517 (70h56m22.822121562s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.651296259 (70h56m23.522379304s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.653766143 (70h56m23.524849178s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.680671276 (70h56m23.551754321s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.700290564 (70h56m23.571373609s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.707126822 (70h56m23.578209868s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.721842694 (70h56m23.592925739s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.730554192 (70h56m23.601637237s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670621120 (70h56m24.541704185s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670647519 (70h56m24.541730564s) [seg_pos={0 0}, upload=62126]: Segment reached optimal threshold of 13 pieces. Dec 8 14:25:24.670967409 (70h56m24.542050454s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection Dec 8 14:25:24.670977318 (70h56m24.542060363s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671144947 (70h56m24.542227982s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: read tcp 188.166.225.170:39576->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 14:25:24.671168231 (70h56m24.542251306s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671490626 (70h56m24.542573661s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 8 14:25:24.671499091 (70h56m24.542582126s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671551481 (70h56m24.542634526s) [seg_pos={0 0}, upload=62126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 14:25:24.671640871 (70h56m24.542723906s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 14:25:24.671647103 (70h56m24.542730138s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 14:25:25.114013719 (70h56m24.985096764s) [upload=62126]: Done uploading segments... Dec 8 14:25:25.114042193 (70h56m24.985125258s) [upload=62126]: Upload finished. err: <nil> Duration: 27.691766364s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 56 out of 60 is failed (threshold=6) [XXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XXXXX.] duration: 5.000541118s trace: https://jaeger.dev.storj.io/trace/60af04318bd635c9 FAIL (2025-12-08 15:11:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000602936s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 1.02345525s remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4a31039498bdbd4f FAIL (2025-12-08 15:11:46 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 1.023484796s core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 18.263699937s trace: https://jaeger.dev.storj.io/trace/29daaac43eca6484 FAIL (2025-12-08 15:11:38 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 18.263730637s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 57 out of 60 is failed (threshold=6) [XXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XXXXX] duration: 5.000656978s trace: https://jaeger.dev.storj.io/trace/60af04318bd635c9 FAIL (2025-12-08 15:11:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000687807s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 692.813404ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4a31039498bdbd4f FAIL (2025-12-08 15:11:16 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 692.835947ms core satellite us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X.......X..X.....................................X.......] duration: 7.521714732s trace: https://jaeger.dev.storj.io/trace/dbd17afbd215569 FAIL (2025-12-08 15:11:01 UTC) orbiter warning: context deadline exceeded Duration: 7.521735954s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.X...X.....X..........X...X.................................] duration: 10.064226669s trace: https://jaeger.dev.storj.io/trace/9d8451999396f06 FAIL (2025-12-08 15:10:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.064245484s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X.......X..X.....................................X......] duration: 6.679895542s trace: https://jaeger.dev.storj.io/trace/dbd17afbd215569 FAIL (2025-12-08 15:10:43 UTC) orbiter warning: context deadline exceeded Duration: 6.679927723s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 9.114961278s trace: https://jaeger.dev.storj.io/trace/5ce839c1456bbf8a FAIL (2025-12-08 15:10:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 25:20.182936263 (70h56m20.054019298s) [upload=62126]: Starting upload Dec 8 14:25:20.183037416 (70h56m20.054120461s) [upload=62126]: Done starting upload Dec 8 14:25:20.183136214 (70h56m20.054219259s) [upload=62126]: Uploading segments... Dec 8 14:25:20.183291900 (70h56m20.054374945s) [upload=62126]: Got next segment. Inline: false Dec 8 14:25:20.183343058 (70h56m20.054426103s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 14:25:20.183349700 (70h56m20.054432745s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 14:25:20.560968522 (70h56m20.432051567s) [seg_pos={0 0}, upload=62126]: Begin upload segment... Dec 8 14:25:20.561021322 (70h56m20.432104357s) [seg_pos={0 0}, upload=62126]: Done begin upload segment. Dec 8 14:25:20.561050337 (70h56m20.432133383s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.562561536 (70h56m20.433644581s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563077198 (70h56m20.434160243s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563560729 (70h56m20.434643774s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564043677 (70h56m20.435126732s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564505988 (70h56m20.435589043s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564975802 (70h56m20.436058848s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.566789025 (70h56m20.43787207s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.568723158 (70h56m20.439806204s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.570985847 (70h56m20.442068892s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573462843 (70h56m20.444545889s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573987674 (70h56m20.445070719s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.575740802 (70h56m20.446823837s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.577275113 (70h56m20.448358158s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.578540883 (70h56m20.449623928s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.580403731 (70h56m20.451486776s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:22.334236681 (70h56m22.205319726s) [upload=62126]: Next returned nil segment Dec 8 14:25:22.334251700 (70h56m22.205334735s) [upload=62126]: Waiting for error group managing segments... Dec 8 14:25:22.746429339 (70h56m22.617512384s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.747165611 (70h56m22.618248646s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.753178531 (70h56m22.624261566s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.922057260 (70h56m22.793140305s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.951038517 (70h56m22.822121562s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.651296259 (70h56m23.522379304s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.653766143 (70h56m23.524849178s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.680671276 (70h56m23.551754321s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.700290564 (70h56m23.571373609s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.707126822 (70h56m23.578209868s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.721842694 (70h56m23.592925739s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.730554192 (70h56m23.601637237s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670621120 (70h56m24.541704185s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670647519 (70h56m24.541730564s) [seg_pos={0 0}, upload=62126]: Segment reached optimal threshold of 13 pieces. Dec 8 14:25:24.670967409 (70h56m24.542050454s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection Dec 8 14:25:24.670977318 (70h56m24.542060363s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671144947 (70h56m24.542227982s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: read tcp 188.166.225.170:39576->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 14:25:24.671168231 (70h56m24.542251306s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671490626 (70h56m24.542573661s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 8 14:25:24.671499091 (70h56m24.542582126s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671551481 (70h56m24.542634526s) [seg_pos={0 0}, upload=62126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 14:25:24.671640871 (70h56m24.542723906s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 14:25:24.671647103 (70h56m24.542730138s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 14:25:25.114013719 (70h56m24.985096764s) [upload=62126]: Done uploading segments... Dec 8 14:25:25.114042193 (70h56m24.985125258s) [upload=62126]: Upload finished. err: <nil> Duration: 9.114979975s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [X.X...X.....X..........X...X................................] duration: 31.643957457s trace: https://jaeger.dev.storj.io/trace/9d8451999396f06 FAIL (2025-12-08 15:10:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 31.643988355s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 57 out of 60 is failed (threshold=6) [XXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XXXX] duration: 5.003834301s trace: https://jaeger.dev.storj.io/trace/1377c23285a50baf FAIL (2025-12-08 15:10:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.003865539s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 20.541848904s trace: https://jaeger.dev.storj.io/trace/29daaac43eca6484 FAIL (2025-12-08 15:10:34 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 20.541890835s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 682.361071ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4a31039498bdbd4f FAIL (2025-12-08 15:10:17 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 682.389514ms core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 57 out of 60 is failed (threshold=6) [XXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XXX] duration: 5.001778565s trace: https://jaeger.dev.storj.io/trace/3f75ba36f3f32149 FAIL (2025-12-08 15:10:12 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001805147s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X....X.......X..X.....................................X.....] duration: 11.762034744s trace: https://jaeger.dev.storj.io/trace/dbd17afbd215569 FAIL (2025-12-08 15:10:09 UTC) orbiter warning: context deadline exceeded Duration: 11.762064741s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 57 out of 60 is failed (threshold=6) [XXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XX] duration: 5.001591569s trace: https://jaeger.dev.storj.io/trace/5b768bd776106551 FAIL (2025-12-08 15:09:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001612078s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 26.052113359s trace: https://jaeger.dev.storj.io/trace/5ce839c1456bbf8a FAIL (2025-12-08 15:09:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 25:20.182936263 (70h56m20.054019298s) [upload=62126]: Starting upload Dec 8 14:25:20.183037416 (70h56m20.054120461s) [upload=62126]: Done starting upload Dec 8 14:25:20.183136214 (70h56m20.054219259s) [upload=62126]: Uploading segments... Dec 8 14:25:20.183291900 (70h56m20.054374945s) [upload=62126]: Got next segment. Inline: false Dec 8 14:25:20.183343058 (70h56m20.054426103s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 14:25:20.183349700 (70h56m20.054432745s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 14:25:20.560968522 (70h56m20.432051567s) [seg_pos={0 0}, upload=62126]: Begin upload segment... Dec 8 14:25:20.561021322 (70h56m20.432104357s) [seg_pos={0 0}, upload=62126]: Done begin upload segment. Dec 8 14:25:20.561050337 (70h56m20.432133383s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.562561536 (70h56m20.433644581s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563077198 (70h56m20.434160243s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563560729 (70h56m20.434643774s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564043677 (70h56m20.435126732s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564505988 (70h56m20.435589043s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564975802 (70h56m20.436058848s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.566789025 (70h56m20.43787207s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.568723158 (70h56m20.439806204s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.570985847 (70h56m20.442068892s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573462843 (70h56m20.444545889s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573987674 (70h56m20.445070719s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.575740802 (70h56m20.446823837s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.577275113 (70h56m20.448358158s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.578540883 (70h56m20.449623928s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.580403731 (70h56m20.451486776s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:22.334236681 (70h56m22.205319726s) [upload=62126]: Next returned nil segment Dec 8 14:25:22.334251700 (70h56m22.205334735s) [upload=62126]: Waiting for error group managing segments... Dec 8 14:25:22.746429339 (70h56m22.617512384s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.747165611 (70h56m22.618248646s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.753178531 (70h56m22.624261566s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.922057260 (70h56m22.793140305s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.951038517 (70h56m22.822121562s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.651296259 (70h56m23.522379304s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.653766143 (70h56m23.524849178s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.680671276 (70h56m23.551754321s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.700290564 (70h56m23.571373609s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.707126822 (70h56m23.578209868s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.721842694 (70h56m23.592925739s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.730554192 (70h56m23.601637237s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670621120 (70h56m24.541704185s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670647519 (70h56m24.541730564s) [seg_pos={0 0}, upload=62126]: Segment reached optimal threshold of 13 pieces. Dec 8 14:25:24.670967409 (70h56m24.542050454s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection Dec 8 14:25:24.670977318 (70h56m24.542060363s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671144947 (70h56m24.542227982s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: read tcp 188.166.225.170:39576->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 14:25:24.671168231 (70h56m24.542251306s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671490626 (70h56m24.542573661s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 8 14:25:24.671499091 (70h56m24.542582126s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671551481 (70h56m24.542634526s) [seg_pos={0 0}, upload=62126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 14:25:24.671640871 (70h56m24.542723906s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 14:25:24.671647103 (70h56m24.542730138s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 14:25:25.114013719 (70h56m24.985096764s) [upload=62126]: Done uploading segments... Dec 8 14:25:25.114042193 (70h56m24.985125258s) [upload=62126]: Upload finished. err: <nil> Duration: 26.052143717s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 677.915156ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4a31039498bdbd4f FAIL (2025-12-08 15:09:36 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 677.943821ms core satellite us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [XX....X.......X..X.....................................X....] duration: 6.61789264s trace: https://jaeger.dev.storj.io/trace/dbd17afbd215569 FAIL (2025-12-08 15:09:35 UTC) orbiter warning: context deadline exceeded Duration: 6.617931114s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 14.468579877s trace: https://jaeger.dev.storj.io/trace/29daaac43eca6484 FAIL (2025-12-08 15:09:32 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 14.468609523s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.XX....X.......X..X.....................................X...] duration: 7.578790302s trace: https://jaeger.dev.storj.io/trace/dbd17afbd215569 FAIL (2025-12-08 15:09:14 UTC) orbiter warning: context deadline exceeded Duration: 7.578819036s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [XX.X...X.....X..........X...X...............................] duration: 7.282029017s trace: https://jaeger.dev.storj.io/trace/9d8451999396f06 FAIL (2025-12-08 15:09:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.282062029s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 696.54613ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4a31039498bdbd4f FAIL (2025-12-08 15:08:57 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 696.571858ms core satellite us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..XX....X.......X..X.....................................X..] duration: 7.278442915s trace: https://jaeger.dev.storj.io/trace/dbd17afbd215569 FAIL (2025-12-08 15:08:53 UTC) orbiter warning: context deadline exceeded Duration: 7.278472511s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 57 out of 60 is failed (threshold=6) [XXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.X] duration: 5.001046654s trace: https://jaeger.dev.storj.io/trace/197a04b6c541269a FAIL (2025-12-08 15:08:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001103032s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 10.968410531s trace: https://jaeger.dev.storj.io/trace/29daaac43eca6484 FAIL (2025-12-08 15:08:46 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 10.968437002s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 675.016897ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4a31039498bdbd4f FAIL (2025-12-08 15:08:25 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 675.054187ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 9.236085935s trace: https://jaeger.dev.storj.io/trace/5ce839c1456bbf8a FAIL (2025-12-08 15:08:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 25:20.182936263 (70h56m20.054019298s) [upload=62126]: Starting upload Dec 8 14:25:20.183037416 (70h56m20.054120461s) [upload=62126]: Done starting upload Dec 8 14:25:20.183136214 (70h56m20.054219259s) [upload=62126]: Uploading segments... Dec 8 14:25:20.183291900 (70h56m20.054374945s) [upload=62126]: Got next segment. Inline: false Dec 8 14:25:20.183343058 (70h56m20.054426103s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 14:25:20.183349700 (70h56m20.054432745s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 14:25:20.560968522 (70h56m20.432051567s) [seg_pos={0 0}, upload=62126]: Begin upload segment... Dec 8 14:25:20.561021322 (70h56m20.432104357s) [seg_pos={0 0}, upload=62126]: Done begin upload segment. Dec 8 14:25:20.561050337 (70h56m20.432133383s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.562561536 (70h56m20.433644581s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563077198 (70h56m20.434160243s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563560729 (70h56m20.434643774s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564043677 (70h56m20.435126732s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564505988 (70h56m20.435589043s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564975802 (70h56m20.436058848s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.566789025 (70h56m20.43787207s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.568723158 (70h56m20.439806204s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.570985847 (70h56m20.442068892s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573462843 (70h56m20.444545889s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573987674 (70h56m20.445070719s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.575740802 (70h56m20.446823837s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.577275113 (70h56m20.448358158s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.578540883 (70h56m20.449623928s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.580403731 (70h56m20.451486776s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:22.334236681 (70h56m22.205319726s) [upload=62126]: Next returned nil segment Dec 8 14:25:22.334251700 (70h56m22.205334735s) [upload=62126]: Waiting for error group managing segments... Dec 8 14:25:22.746429339 (70h56m22.617512384s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.747165611 (70h56m22.618248646s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.753178531 (70h56m22.624261566s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.922057260 (70h56m22.793140305s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.951038517 (70h56m22.822121562s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.651296259 (70h56m23.522379304s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.653766143 (70h56m23.524849178s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.680671276 (70h56m23.551754321s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.700290564 (70h56m23.571373609s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.707126822 (70h56m23.578209868s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.721842694 (70h56m23.592925739s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.730554192 (70h56m23.601637237s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670621120 (70h56m24.541704185s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670647519 (70h56m24.541730564s) [seg_pos={0 0}, upload=62126]: Segment reached optimal threshold of 13 pieces. Dec 8 14:25:24.670967409 (70h56m24.542050454s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection Dec 8 14:25:24.670977318 (70h56m24.542060363s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671144947 (70h56m24.542227982s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: read tcp 188.166.225.170:39576->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 14:25:24.671168231 (70h56m24.542251306s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671490626 (70h56m24.542573661s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 8 14:25:24.671499091 (70h56m24.542582126s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671551481 (70h56m24.542634526s) [seg_pos={0 0}, upload=62126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 14:25:24.671640871 (70h56m24.542723906s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 14:25:24.671647103 (70h56m24.542730138s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 14:25:25.114013719 (70h56m24.985096764s) [upload=62126]: Done uploading segments... Dec 8 14:25:25.114042193 (70h56m24.985125258s) [upload=62126]: Upload finished. err: <nil> Duration: 9.236112404s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X..XX....X.......X..X.....................................X.] duration: 5.846633186s trace: https://jaeger.dev.storj.io/trace/dbd17afbd215569 FAIL (2025-12-08 15:08:08 UTC) orbiter warning: context deadline exceeded Duration: 5.846665357s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [XXX.X...X.....X..........X...X..............................] duration: 11.654596346s trace: https://jaeger.dev.storj.io/trace/9d8451999396f06 FAIL (2025-12-08 15:07:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.654620382s core uplink us-select-1 us1-placement-20

1
...
11710
11711
11712
11713
11714
...
17282