Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 864097 reports (Page 11713 of 17282)

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 57 out of 60 is failed (threshold=6) [XXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.] duration: 4.964982509s trace: https://jaeger.dev.storj.io/trace/6853c61ddb8be53a FAIL (2025-12-08 15:07:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.96500844s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 12.932677118s trace: https://jaeger.dev.storj.io/trace/29daaac43eca6484 FAIL (2025-12-08 15:07:46 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 12.932707838s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 57 out of 60 is failed (threshold=6) [.XXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X] duration: 5.000497178s trace: https://jaeger.dev.storj.io/trace/6853c61ddb8be53a FAIL (2025-12-08 15:07:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000596315s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 1.011790389s remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4a31039498bdbd4f FAIL (2025-12-08 15:07:28 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 1.011833191s core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 10.227373123s trace: https://jaeger.dev.storj.io/trace/5ce839c1456bbf8a FAIL (2025-12-08 15:07:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 25:20.182936263 (70h56m20.054019298s) [upload=62126]: Starting upload Dec 8 14:25:20.183037416 (70h56m20.054120461s) [upload=62126]: Done starting upload Dec 8 14:25:20.183136214 (70h56m20.054219259s) [upload=62126]: Uploading segments... Dec 8 14:25:20.183291900 (70h56m20.054374945s) [upload=62126]: Got next segment. Inline: false Dec 8 14:25:20.183343058 (70h56m20.054426103s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 14:25:20.183349700 (70h56m20.054432745s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 14:25:20.560968522 (70h56m20.432051567s) [seg_pos={0 0}, upload=62126]: Begin upload segment... Dec 8 14:25:20.561021322 (70h56m20.432104357s) [seg_pos={0 0}, upload=62126]: Done begin upload segment. Dec 8 14:25:20.561050337 (70h56m20.432133383s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.562561536 (70h56m20.433644581s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563077198 (70h56m20.434160243s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563560729 (70h56m20.434643774s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564043677 (70h56m20.435126732s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564505988 (70h56m20.435589043s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564975802 (70h56m20.436058848s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.566789025 (70h56m20.43787207s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.568723158 (70h56m20.439806204s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.570985847 (70h56m20.442068892s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573462843 (70h56m20.444545889s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573987674 (70h56m20.445070719s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.575740802 (70h56m20.446823837s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.577275113 (70h56m20.448358158s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.578540883 (70h56m20.449623928s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.580403731 (70h56m20.451486776s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:22.334236681 (70h56m22.205319726s) [upload=62126]: Next returned nil segment Dec 8 14:25:22.334251700 (70h56m22.205334735s) [upload=62126]: Waiting for error group managing segments... Dec 8 14:25:22.746429339 (70h56m22.617512384s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.747165611 (70h56m22.618248646s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.753178531 (70h56m22.624261566s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.922057260 (70h56m22.793140305s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.951038517 (70h56m22.822121562s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.651296259 (70h56m23.522379304s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.653766143 (70h56m23.524849178s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.680671276 (70h56m23.551754321s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.700290564 (70h56m23.571373609s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.707126822 (70h56m23.578209868s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.721842694 (70h56m23.592925739s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.730554192 (70h56m23.601637237s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670621120 (70h56m24.541704185s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670647519 (70h56m24.541730564s) [seg_pos={0 0}, upload=62126]: Segment reached optimal threshold of 13 pieces. Dec 8 14:25:24.670967409 (70h56m24.542050454s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection Dec 8 14:25:24.670977318 (70h56m24.542060363s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671144947 (70h56m24.542227982s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: read tcp 188.166.225.170:39576->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 14:25:24.671168231 (70h56m24.542251306s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671490626 (70h56m24.542573661s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 8 14:25:24.671499091 (70h56m24.542582126s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671551481 (70h56m24.542634526s) [seg_pos={0 0}, upload=62126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 14:25:24.671640871 (70h56m24.542723906s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 14:25:24.671647103 (70h56m24.542730138s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 14:25:25.114013719 (70h56m24.985096764s) [upload=62126]: Done uploading segments... Dec 8 14:25:25.114042193 (70h56m24.985125258s) [upload=62126]: Upload finished. err: <nil> Duration: 10.227400968s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X..XX....X.......X..X.....................................X] duration: 30.002241854s trace: https://jaeger.dev.storj.io/trace/dbd17afbd215569 FAIL (2025-12-08 15:07:19 UTC) orbiter warning: context deadline exceeded Duration: 30.00226618s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 57 out of 60 is failed (threshold=6) [X.XXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.] duration: 3.646103609s trace: https://jaeger.dev.storj.io/trace/1e278e097b50fead FAIL (2025-12-08 15:06:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.646156849s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.XXX.X...X.....X..........X...X.............................] duration: 11.536809058s trace: https://jaeger.dev.storj.io/trace/9d8451999396f06 FAIL (2025-12-08 15:06:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.536839616s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 11.949463304s trace: https://jaeger.dev.storj.io/trace/29daaac43eca6484 FAIL (2025-12-08 15:06:44 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 11.949490205s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [X.XXX.X...X.....X..........X...X............................] duration: 11.237181055s trace: https://jaeger.dev.storj.io/trace/9d8451999396f06 FAIL (2025-12-08 15:06:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.237205871s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 1.039234456s remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4a31039498bdbd4f FAIL (2025-12-08 15:06:21 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 1.039267752s core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XX.XXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001599244s trace: https://jaeger.dev.storj.io/trace/1e278e097b50fead FAIL (2025-12-08 15:06:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001631193s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 9.398583183s trace: https://jaeger.dev.storj.io/trace/5ce839c1456bbf8a FAIL (2025-12-08 15:06:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 25:20.182936263 (70h56m20.054019298s) [upload=62126]: Starting upload Dec 8 14:25:20.183037416 (70h56m20.054120461s) [upload=62126]: Done starting upload Dec 8 14:25:20.183136214 (70h56m20.054219259s) [upload=62126]: Uploading segments... Dec 8 14:25:20.183291900 (70h56m20.054374945s) [upload=62126]: Got next segment. Inline: false Dec 8 14:25:20.183343058 (70h56m20.054426103s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 14:25:20.183349700 (70h56m20.054432745s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 14:25:20.560968522 (70h56m20.432051567s) [seg_pos={0 0}, upload=62126]: Begin upload segment... Dec 8 14:25:20.561021322 (70h56m20.432104357s) [seg_pos={0 0}, upload=62126]: Done begin upload segment. Dec 8 14:25:20.561050337 (70h56m20.432133383s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.562561536 (70h56m20.433644581s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563077198 (70h56m20.434160243s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563560729 (70h56m20.434643774s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564043677 (70h56m20.435126732s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564505988 (70h56m20.435589043s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564975802 (70h56m20.436058848s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.566789025 (70h56m20.43787207s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.568723158 (70h56m20.439806204s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.570985847 (70h56m20.442068892s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573462843 (70h56m20.444545889s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573987674 (70h56m20.445070719s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.575740802 (70h56m20.446823837s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.577275113 (70h56m20.448358158s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.578540883 (70h56m20.449623928s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.580403731 (70h56m20.451486776s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:22.334236681 (70h56m22.205319726s) [upload=62126]: Next returned nil segment Dec 8 14:25:22.334251700 (70h56m22.205334735s) [upload=62126]: Waiting for error group managing segments... Dec 8 14:25:22.746429339 (70h56m22.617512384s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.747165611 (70h56m22.618248646s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.753178531 (70h56m22.624261566s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.922057260 (70h56m22.793140305s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.951038517 (70h56m22.822121562s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.651296259 (70h56m23.522379304s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.653766143 (70h56m23.524849178s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.680671276 (70h56m23.551754321s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.700290564 (70h56m23.571373609s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.707126822 (70h56m23.578209868s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.721842694 (70h56m23.592925739s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.730554192 (70h56m23.601637237s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670621120 (70h56m24.541704185s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670647519 (70h56m24.541730564s) [seg_pos={0 0}, upload=62126]: Segment reached optimal threshold of 13 pieces. Dec 8 14:25:24.670967409 (70h56m24.542050454s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection Dec 8 14:25:24.670977318 (70h56m24.542060363s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671144947 (70h56m24.542227982s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: read tcp 188.166.225.170:39576->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 14:25:24.671168231 (70h56m24.542251306s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671490626 (70h56m24.542573661s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 8 14:25:24.671499091 (70h56m24.542582126s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671551481 (70h56m24.542634526s) [seg_pos={0 0}, upload=62126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 14:25:24.671640871 (70h56m24.542723906s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 14:25:24.671647103 (70h56m24.542730138s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 14:25:25.114013719 (70h56m24.985096764s) [upload=62126]: Done uploading segments... Dec 8 14:25:25.114042193 (70h56m24.985125258s) [upload=62126]: Upload finished. err: <nil> Duration: 9.398698964s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X.X..XX....X.......X..X.....................................] duration: 6.645785931s trace: https://jaeger.dev.storj.io/trace/286e49a04bd90f74 FAIL (2025-12-08 15:06:13 UTC) orbiter warning: context deadline exceeded Duration: 6.645816548s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXX.XXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.000893662s trace: https://jaeger.dev.storj.io/trace/7f32ec44e65ccc36 FAIL (2025-12-08 15:05:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000928408s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 8.692474522s remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4a31039498bdbd4f FAIL (2025-12-08 15:05:47 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 8.692533114s core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 9.654361624s trace: https://jaeger.dev.storj.io/trace/29daaac43eca6484 FAIL (2025-12-08 15:05:45 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 9.65439066s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [XX.X..XX....X.......X..X....................................] duration: 27.626656013s trace: https://jaeger.dev.storj.io/trace/286e49a04bd90f74 FAIL (2025-12-08 15:05:34 UTC) orbiter warning: context deadline exceeded Duration: 27.626686911s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [.X.XXX.X...X.....X..........X...X...........................] duration: 17.384905206s trace: https://jaeger.dev.storj.io/trace/9d8451999396f06 FAIL (2025-12-08 15:05:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.384935403s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXX.XXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.002584011s trace: https://jaeger.dev.storj.io/trace/745d687995ca662b FAIL (2025-12-08 15:05:21 UTC) orbiter warning: context deadline exceeded Duration: 5.002610642s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 8.04386938s trace: https://jaeger.dev.storj.io/trace/5ce839c1456bbf8a FAIL (2025-12-08 15:05:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 25:20.182936263 (70h56m20.054019298s) [upload=62126]: Starting upload Dec 8 14:25:20.183037416 (70h56m20.054120461s) [upload=62126]: Done starting upload Dec 8 14:25:20.183136214 (70h56m20.054219259s) [upload=62126]: Uploading segments... Dec 8 14:25:20.183291900 (70h56m20.054374945s) [upload=62126]: Got next segment. Inline: false Dec 8 14:25:20.183343058 (70h56m20.054426103s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 14:25:20.183349700 (70h56m20.054432745s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 14:25:20.560968522 (70h56m20.432051567s) [seg_pos={0 0}, upload=62126]: Begin upload segment... Dec 8 14:25:20.561021322 (70h56m20.432104357s) [seg_pos={0 0}, upload=62126]: Done begin upload segment. Dec 8 14:25:20.561050337 (70h56m20.432133383s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.562561536 (70h56m20.433644581s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563077198 (70h56m20.434160243s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563560729 (70h56m20.434643774s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564043677 (70h56m20.435126732s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564505988 (70h56m20.435589043s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564975802 (70h56m20.436058848s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.566789025 (70h56m20.43787207s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.568723158 (70h56m20.439806204s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.570985847 (70h56m20.442068892s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573462843 (70h56m20.444545889s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573987674 (70h56m20.445070719s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.575740802 (70h56m20.446823837s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.577275113 (70h56m20.448358158s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.578540883 (70h56m20.449623928s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.580403731 (70h56m20.451486776s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:22.334236681 (70h56m22.205319726s) [upload=62126]: Next returned nil segment Dec 8 14:25:22.334251700 (70h56m22.205334735s) [upload=62126]: Waiting for error group managing segments... Dec 8 14:25:22.746429339 (70h56m22.617512384s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.747165611 (70h56m22.618248646s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.753178531 (70h56m22.624261566s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.922057260 (70h56m22.793140305s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.951038517 (70h56m22.822121562s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.651296259 (70h56m23.522379304s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.653766143 (70h56m23.524849178s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.680671276 (70h56m23.551754321s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.700290564 (70h56m23.571373609s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.707126822 (70h56m23.578209868s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.721842694 (70h56m23.592925739s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.730554192 (70h56m23.601637237s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670621120 (70h56m24.541704185s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670647519 (70h56m24.541730564s) [seg_pos={0 0}, upload=62126]: Segment reached optimal threshold of 13 pieces. Dec 8 14:25:24.670967409 (70h56m24.542050454s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection Dec 8 14:25:24.670977318 (70h56m24.542060363s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671144947 (70h56m24.542227982s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: read tcp 188.166.225.170:39576->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 14:25:24.671168231 (70h56m24.542251306s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671490626 (70h56m24.542573661s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 8 14:25:24.671499091 (70h56m24.542582126s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671551481 (70h56m24.542634526s) [seg_pos={0 0}, upload=62126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 14:25:24.671640871 (70h56m24.542723906s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 14:25:24.671647103 (70h56m24.542730138s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 14:25:25.114013719 (70h56m24.985096764s) [upload=62126]: Done uploading segments... Dec 8 14:25:25.114042193 (70h56m24.985125258s) [upload=62126]: Upload finished. err: <nil> Duration: 8.043902284s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [XXX.X..XX....X.......X..X...................................] duration: 13.878979816s trace: https://jaeger.dev.storj.io/trace/286e49a04bd90f74 FAIL (2025-12-08 15:04:57 UTC) orbiter warning: context deadline exceeded Duration: 13.879006287s edge gateway us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 698.793963ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4a31039498bdbd4f FAIL (2025-12-08 15:04:44 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 698.820052ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 10.987556796s trace: https://jaeger.dev.storj.io/trace/29daaac43eca6484 FAIL (2025-12-08 15:04:39 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 10.987577787s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.XXX.X..XX....X.......X..X..................................] duration: 16.547605064s trace: https://jaeger.dev.storj.io/trace/286e49a04bd90f74 FAIL (2025-12-08 15:04:38 UTC) orbiter warning: context deadline exceeded Duration: 16.547803743s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [X.X.XXX.X...X.....X..........X...X..........................] duration: 12.505956627s trace: https://jaeger.dev.storj.io/trace/9d8451999396f06 FAIL (2025-12-08 15:04:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.505987025s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 8.336140019s trace: https://jaeger.dev.storj.io/trace/5ce839c1456bbf8a FAIL (2025-12-08 15:04:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 25:20.182936263 (70h56m20.054019298s) [upload=62126]: Starting upload Dec 8 14:25:20.183037416 (70h56m20.054120461s) [upload=62126]: Done starting upload Dec 8 14:25:20.183136214 (70h56m20.054219259s) [upload=62126]: Uploading segments... Dec 8 14:25:20.183291900 (70h56m20.054374945s) [upload=62126]: Got next segment. Inline: false Dec 8 14:25:20.183343058 (70h56m20.054426103s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 14:25:20.183349700 (70h56m20.054432745s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 14:25:20.560968522 (70h56m20.432051567s) [seg_pos={0 0}, upload=62126]: Begin upload segment... Dec 8 14:25:20.561021322 (70h56m20.432104357s) [seg_pos={0 0}, upload=62126]: Done begin upload segment. Dec 8 14:25:20.561050337 (70h56m20.432133383s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.562561536 (70h56m20.433644581s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563077198 (70h56m20.434160243s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563560729 (70h56m20.434643774s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564043677 (70h56m20.435126732s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564505988 (70h56m20.435589043s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564975802 (70h56m20.436058848s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.566789025 (70h56m20.43787207s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.568723158 (70h56m20.439806204s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.570985847 (70h56m20.442068892s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573462843 (70h56m20.444545889s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573987674 (70h56m20.445070719s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.575740802 (70h56m20.446823837s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.577275113 (70h56m20.448358158s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.578540883 (70h56m20.449623928s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.580403731 (70h56m20.451486776s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:22.334236681 (70h56m22.205319726s) [upload=62126]: Next returned nil segment Dec 8 14:25:22.334251700 (70h56m22.205334735s) [upload=62126]: Waiting for error group managing segments... Dec 8 14:25:22.746429339 (70h56m22.617512384s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.747165611 (70h56m22.618248646s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.753178531 (70h56m22.624261566s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.922057260 (70h56m22.793140305s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.951038517 (70h56m22.822121562s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.651296259 (70h56m23.522379304s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.653766143 (70h56m23.524849178s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.680671276 (70h56m23.551754321s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.700290564 (70h56m23.571373609s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.707126822 (70h56m23.578209868s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.721842694 (70h56m23.592925739s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.730554192 (70h56m23.601637237s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670621120 (70h56m24.541704185s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670647519 (70h56m24.541730564s) [seg_pos={0 0}, upload=62126]: Segment reached optimal threshold of 13 pieces. Dec 8 14:25:24.670967409 (70h56m24.542050454s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection Dec 8 14:25:24.670977318 (70h56m24.542060363s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671144947 (70h56m24.542227982s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: read tcp 188.166.225.170:39576->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 14:25:24.671168231 (70h56m24.542251306s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671490626 (70h56m24.542573661s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 8 14:25:24.671499091 (70h56m24.542582126s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671551481 (70h56m24.542634526s) [seg_pos={0 0}, upload=62126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 14:25:24.671640871 (70h56m24.542723906s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 14:25:24.671647103 (70h56m24.542730138s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 14:25:25.114013719 (70h56m24.985096764s) [upload=62126]: Done uploading segments... Dec 8 14:25:25.114042193 (70h56m24.985125258s) [upload=62126]: Upload finished. err: <nil> Duration: 8.336170499s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXX.XXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.002011495s trace: https://jaeger.dev.storj.io/trace/3b90a0b6eeeaa7a FAIL (2025-12-08 15:04:29 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002037214s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 672.842836ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4a31039498bdbd4f FAIL (2025-12-08 15:04:24 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 672.867893ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 12.758920921s trace: https://jaeger.dev.storj.io/trace/29daaac43eca6484 FAIL (2025-12-08 15:03:51 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 12.758949897s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [X.XXX.X..XX....X.......X..X.................................] duration: 20.379293929s trace: https://jaeger.dev.storj.io/trace/286e49a04bd90f74 FAIL (2025-12-08 15:03:51 UTC) orbiter warning: context deadline exceeded Duration: 20.379325059s edge gateway us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 975.478883ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4a31039498bdbd4f FAIL (2025-12-08 15:03:49 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 975.505074ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 9.800210622s trace: https://jaeger.dev.storj.io/trace/5ce839c1456bbf8a FAIL (2025-12-08 15:03:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 25:20.182936263 (70h56m20.054019298s) [upload=62126]: Starting upload Dec 8 14:25:20.183037416 (70h56m20.054120461s) [upload=62126]: Done starting upload Dec 8 14:25:20.183136214 (70h56m20.054219259s) [upload=62126]: Uploading segments... Dec 8 14:25:20.183291900 (70h56m20.054374945s) [upload=62126]: Got next segment. Inline: false Dec 8 14:25:20.183343058 (70h56m20.054426103s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 14:25:20.183349700 (70h56m20.054432745s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 14:25:20.560968522 (70h56m20.432051567s) [seg_pos={0 0}, upload=62126]: Begin upload segment... Dec 8 14:25:20.561021322 (70h56m20.432104357s) [seg_pos={0 0}, upload=62126]: Done begin upload segment. Dec 8 14:25:20.561050337 (70h56m20.432133383s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.562561536 (70h56m20.433644581s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563077198 (70h56m20.434160243s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563560729 (70h56m20.434643774s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564043677 (70h56m20.435126732s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564505988 (70h56m20.435589043s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564975802 (70h56m20.436058848s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.566789025 (70h56m20.43787207s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.568723158 (70h56m20.439806204s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.570985847 (70h56m20.442068892s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573462843 (70h56m20.444545889s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573987674 (70h56m20.445070719s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.575740802 (70h56m20.446823837s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.577275113 (70h56m20.448358158s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.578540883 (70h56m20.449623928s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.580403731 (70h56m20.451486776s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:22.334236681 (70h56m22.205319726s) [upload=62126]: Next returned nil segment Dec 8 14:25:22.334251700 (70h56m22.205334735s) [upload=62126]: Waiting for error group managing segments... Dec 8 14:25:22.746429339 (70h56m22.617512384s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.747165611 (70h56m22.618248646s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.753178531 (70h56m22.624261566s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.922057260 (70h56m22.793140305s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.951038517 (70h56m22.822121562s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.651296259 (70h56m23.522379304s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.653766143 (70h56m23.524849178s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.680671276 (70h56m23.551754321s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.700290564 (70h56m23.571373609s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.707126822 (70h56m23.578209868s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.721842694 (70h56m23.592925739s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.730554192 (70h56m23.601637237s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670621120 (70h56m24.541704185s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670647519 (70h56m24.541730564s) [seg_pos={0 0}, upload=62126]: Segment reached optimal threshold of 13 pieces. Dec 8 14:25:24.670967409 (70h56m24.542050454s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection Dec 8 14:25:24.670977318 (70h56m24.542060363s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671144947 (70h56m24.542227982s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: read tcp 188.166.225.170:39576->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 14:25:24.671168231 (70h56m24.542251306s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671490626 (70h56m24.542573661s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 8 14:25:24.671499091 (70h56m24.542582126s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671551481 (70h56m24.542634526s) [seg_pos={0 0}, upload=62126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 14:25:24.671640871 (70h56m24.542723906s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 14:25:24.671647103 (70h56m24.542730138s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 14:25:25.114013719 (70h56m24.985096764s) [upload=62126]: Done uploading segments... Dec 8 14:25:25.114042193 (70h56m24.985125258s) [upload=62126]: Upload finished. err: <nil> Duration: 9.800241801s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [XX.X.XXX.X...X.....X..........X...X.........................] duration: 22.442587189s trace: https://jaeger.dev.storj.io/trace/9d8451999396f06 FAIL (2025-12-08 15:03:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.442617017s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXXX.XXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.000933998s trace: https://jaeger.dev.storj.io/trace/2c151adf8eec672b FAIL (2025-12-08 15:03:45 UTC) orbiter warning: context deadline exceeded Duration: 5.000960318s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.X.XXX.X..XX....X.......X..X................................] duration: 20.050469543s trace: https://jaeger.dev.storj.io/trace/286e49a04bd90f74 FAIL (2025-12-08 15:03:22 UTC) orbiter warning: context deadline exceeded Duration: 20.050500891s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 9.342603968s trace: https://jaeger.dev.storj.io/trace/29daaac43eca6484 FAIL (2025-12-08 15:03:20 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 9.342636921s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [XXX.X.XXX.X...X.....X..........X...X........................] duration: 15.323351422s trace: https://jaeger.dev.storj.io/trace/9d8451999396f06 FAIL (2025-12-08 15:03:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.32338058s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXXXX.XXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001608923s trace: https://jaeger.dev.storj.io/trace/aa322b15abe123c FAIL (2025-12-08 15:02:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001646124s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 13.894386844s trace: https://jaeger.dev.storj.io/trace/29daaac43eca6484 FAIL (2025-12-08 15:02:51 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 13.89441582s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X..........X................................................] duration: 14.641199786s trace: https://jaeger.dev.storj.io/trace/5ce839c1456bbf8a FAIL (2025-12-08 15:02:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 25:20.182936263 (70h56m20.054019298s) [upload=62126]: Starting upload Dec 8 14:25:20.183037416 (70h56m20.054120461s) [upload=62126]: Done starting upload Dec 8 14:25:20.183136214 (70h56m20.054219259s) [upload=62126]: Uploading segments... Dec 8 14:25:20.183291900 (70h56m20.054374945s) [upload=62126]: Got next segment. Inline: false Dec 8 14:25:20.183343058 (70h56m20.054426103s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 14:25:20.183349700 (70h56m20.054432745s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 14:25:20.560968522 (70h56m20.432051567s) [seg_pos={0 0}, upload=62126]: Begin upload segment... Dec 8 14:25:20.561021322 (70h56m20.432104357s) [seg_pos={0 0}, upload=62126]: Done begin upload segment. Dec 8 14:25:20.561050337 (70h56m20.432133383s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.562561536 (70h56m20.433644581s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563077198 (70h56m20.434160243s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563560729 (70h56m20.434643774s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564043677 (70h56m20.435126732s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564505988 (70h56m20.435589043s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564975802 (70h56m20.436058848s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.566789025 (70h56m20.43787207s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.568723158 (70h56m20.439806204s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.570985847 (70h56m20.442068892s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573462843 (70h56m20.444545889s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573987674 (70h56m20.445070719s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.575740802 (70h56m20.446823837s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.577275113 (70h56m20.448358158s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.578540883 (70h56m20.449623928s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.580403731 (70h56m20.451486776s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:22.334236681 (70h56m22.205319726s) [upload=62126]: Next returned nil segment Dec 8 14:25:22.334251700 (70h56m22.205334735s) [upload=62126]: Waiting for error group managing segments... Dec 8 14:25:22.746429339 (70h56m22.617512384s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.747165611 (70h56m22.618248646s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.753178531 (70h56m22.624261566s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.922057260 (70h56m22.793140305s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.951038517 (70h56m22.822121562s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.651296259 (70h56m23.522379304s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.653766143 (70h56m23.524849178s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.680671276 (70h56m23.551754321s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.700290564 (70h56m23.571373609s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.707126822 (70h56m23.578209868s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.721842694 (70h56m23.592925739s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.730554192 (70h56m23.601637237s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670621120 (70h56m24.541704185s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670647519 (70h56m24.541730564s) [seg_pos={0 0}, upload=62126]: Segment reached optimal threshold of 13 pieces. Dec 8 14:25:24.670967409 (70h56m24.542050454s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection Dec 8 14:25:24.670977318 (70h56m24.542060363s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671144947 (70h56m24.542227982s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: read tcp 188.166.225.170:39576->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 14:25:24.671168231 (70h56m24.542251306s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671490626 (70h56m24.542573661s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 8 14:25:24.671499091 (70h56m24.542582126s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671551481 (70h56m24.542634526s) [seg_pos={0 0}, upload=62126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 14:25:24.671640871 (70h56m24.542723906s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 14:25:24.671647103 (70h56m24.542730138s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 14:25:25.114013719 (70h56m24.985096764s) [upload=62126]: Done uploading segments... Dec 8 14:25:25.114042193 (70h56m24.985125258s) [upload=62126]: Upload finished. err: <nil> Duration: 14.641223461s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 893.253472ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4a31039498bdbd4f FAIL (2025-12-08 15:02:41 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 893.281654ms core satellite us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..X.XXX.X..XX....X.......X..X...............................] duration: 22.643612778s trace: https://jaeger.dev.storj.io/trace/286e49a04bd90f74 FAIL (2025-12-08 15:02:38 UTC) orbiter warning: context deadline exceeded Duration: 22.643643096s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [.XXX.X.XXX.X...X.....X..........X...X.......................] duration: 23.824477454s trace: https://jaeger.dev.storj.io/trace/9d8451999396f06 FAIL (2025-12-08 15:02:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 23.824548921s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 1.055984323s remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4a31039498bdbd4f FAIL (2025-12-08 15:02:07 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 1.056016795s core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 16.866879715s trace: https://jaeger.dev.storj.io/trace/29daaac43eca6484 FAIL (2025-12-08 15:02:04 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 16.866910555s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X..........X...............................................] duration: 10.841736841s trace: https://jaeger.dev.storj.io/trace/5ce839c1456bbf8a FAIL (2025-12-08 15:01:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 25:20.182936263 (70h56m20.054019298s) [upload=62126]: Starting upload Dec 8 14:25:20.183037416 (70h56m20.054120461s) [upload=62126]: Done starting upload Dec 8 14:25:20.183136214 (70h56m20.054219259s) [upload=62126]: Uploading segments... Dec 8 14:25:20.183291900 (70h56m20.054374945s) [upload=62126]: Got next segment. Inline: false Dec 8 14:25:20.183343058 (70h56m20.054426103s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 14:25:20.183349700 (70h56m20.054432745s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 14:25:20.560968522 (70h56m20.432051567s) [seg_pos={0 0}, upload=62126]: Begin upload segment... Dec 8 14:25:20.561021322 (70h56m20.432104357s) [seg_pos={0 0}, upload=62126]: Done begin upload segment. Dec 8 14:25:20.561050337 (70h56m20.432133383s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.562561536 (70h56m20.433644581s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563077198 (70h56m20.434160243s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563560729 (70h56m20.434643774s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564043677 (70h56m20.435126732s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564505988 (70h56m20.435589043s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564975802 (70h56m20.436058848s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.566789025 (70h56m20.43787207s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.568723158 (70h56m20.439806204s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.570985847 (70h56m20.442068892s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573462843 (70h56m20.444545889s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573987674 (70h56m20.445070719s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.575740802 (70h56m20.446823837s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.577275113 (70h56m20.448358158s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.578540883 (70h56m20.449623928s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.580403731 (70h56m20.451486776s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:22.334236681 (70h56m22.205319726s) [upload=62126]: Next returned nil segment Dec 8 14:25:22.334251700 (70h56m22.205334735s) [upload=62126]: Waiting for error group managing segments... Dec 8 14:25:22.746429339 (70h56m22.617512384s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.747165611 (70h56m22.618248646s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.753178531 (70h56m22.624261566s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.922057260 (70h56m22.793140305s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.951038517 (70h56m22.822121562s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.651296259 (70h56m23.522379304s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.653766143 (70h56m23.524849178s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.680671276 (70h56m23.551754321s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.700290564 (70h56m23.571373609s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.707126822 (70h56m23.578209868s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.721842694 (70h56m23.592925739s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.730554192 (70h56m23.601637237s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670621120 (70h56m24.541704185s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670647519 (70h56m24.541730564s) [seg_pos={0 0}, upload=62126]: Segment reached optimal threshold of 13 pieces. Dec 8 14:25:24.670967409 (70h56m24.542050454s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection Dec 8 14:25:24.670977318 (70h56m24.542060363s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671144947 (70h56m24.542227982s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: read tcp 188.166.225.170:39576->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 14:25:24.671168231 (70h56m24.542251306s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671490626 (70h56m24.542573661s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 8 14:25:24.671499091 (70h56m24.542582126s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671551481 (70h56m24.542634526s) [seg_pos={0 0}, upload=62126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 14:25:24.671640871 (70h56m24.542723906s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 14:25:24.671647103 (70h56m24.542730138s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 14:25:25.114013719 (70h56m24.985096764s) [upload=62126]: Done uploading segments... Dec 8 14:25:25.114042193 (70h56m24.985125258s) [upload=62126]: Upload finished. err: <nil> Duration: 10.841764484s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXXXXX.XXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001595039s trace: https://jaeger.dev.storj.io/trace/717d3290a421055c FAIL (2025-12-08 15:01:49 UTC) orbiter warning: context deadline exceeded Duration: 5.001611349s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 1.035232939s remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4a31039498bdbd4f FAIL (2025-12-08 15:01:34 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 1.035258617s core satellite us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...X.XXX.X..XX....X.......X..X..............................] duration: 21.719054809s trace: https://jaeger.dev.storj.io/trace/286e49a04bd90f74 FAIL (2025-12-08 15:01:23 UTC) orbiter warning: context deadline exceeded Duration: 21.719095506s edge gateway us-select-1

1
...
11711
11712
11713
11714
11715
...
17282