Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 864086 reports (Page 11714 of 17282)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 11.195258986s trace: https://jaeger.dev.storj.io/trace/29daaac43eca6484 FAIL (2025-12-08 15:00:09 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 11.195292448s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [X..XXX.X.XXX.X...X.....X..........X...X.....................] duration: 12.535453606s trace: https://jaeger.dev.storj.io/trace/9d8451999396f06 FAIL (2025-12-08 15:00:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.535483594s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X..........X............................................] duration: 8.743211434s trace: https://jaeger.dev.storj.io/trace/5ce839c1456bbf8a FAIL (2025-12-08 14:59:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 25:20.182936263 (70h56m20.054019298s) [upload=62126]: Starting upload Dec 8 14:25:20.183037416 (70h56m20.054120461s) [upload=62126]: Done starting upload Dec 8 14:25:20.183136214 (70h56m20.054219259s) [upload=62126]: Uploading segments... Dec 8 14:25:20.183291900 (70h56m20.054374945s) [upload=62126]: Got next segment. Inline: false Dec 8 14:25:20.183343058 (70h56m20.054426103s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 14:25:20.183349700 (70h56m20.054432745s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 14:25:20.560968522 (70h56m20.432051567s) [seg_pos={0 0}, upload=62126]: Begin upload segment... Dec 8 14:25:20.561021322 (70h56m20.432104357s) [seg_pos={0 0}, upload=62126]: Done begin upload segment. Dec 8 14:25:20.561050337 (70h56m20.432133383s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.562561536 (70h56m20.433644581s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563077198 (70h56m20.434160243s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563560729 (70h56m20.434643774s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564043677 (70h56m20.435126732s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564505988 (70h56m20.435589043s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564975802 (70h56m20.436058848s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.566789025 (70h56m20.43787207s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.568723158 (70h56m20.439806204s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.570985847 (70h56m20.442068892s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573462843 (70h56m20.444545889s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573987674 (70h56m20.445070719s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.575740802 (70h56m20.446823837s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.577275113 (70h56m20.448358158s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.578540883 (70h56m20.449623928s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.580403731 (70h56m20.451486776s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:22.334236681 (70h56m22.205319726s) [upload=62126]: Next returned nil segment Dec 8 14:25:22.334251700 (70h56m22.205334735s) [upload=62126]: Waiting for error group managing segments... Dec 8 14:25:22.746429339 (70h56m22.617512384s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.747165611 (70h56m22.618248646s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.753178531 (70h56m22.624261566s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.922057260 (70h56m22.793140305s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.951038517 (70h56m22.822121562s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.651296259 (70h56m23.522379304s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.653766143 (70h56m23.524849178s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.680671276 (70h56m23.551754321s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.700290564 (70h56m23.571373609s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.707126822 (70h56m23.578209868s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.721842694 (70h56m23.592925739s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.730554192 (70h56m23.601637237s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670621120 (70h56m24.541704185s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670647519 (70h56m24.541730564s) [seg_pos={0 0}, upload=62126]: Segment reached optimal threshold of 13 pieces. Dec 8 14:25:24.670967409 (70h56m24.542050454s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection Dec 8 14:25:24.670977318 (70h56m24.542060363s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671144947 (70h56m24.542227982s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: read tcp 188.166.225.170:39576->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 14:25:24.671168231 (70h56m24.542251306s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671490626 (70h56m24.542573661s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 8 14:25:24.671499091 (70h56m24.542582126s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671551481 (70h56m24.542634526s) [seg_pos={0 0}, upload=62126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 14:25:24.671640871 (70h56m24.542723906s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 14:25:24.671647103 (70h56m24.542730138s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 14:25:25.114013719 (70h56m24.985096764s) [upload=62126]: Done uploading segments... Dec 8 14:25:25.114042193 (70h56m24.985125258s) [upload=62126]: Upload finished. err: <nil> Duration: 8.743242834s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.X....X.XXX.X..XX....X.......X..X...........................] duration: 22.644326309s trace: https://jaeger.dev.storj.io/trace/286e49a04bd90f74 FAIL (2025-12-08 14:59:53 UTC) orbiter warning: context deadline exceeded Duration: 22.644354623s edge gateway us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 955.247806ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4a31039498bdbd4f FAIL (2025-12-08 14:59:28 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 955.276811ms core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [.X..XXX.X.XXX.X...X.....X..........X...X....................] duration: 17.210535105s trace: https://jaeger.dev.storj.io/trace/9d8451999396f06 FAIL (2025-12-08 14:59:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.21055907s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXXXXXXXXX.XXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001683358s trace: https://jaeger.dev.storj.io/trace/23a35addb674ac10 FAIL (2025-12-08 14:59:13 UTC) orbiter warning: context deadline exceeded Duration: 5.001707513s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X.X....X.XXX.X..XX....X.......X..X..........................] duration: 7.02691131s trace: https://jaeger.dev.storj.io/trace/286e49a04bd90f74 FAIL (2025-12-08 14:59:10 UTC) orbiter warning: context deadline exceeded Duration: 7.02693653s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.X.X....X.XXX.X..XX....X.......X..X.........................] duration: 15.268914794s trace: https://jaeger.dev.storj.io/trace/286e49a04bd90f74 FAIL (2025-12-08 14:58:58 UTC) orbiter warning: context deadline exceeded Duration: 15.268936084s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 7.966840339s trace: https://jaeger.dev.storj.io/trace/29daaac43eca6484 FAIL (2025-12-08 14:58:52 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 7.966864817s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X..........X...........................................] duration: 9.297055331s trace: https://jaeger.dev.storj.io/trace/5ce839c1456bbf8a FAIL (2025-12-08 14:58:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 25:20.182936263 (70h56m20.054019298s) [upload=62126]: Starting upload Dec 8 14:25:20.183037416 (70h56m20.054120461s) [upload=62126]: Done starting upload Dec 8 14:25:20.183136214 (70h56m20.054219259s) [upload=62126]: Uploading segments... Dec 8 14:25:20.183291900 (70h56m20.054374945s) [upload=62126]: Got next segment. Inline: false Dec 8 14:25:20.183343058 (70h56m20.054426103s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 14:25:20.183349700 (70h56m20.054432745s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 14:25:20.560968522 (70h56m20.432051567s) [seg_pos={0 0}, upload=62126]: Begin upload segment... Dec 8 14:25:20.561021322 (70h56m20.432104357s) [seg_pos={0 0}, upload=62126]: Done begin upload segment. Dec 8 14:25:20.561050337 (70h56m20.432133383s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.562561536 (70h56m20.433644581s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563077198 (70h56m20.434160243s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563560729 (70h56m20.434643774s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564043677 (70h56m20.435126732s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564505988 (70h56m20.435589043s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564975802 (70h56m20.436058848s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.566789025 (70h56m20.43787207s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.568723158 (70h56m20.439806204s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.570985847 (70h56m20.442068892s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573462843 (70h56m20.444545889s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573987674 (70h56m20.445070719s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.575740802 (70h56m20.446823837s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.577275113 (70h56m20.448358158s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.578540883 (70h56m20.449623928s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.580403731 (70h56m20.451486776s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:22.334236681 (70h56m22.205319726s) [upload=62126]: Next returned nil segment Dec 8 14:25:22.334251700 (70h56m22.205334735s) [upload=62126]: Waiting for error group managing segments... Dec 8 14:25:22.746429339 (70h56m22.617512384s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.747165611 (70h56m22.618248646s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.753178531 (70h56m22.624261566s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.922057260 (70h56m22.793140305s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.951038517 (70h56m22.822121562s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.651296259 (70h56m23.522379304s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.653766143 (70h56m23.524849178s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.680671276 (70h56m23.551754321s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.700290564 (70h56m23.571373609s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.707126822 (70h56m23.578209868s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.721842694 (70h56m23.592925739s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.730554192 (70h56m23.601637237s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670621120 (70h56m24.541704185s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670647519 (70h56m24.541730564s) [seg_pos={0 0}, upload=62126]: Segment reached optimal threshold of 13 pieces. Dec 8 14:25:24.670967409 (70h56m24.542050454s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection Dec 8 14:25:24.670977318 (70h56m24.542060363s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671144947 (70h56m24.542227982s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: read tcp 188.166.225.170:39576->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 14:25:24.671168231 (70h56m24.542251306s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671490626 (70h56m24.542573661s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 8 14:25:24.671499091 (70h56m24.542582126s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671551481 (70h56m24.542634526s) [seg_pos={0 0}, upload=62126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 14:25:24.671640871 (70h56m24.542723906s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 14:25:24.671647103 (70h56m24.542730138s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 14:25:25.114013719 (70h56m24.985096764s) [upload=62126]: Done uploading segments... Dec 8 14:25:25.114042193 (70h56m24.985125258s) [upload=62126]: Upload finished. err: <nil> Duration: 9.297088845s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [X.X..XXX.X.XXX.X...X.....X..........X...X...................] duration: 15.685263923s trace: https://jaeger.dev.storj.io/trace/9d8451999396f06 FAIL (2025-12-08 14:58:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.685290083s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXX.XXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.000852076s trace: https://jaeger.dev.storj.io/trace/63b5c94efe721631 FAIL (2025-12-08 14:58:28 UTC) orbiter warning: context deadline exceeded Duration: 5.000876722s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X..........X..........................................] duration: 9.340932165s trace: https://jaeger.dev.storj.io/trace/5ce839c1456bbf8a FAIL (2025-12-08 14:58:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 25:20.182936263 (70h56m20.054019298s) [upload=62126]: Starting upload Dec 8 14:25:20.183037416 (70h56m20.054120461s) [upload=62126]: Done starting upload Dec 8 14:25:20.183136214 (70h56m20.054219259s) [upload=62126]: Uploading segments... Dec 8 14:25:20.183291900 (70h56m20.054374945s) [upload=62126]: Got next segment. Inline: false Dec 8 14:25:20.183343058 (70h56m20.054426103s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 14:25:20.183349700 (70h56m20.054432745s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 14:25:20.560968522 (70h56m20.432051567s) [seg_pos={0 0}, upload=62126]: Begin upload segment... Dec 8 14:25:20.561021322 (70h56m20.432104357s) [seg_pos={0 0}, upload=62126]: Done begin upload segment. Dec 8 14:25:20.561050337 (70h56m20.432133383s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.562561536 (70h56m20.433644581s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563077198 (70h56m20.434160243s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563560729 (70h56m20.434643774s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564043677 (70h56m20.435126732s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564505988 (70h56m20.435589043s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564975802 (70h56m20.436058848s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.566789025 (70h56m20.43787207s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.568723158 (70h56m20.439806204s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.570985847 (70h56m20.442068892s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573462843 (70h56m20.444545889s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573987674 (70h56m20.445070719s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.575740802 (70h56m20.446823837s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.577275113 (70h56m20.448358158s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.578540883 (70h56m20.449623928s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.580403731 (70h56m20.451486776s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:22.334236681 (70h56m22.205319726s) [upload=62126]: Next returned nil segment Dec 8 14:25:22.334251700 (70h56m22.205334735s) [upload=62126]: Waiting for error group managing segments... Dec 8 14:25:22.746429339 (70h56m22.617512384s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.747165611 (70h56m22.618248646s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.753178531 (70h56m22.624261566s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.922057260 (70h56m22.793140305s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.951038517 (70h56m22.822121562s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.651296259 (70h56m23.522379304s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.653766143 (70h56m23.524849178s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.680671276 (70h56m23.551754321s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.700290564 (70h56m23.571373609s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.707126822 (70h56m23.578209868s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.721842694 (70h56m23.592925739s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.730554192 (70h56m23.601637237s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670621120 (70h56m24.541704185s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670647519 (70h56m24.541730564s) [seg_pos={0 0}, upload=62126]: Segment reached optimal threshold of 13 pieces. Dec 8 14:25:24.670967409 (70h56m24.542050454s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection Dec 8 14:25:24.670977318 (70h56m24.542060363s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671144947 (70h56m24.542227982s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: read tcp 188.166.225.170:39576->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 14:25:24.671168231 (70h56m24.542251306s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671490626 (70h56m24.542573661s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 8 14:25:24.671499091 (70h56m24.542582126s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671551481 (70h56m24.542634526s) [seg_pos={0 0}, upload=62126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 14:25:24.671640871 (70h56m24.542723906s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 14:25:24.671647103 (70h56m24.542730138s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 14:25:25.114013719 (70h56m24.985096764s) [upload=62126]: Done uploading segments... Dec 8 14:25:25.114042193 (70h56m24.985125258s) [upload=62126]: Upload finished. err: <nil> Duration: 9.340957714s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 1.051938855s remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4a31039498bdbd4f FAIL (2025-12-08 14:58:16 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 1.051960927s core satellite us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X.X.X....X.XXX.X..XX....X.......X..X........................] duration: 18.014367367s trace: https://jaeger.dev.storj.io/trace/286e49a04bd90f74 FAIL (2025-12-08 14:58:15 UTC) orbiter warning: context deadline exceeded Duration: 18.014395471s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXX.XXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001688357s trace: https://jaeger.dev.storj.io/trace/495b16a1fe602e2c FAIL (2025-12-08 14:57:48 UTC) orbiter warning: context deadline exceeded Duration: 5.001709708s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 9.804265003s trace: https://jaeger.dev.storj.io/trace/29daaac43eca6484 FAIL (2025-12-08 14:57:42 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 9.804285973s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 1.010472074s remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4a31039498bdbd4f FAIL (2025-12-08 14:57:38 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 1.01049513s core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [XX.X..XXX.X.XXX.X...X.....X..........X...X..................] duration: 16.0091382s trace: https://jaeger.dev.storj.io/trace/9d8451999396f06 FAIL (2025-12-08 14:57:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.009263188s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXX.XXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001877296s trace: https://jaeger.dev.storj.io/trace/30f3026d28f54ec6 FAIL (2025-12-08 14:57:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001902935s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X..........X.........................................] duration: 14.584772981s trace: https://jaeger.dev.storj.io/trace/5ce839c1456bbf8a FAIL (2025-12-08 14:57:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 25:20.182936263 (70h56m20.054019298s) [upload=62126]: Starting upload Dec 8 14:25:20.183037416 (70h56m20.054120461s) [upload=62126]: Done starting upload Dec 8 14:25:20.183136214 (70h56m20.054219259s) [upload=62126]: Uploading segments... Dec 8 14:25:20.183291900 (70h56m20.054374945s) [upload=62126]: Got next segment. Inline: false Dec 8 14:25:20.183343058 (70h56m20.054426103s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 14:25:20.183349700 (70h56m20.054432745s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 14:25:20.560968522 (70h56m20.432051567s) [seg_pos={0 0}, upload=62126]: Begin upload segment... Dec 8 14:25:20.561021322 (70h56m20.432104357s) [seg_pos={0 0}, upload=62126]: Done begin upload segment. Dec 8 14:25:20.561050337 (70h56m20.432133383s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.562561536 (70h56m20.433644581s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563077198 (70h56m20.434160243s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563560729 (70h56m20.434643774s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564043677 (70h56m20.435126732s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564505988 (70h56m20.435589043s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564975802 (70h56m20.436058848s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.566789025 (70h56m20.43787207s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.568723158 (70h56m20.439806204s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.570985847 (70h56m20.442068892s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573462843 (70h56m20.444545889s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573987674 (70h56m20.445070719s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.575740802 (70h56m20.446823837s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.577275113 (70h56m20.448358158s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.578540883 (70h56m20.449623928s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.580403731 (70h56m20.451486776s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:22.334236681 (70h56m22.205319726s) [upload=62126]: Next returned nil segment Dec 8 14:25:22.334251700 (70h56m22.205334735s) [upload=62126]: Waiting for error group managing segments... Dec 8 14:25:22.746429339 (70h56m22.617512384s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.747165611 (70h56m22.618248646s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.753178531 (70h56m22.624261566s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.922057260 (70h56m22.793140305s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.951038517 (70h56m22.822121562s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.651296259 (70h56m23.522379304s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.653766143 (70h56m23.524849178s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.680671276 (70h56m23.551754321s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.700290564 (70h56m23.571373609s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.707126822 (70h56m23.578209868s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.721842694 (70h56m23.592925739s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.730554192 (70h56m23.601637237s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670621120 (70h56m24.541704185s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670647519 (70h56m24.541730564s) [seg_pos={0 0}, upload=62126]: Segment reached optimal threshold of 13 pieces. Dec 8 14:25:24.670967409 (70h56m24.542050454s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection Dec 8 14:25:24.670977318 (70h56m24.542060363s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671144947 (70h56m24.542227982s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: read tcp 188.166.225.170:39576->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 14:25:24.671168231 (70h56m24.542251306s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671490626 (70h56m24.542573661s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 8 14:25:24.671499091 (70h56m24.542582126s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671551481 (70h56m24.542634526s) [seg_pos={0 0}, upload=62126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 14:25:24.671640871 (70h56m24.542723906s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 14:25:24.671647103 (70h56m24.542730138s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 14:25:25.114013719 (70h56m24.985096764s) [upload=62126]: Done uploading segments... Dec 8 14:25:25.114042193 (70h56m24.985125258s) [upload=62126]: Upload finished. err: <nil> Duration: 14.584793521s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 891.911743ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4a31039498bdbd4f FAIL (2025-12-08 14:57:21 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 891.940878ms core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [XXX.X..XXX.X.XXX.X...X.....X..........X...X.................] duration: 14.120572588s trace: https://jaeger.dev.storj.io/trace/9d8451999396f06 FAIL (2025-12-08 14:57:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.120610521s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X..........X........................................] duration: 26.053183037s trace: https://jaeger.dev.storj.io/trace/5ce839c1456bbf8a FAIL (2025-12-08 14:57:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 25:20.182936263 (70h56m20.054019298s) [upload=62126]: Starting upload Dec 8 14:25:20.183037416 (70h56m20.054120461s) [upload=62126]: Done starting upload Dec 8 14:25:20.183136214 (70h56m20.054219259s) [upload=62126]: Uploading segments... Dec 8 14:25:20.183291900 (70h56m20.054374945s) [upload=62126]: Got next segment. Inline: false Dec 8 14:25:20.183343058 (70h56m20.054426103s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 14:25:20.183349700 (70h56m20.054432745s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 14:25:20.560968522 (70h56m20.432051567s) [seg_pos={0 0}, upload=62126]: Begin upload segment... Dec 8 14:25:20.561021322 (70h56m20.432104357s) [seg_pos={0 0}, upload=62126]: Done begin upload segment. Dec 8 14:25:20.561050337 (70h56m20.432133383s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.562561536 (70h56m20.433644581s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563077198 (70h56m20.434160243s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563560729 (70h56m20.434643774s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564043677 (70h56m20.435126732s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564505988 (70h56m20.435589043s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564975802 (70h56m20.436058848s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.566789025 (70h56m20.43787207s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.568723158 (70h56m20.439806204s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.570985847 (70h56m20.442068892s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573462843 (70h56m20.444545889s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573987674 (70h56m20.445070719s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.575740802 (70h56m20.446823837s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.577275113 (70h56m20.448358158s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.578540883 (70h56m20.449623928s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.580403731 (70h56m20.451486776s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:22.334236681 (70h56m22.205319726s) [upload=62126]: Next returned nil segment Dec 8 14:25:22.334251700 (70h56m22.205334735s) [upload=62126]: Waiting for error group managing segments... Dec 8 14:25:22.746429339 (70h56m22.617512384s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.747165611 (70h56m22.618248646s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.753178531 (70h56m22.624261566s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.922057260 (70h56m22.793140305s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.951038517 (70h56m22.822121562s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.651296259 (70h56m23.522379304s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.653766143 (70h56m23.524849178s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.680671276 (70h56m23.551754321s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.700290564 (70h56m23.571373609s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.707126822 (70h56m23.578209868s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.721842694 (70h56m23.592925739s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.730554192 (70h56m23.601637237s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670621120 (70h56m24.541704185s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670647519 (70h56m24.541730564s) [seg_pos={0 0}, upload=62126]: Segment reached optimal threshold of 13 pieces. Dec 8 14:25:24.670967409 (70h56m24.542050454s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection Dec 8 14:25:24.670977318 (70h56m24.542060363s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671144947 (70h56m24.542227982s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: read tcp 188.166.225.170:39576->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 14:25:24.671168231 (70h56m24.542251306s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671490626 (70h56m24.542573661s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 8 14:25:24.671499091 (70h56m24.542582126s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671551481 (70h56m24.542634526s) [seg_pos={0 0}, upload=62126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 14:25:24.671640871 (70h56m24.542723906s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 14:25:24.671647103 (70h56m24.542730138s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 14:25:25.114013719 (70h56m24.985096764s) [upload=62126]: Done uploading segments... Dec 8 14:25:25.114042193 (70h56m24.985125258s) [upload=62126]: Upload finished. err: <nil> Duration: 26.053209577s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [XX.X.X....X.XXX.X..XX....X.......X..X.......................] duration: 7.211837166s trace: https://jaeger.dev.storj.io/trace/286e49a04bd90f74 FAIL (2025-12-08 14:56:57 UTC) orbiter warning: context deadline exceeded Duration: 7.211868665s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 8.626795891s trace: https://jaeger.dev.storj.io/trace/29daaac43eca6484 FAIL (2025-12-08 14:56:53 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 8.62682702s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 1.021780867s remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4a31039498bdbd4f FAIL (2025-12-08 14:56:46 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 1.021817677s core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXX.XXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001180057s trace: https://jaeger.dev.storj.io/trace/424d67cb6087973a FAIL (2025-12-08 14:56:40 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001199986s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [.XXX.X..XXX.X.XXX.X...X.....X..........X...X................] duration: 21.886733264s trace: https://jaeger.dev.storj.io/trace/9d8451999396f06 FAIL (2025-12-08 14:56:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.886763382s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 10.661439163s trace: https://jaeger.dev.storj.io/trace/29daaac43eca6484 FAIL (2025-12-08 14:56:28 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 10.661469341s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [X.XXX.X..XXX.X.XXX.X...X.....X..........X...X...............] duration: 12.437267013s trace: https://jaeger.dev.storj.io/trace/9d8451999396f06 FAIL (2025-12-08 14:56:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.437293683s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [XXX.X.X....X.XXX.X..XX....X.......X..X......................] duration: 24.495097844s trace: https://jaeger.dev.storj.io/trace/286e49a04bd90f74 FAIL (2025-12-08 14:56:00 UTC) orbiter warning: context deadline exceeded Duration: 24.495118643s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X..........X.......................................] duration: 8.591664233s trace: https://jaeger.dev.storj.io/trace/5ce839c1456bbf8a FAIL (2025-12-08 14:55:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 25:20.182936263 (70h56m20.054019298s) [upload=62126]: Starting upload Dec 8 14:25:20.183037416 (70h56m20.054120461s) [upload=62126]: Done starting upload Dec 8 14:25:20.183136214 (70h56m20.054219259s) [upload=62126]: Uploading segments... Dec 8 14:25:20.183291900 (70h56m20.054374945s) [upload=62126]: Got next segment. Inline: false Dec 8 14:25:20.183343058 (70h56m20.054426103s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 14:25:20.183349700 (70h56m20.054432745s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 14:25:20.560968522 (70h56m20.432051567s) [seg_pos={0 0}, upload=62126]: Begin upload segment... Dec 8 14:25:20.561021322 (70h56m20.432104357s) [seg_pos={0 0}, upload=62126]: Done begin upload segment. Dec 8 14:25:20.561050337 (70h56m20.432133383s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.562561536 (70h56m20.433644581s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563077198 (70h56m20.434160243s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563560729 (70h56m20.434643774s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564043677 (70h56m20.435126732s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564505988 (70h56m20.435589043s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564975802 (70h56m20.436058848s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.566789025 (70h56m20.43787207s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.568723158 (70h56m20.439806204s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.570985847 (70h56m20.442068892s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573462843 (70h56m20.444545889s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573987674 (70h56m20.445070719s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.575740802 (70h56m20.446823837s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.577275113 (70h56m20.448358158s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.578540883 (70h56m20.449623928s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.580403731 (70h56m20.451486776s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:22.334236681 (70h56m22.205319726s) [upload=62126]: Next returned nil segment Dec 8 14:25:22.334251700 (70h56m22.205334735s) [upload=62126]: Waiting for error group managing segments... Dec 8 14:25:22.746429339 (70h56m22.617512384s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.747165611 (70h56m22.618248646s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.753178531 (70h56m22.624261566s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.922057260 (70h56m22.793140305s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.951038517 (70h56m22.822121562s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.651296259 (70h56m23.522379304s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.653766143 (70h56m23.524849178s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.680671276 (70h56m23.551754321s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.700290564 (70h56m23.571373609s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.707126822 (70h56m23.578209868s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.721842694 (70h56m23.592925739s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.730554192 (70h56m23.601637237s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670621120 (70h56m24.541704185s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670647519 (70h56m24.541730564s) [seg_pos={0 0}, upload=62126]: Segment reached optimal threshold of 13 pieces. Dec 8 14:25:24.670967409 (70h56m24.542050454s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection Dec 8 14:25:24.670977318 (70h56m24.542060363s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671144947 (70h56m24.542227982s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: read tcp 188.166.225.170:39576->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 14:25:24.671168231 (70h56m24.542251306s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671490626 (70h56m24.542573661s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 8 14:25:24.671499091 (70h56m24.542582126s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671551481 (70h56m24.542634526s) [seg_pos={0 0}, upload=62126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 14:25:24.671640871 (70h56m24.542723906s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 14:25:24.671647103 (70h56m24.542730138s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 14:25:25.114013719 (70h56m24.985096764s) [upload=62126]: Done uploading segments... Dec 8 14:25:25.114042193 (70h56m24.985125258s) [upload=62126]: Upload finished. err: <nil> Duration: 8.591690384s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [XX.XXX.X..XXX.X.XXX.X...X.....X..........X...X..............] duration: 15.805122677s trace: https://jaeger.dev.storj.io/trace/9d8451999396f06 FAIL (2025-12-08 14:55:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.805151292s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXX.XXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001191524s trace: https://jaeger.dev.storj.io/trace/18fdc867553573de FAIL (2025-12-08 14:55:38 UTC) orbiter warning: context deadline exceeded Duration: 5.001214547s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [XXXX.X.X....X.XXX.X..XX....X.......X..X.....................] duration: 20.744320868s trace: https://jaeger.dev.storj.io/trace/286e49a04bd90f74 FAIL (2025-12-08 14:55:33 UTC) orbiter warning: context deadline exceeded Duration: 20.744351257s edge gateway us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 1.000305104s remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4a31039498bdbd4f FAIL (2025-12-08 14:55:32 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 1.000358205s core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [.XX.XXX.X..XXX.X.XXX.X...X.....X..........X...X.............] duration: 18.677581228s trace: https://jaeger.dev.storj.io/trace/9d8451999396f06 FAIL (2025-12-08 14:55:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.677613238s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 9.228712688s trace: https://jaeger.dev.storj.io/trace/29daaac43eca6484 FAIL (2025-12-08 14:55:27 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 9.228764026s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 1.036143008s remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4a31039498bdbd4f FAIL (2025-12-08 14:55:16 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 1.03618604s core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 10.892686694s trace: https://jaeger.dev.storj.io/trace/29daaac43eca6484 FAIL (2025-12-08 14:55:02 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 10.892724516s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [XXXXX.X.X....X.XXX.X..XX....X.......X..X....................] duration: 26.773994331s trace: https://jaeger.dev.storj.io/trace/286e49a04bd90f74 FAIL (2025-12-08 14:54:58 UTC) orbiter warning: context deadline exceeded Duration: 26.774026702s edge gateway us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 1.05565973s remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4a31039498bdbd4f FAIL (2025-12-08 14:54:55 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 1.055681671s core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 19 out of 60 is failed (threshold=3) [X.XX.XXX.X..XXX.X.XXX.X...X.....X..........X...X............] duration: 15.952268339s trace: https://jaeger.dev.storj.io/trace/9d8451999396f06 FAIL (2025-12-08 14:54:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.952296301s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X..........X......................................] duration: 9.948495019s trace: https://jaeger.dev.storj.io/trace/5ce839c1456bbf8a FAIL (2025-12-08 14:54:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 25:20.182936263 (70h56m20.054019298s) [upload=62126]: Starting upload Dec 8 14:25:20.183037416 (70h56m20.054120461s) [upload=62126]: Done starting upload Dec 8 14:25:20.183136214 (70h56m20.054219259s) [upload=62126]: Uploading segments... Dec 8 14:25:20.183291900 (70h56m20.054374945s) [upload=62126]: Got next segment. Inline: false Dec 8 14:25:20.183343058 (70h56m20.054426103s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 14:25:20.183349700 (70h56m20.054432745s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 14:25:20.560968522 (70h56m20.432051567s) [seg_pos={0 0}, upload=62126]: Begin upload segment... Dec 8 14:25:20.561021322 (70h56m20.432104357s) [seg_pos={0 0}, upload=62126]: Done begin upload segment. Dec 8 14:25:20.561050337 (70h56m20.432133383s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.562561536 (70h56m20.433644581s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563077198 (70h56m20.434160243s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563560729 (70h56m20.434643774s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564043677 (70h56m20.435126732s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564505988 (70h56m20.435589043s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564975802 (70h56m20.436058848s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.566789025 (70h56m20.43787207s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.568723158 (70h56m20.439806204s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.570985847 (70h56m20.442068892s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573462843 (70h56m20.444545889s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573987674 (70h56m20.445070719s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.575740802 (70h56m20.446823837s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.577275113 (70h56m20.448358158s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.578540883 (70h56m20.449623928s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.580403731 (70h56m20.451486776s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:22.334236681 (70h56m22.205319726s) [upload=62126]: Next returned nil segment Dec 8 14:25:22.334251700 (70h56m22.205334735s) [upload=62126]: Waiting for error group managing segments... Dec 8 14:25:22.746429339 (70h56m22.617512384s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.747165611 (70h56m22.618248646s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.753178531 (70h56m22.624261566s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.922057260 (70h56m22.793140305s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.951038517 (70h56m22.822121562s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.651296259 (70h56m23.522379304s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.653766143 (70h56m23.524849178s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.680671276 (70h56m23.551754321s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.700290564 (70h56m23.571373609s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.707126822 (70h56m23.578209868s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.721842694 (70h56m23.592925739s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.730554192 (70h56m23.601637237s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670621120 (70h56m24.541704185s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670647519 (70h56m24.541730564s) [seg_pos={0 0}, upload=62126]: Segment reached optimal threshold of 13 pieces. Dec 8 14:25:24.670967409 (70h56m24.542050454s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection Dec 8 14:25:24.670977318 (70h56m24.542060363s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671144947 (70h56m24.542227982s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: read tcp 188.166.225.170:39576->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 14:25:24.671168231 (70h56m24.542251306s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671490626 (70h56m24.542573661s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 8 14:25:24.671499091 (70h56m24.542582126s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671551481 (70h56m24.542634526s) [seg_pos={0 0}, upload=62126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 14:25:24.671640871 (70h56m24.542723906s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 14:25:24.671647103 (70h56m24.542730138s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 14:25:25.114013719 (70h56m24.985096764s) [upload=62126]: Done uploading segments... Dec 8 14:25:25.114042193 (70h56m24.985125258s) [upload=62126]: Upload finished. err: <nil> Duration: 9.948516489s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.002077308s trace: https://jaeger.dev.storj.io/trace/43b9c7a971d910df FAIL (2025-12-08 14:54:42 UTC) orbiter warning: context deadline exceeded Duration: 5.002095683s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 10.81855267s trace: https://jaeger.dev.storj.io/trace/29daaac43eca6484 FAIL (2025-12-08 14:54:10 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 10.818584649s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XXXXXX.X.X....X.XXX.X..XX....X.......X..X...................] duration: 15.550146036s trace: https://jaeger.dev.storj.io/trace/286e49a04bd90f74 FAIL (2025-12-08 14:54:10 UTC) orbiter warning: context deadline exceeded Duration: 15.550162577s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 19 out of 60 is failed (threshold=3) [.X.XX.XXX.X..XXX.X.XXX.X...X.....X..........X...X...........] duration: 6.848941024s trace: https://jaeger.dev.storj.io/trace/9d8451999396f06 FAIL (2025-12-08 14:54:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.848966422s core uplink us-select-1 us1-placement-20

1
...
11712
11713
11714
11715
11716
...
17282