Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 863998 reports (Page 11715 of 17280)

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXX.XXXXXXXXX] duration: 5.001868703s trace: https://jaeger.dev.storj.io/trace/4488ade0f515cb03 FAIL (2025-12-08 14:42:29 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001890745s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [XX....X..........X.XX.XXX.X..XXX.X.XXX.X...X.....X..........] duration: 19.095189446s trace: https://jaeger.dev.storj.io/trace/31cff760bc138e54 FAIL (2025-12-08 14:42:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.095217619s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 1.00645488s remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4a31039498bdbd4f FAIL (2025-12-08 14:42:10 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 1.006471272s core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXX.XXXXXXXX] duration: 5.000944644s trace: https://jaeger.dev.storj.io/trace/66730a775a9b513a FAIL (2025-12-08 14:42:10 UTC) orbiter warning: context deadline exceeded Duration: 5.000969052s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X..........X.......................] duration: 10.048681925s trace: https://jaeger.dev.storj.io/trace/5ce839c1456bbf8a FAIL (2025-12-08 14:42:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 25:20.182936263 (70h56m20.054019298s) [upload=62126]: Starting upload Dec 8 14:25:20.183037416 (70h56m20.054120461s) [upload=62126]: Done starting upload Dec 8 14:25:20.183136214 (70h56m20.054219259s) [upload=62126]: Uploading segments... Dec 8 14:25:20.183291900 (70h56m20.054374945s) [upload=62126]: Got next segment. Inline: false Dec 8 14:25:20.183343058 (70h56m20.054426103s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 14:25:20.183349700 (70h56m20.054432745s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 14:25:20.560968522 (70h56m20.432051567s) [seg_pos={0 0}, upload=62126]: Begin upload segment... Dec 8 14:25:20.561021322 (70h56m20.432104357s) [seg_pos={0 0}, upload=62126]: Done begin upload segment. Dec 8 14:25:20.561050337 (70h56m20.432133383s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.562561536 (70h56m20.433644581s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563077198 (70h56m20.434160243s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563560729 (70h56m20.434643774s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564043677 (70h56m20.435126732s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564505988 (70h56m20.435589043s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564975802 (70h56m20.436058848s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.566789025 (70h56m20.43787207s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.568723158 (70h56m20.439806204s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.570985847 (70h56m20.442068892s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573462843 (70h56m20.444545889s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573987674 (70h56m20.445070719s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.575740802 (70h56m20.446823837s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.577275113 (70h56m20.448358158s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.578540883 (70h56m20.449623928s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.580403731 (70h56m20.451486776s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:22.334236681 (70h56m22.205319726s) [upload=62126]: Next returned nil segment Dec 8 14:25:22.334251700 (70h56m22.205334735s) [upload=62126]: Waiting for error group managing segments... Dec 8 14:25:22.746429339 (70h56m22.617512384s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.747165611 (70h56m22.618248646s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.753178531 (70h56m22.624261566s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.922057260 (70h56m22.793140305s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.951038517 (70h56m22.822121562s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.651296259 (70h56m23.522379304s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.653766143 (70h56m23.524849178s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.680671276 (70h56m23.551754321s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.700290564 (70h56m23.571373609s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.707126822 (70h56m23.578209868s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.721842694 (70h56m23.592925739s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.730554192 (70h56m23.601637237s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670621120 (70h56m24.541704185s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670647519 (70h56m24.541730564s) [seg_pos={0 0}, upload=62126]: Segment reached optimal threshold of 13 pieces. Dec 8 14:25:24.670967409 (70h56m24.542050454s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection Dec 8 14:25:24.670977318 (70h56m24.542060363s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671144947 (70h56m24.542227982s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: read tcp 188.166.225.170:39576->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 14:25:24.671168231 (70h56m24.542251306s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671490626 (70h56m24.542573661s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 8 14:25:24.671499091 (70h56m24.542582126s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671551481 (70h56m24.542634526s) [seg_pos={0 0}, upload=62126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 14:25:24.671640871 (70h56m24.542723906s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 14:25:24.671647103 (70h56m24.542730138s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 14:25:25.114013719 (70h56m24.985096764s) [upload=62126]: Done uploading segments... Dec 8 14:25:25.114042193 (70h56m24.985125258s) [upload=62126]: Upload finished. err: <nil> Duration: 10.048729154s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.809091334s trace: https://jaeger.dev.storj.io/trace/29e54bf3415d9356 FAIL (2025-12-08 14:41:54 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.809120318s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [.XX....X..........X.XX.XXX.X..XXX.X.XXX.X...X.....X.........] duration: 13.750032811s trace: https://jaeger.dev.storj.io/trace/31cff760bc138e54 FAIL (2025-12-08 14:41:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.75005917s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.X..X.XXXX...XXXXXXX.X.X....X.XXX.X..XX....X.......X..X.....] duration: 12.589405817s trace: https://jaeger.dev.storj.io/trace/286e49a04bd90f74 FAIL (2025-12-08 14:41:43 UTC) orbiter warning: context deadline exceeded Duration: 12.589435003s edge gateway us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 1.087537118s remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4a31039498bdbd4f FAIL (2025-12-08 14:41:32 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 1.08759138s core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.93062793s trace: https://jaeger.dev.storj.io/trace/29e54bf3415d9356 FAIL (2025-12-08 14:41:20 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.930666334s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [X.X..X.XXXX...XXXXXXX.X.X....X.XXX.X..XX....X.......X..X....] duration: 13.36770165s trace: https://jaeger.dev.storj.io/trace/286e49a04bd90f74 FAIL (2025-12-08 14:41:13 UTC) orbiter warning: context deadline exceeded Duration: 13.367734714s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X..........X......................] duration: 9.660818343s trace: https://jaeger.dev.storj.io/trace/5ce839c1456bbf8a FAIL (2025-12-08 14:41:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 25:20.182936263 (70h56m20.054019298s) [upload=62126]: Starting upload Dec 8 14:25:20.183037416 (70h56m20.054120461s) [upload=62126]: Done starting upload Dec 8 14:25:20.183136214 (70h56m20.054219259s) [upload=62126]: Uploading segments... Dec 8 14:25:20.183291900 (70h56m20.054374945s) [upload=62126]: Got next segment. Inline: false Dec 8 14:25:20.183343058 (70h56m20.054426103s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 14:25:20.183349700 (70h56m20.054432745s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 14:25:20.560968522 (70h56m20.432051567s) [seg_pos={0 0}, upload=62126]: Begin upload segment... Dec 8 14:25:20.561021322 (70h56m20.432104357s) [seg_pos={0 0}, upload=62126]: Done begin upload segment. Dec 8 14:25:20.561050337 (70h56m20.432133383s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.562561536 (70h56m20.433644581s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563077198 (70h56m20.434160243s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563560729 (70h56m20.434643774s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564043677 (70h56m20.435126732s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564505988 (70h56m20.435589043s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564975802 (70h56m20.436058848s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.566789025 (70h56m20.43787207s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.568723158 (70h56m20.439806204s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.570985847 (70h56m20.442068892s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573462843 (70h56m20.444545889s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573987674 (70h56m20.445070719s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.575740802 (70h56m20.446823837s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.577275113 (70h56m20.448358158s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.578540883 (70h56m20.449623928s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.580403731 (70h56m20.451486776s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:22.334236681 (70h56m22.205319726s) [upload=62126]: Next returned nil segment Dec 8 14:25:22.334251700 (70h56m22.205334735s) [upload=62126]: Waiting for error group managing segments... Dec 8 14:25:22.746429339 (70h56m22.617512384s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.747165611 (70h56m22.618248646s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.753178531 (70h56m22.624261566s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.922057260 (70h56m22.793140305s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.951038517 (70h56m22.822121562s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.651296259 (70h56m23.522379304s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.653766143 (70h56m23.524849178s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.680671276 (70h56m23.551754321s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.700290564 (70h56m23.571373609s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.707126822 (70h56m23.578209868s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.721842694 (70h56m23.592925739s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.730554192 (70h56m23.601637237s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670621120 (70h56m24.541704185s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670647519 (70h56m24.541730564s) [seg_pos={0 0}, upload=62126]: Segment reached optimal threshold of 13 pieces. Dec 8 14:25:24.670967409 (70h56m24.542050454s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection Dec 8 14:25:24.670977318 (70h56m24.542060363s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671144947 (70h56m24.542227982s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: read tcp 188.166.225.170:39576->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 14:25:24.671168231 (70h56m24.542251306s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671490626 (70h56m24.542573661s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 8 14:25:24.671499091 (70h56m24.542582126s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671551481 (70h56m24.542634526s) [seg_pos={0 0}, upload=62126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 14:25:24.671640871 (70h56m24.542723906s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 14:25:24.671647103 (70h56m24.542730138s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 14:25:25.114013719 (70h56m24.985096764s) [upload=62126]: Done uploading segments... Dec 8 14:25:25.114042193 (70h56m24.985125258s) [upload=62126]: Upload finished. err: <nil> Duration: 9.660844553s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXX.XXXXXXX] duration: 5.00155019s trace: https://jaeger.dev.storj.io/trace/27c4e30156c00e65 FAIL (2025-12-08 14:41:09 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001567492s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 1.036725271s remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4a31039498bdbd4f FAIL (2025-12-08 14:40:55 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 1.036751231s core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [..XX....X..........X.XX.XXX.X..XXX.X.XXX.X...X.....X........] duration: 13.258872113s trace: https://jaeger.dev.storj.io/trace/31cff760bc138e54 FAIL (2025-12-08 14:40:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.258900516s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXX.XXXXXX] duration: 5.001273972s trace: https://jaeger.dev.storj.io/trace/1d0bb64fd4534892 FAIL (2025-12-08 14:40:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001298439s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X..........X.....................] duration: 9.122254599s trace: https://jaeger.dev.storj.io/trace/5ce839c1456bbf8a FAIL (2025-12-08 14:40:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 25:20.182936263 (70h56m20.054019298s) [upload=62126]: Starting upload Dec 8 14:25:20.183037416 (70h56m20.054120461s) [upload=62126]: Done starting upload Dec 8 14:25:20.183136214 (70h56m20.054219259s) [upload=62126]: Uploading segments... Dec 8 14:25:20.183291900 (70h56m20.054374945s) [upload=62126]: Got next segment. Inline: false Dec 8 14:25:20.183343058 (70h56m20.054426103s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 14:25:20.183349700 (70h56m20.054432745s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 14:25:20.560968522 (70h56m20.432051567s) [seg_pos={0 0}, upload=62126]: Begin upload segment... Dec 8 14:25:20.561021322 (70h56m20.432104357s) [seg_pos={0 0}, upload=62126]: Done begin upload segment. Dec 8 14:25:20.561050337 (70h56m20.432133383s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.562561536 (70h56m20.433644581s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563077198 (70h56m20.434160243s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563560729 (70h56m20.434643774s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564043677 (70h56m20.435126732s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564505988 (70h56m20.435589043s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564975802 (70h56m20.436058848s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.566789025 (70h56m20.43787207s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.568723158 (70h56m20.439806204s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.570985847 (70h56m20.442068892s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573462843 (70h56m20.444545889s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573987674 (70h56m20.445070719s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.575740802 (70h56m20.446823837s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.577275113 (70h56m20.448358158s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.578540883 (70h56m20.449623928s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.580403731 (70h56m20.451486776s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:22.334236681 (70h56m22.205319726s) [upload=62126]: Next returned nil segment Dec 8 14:25:22.334251700 (70h56m22.205334735s) [upload=62126]: Waiting for error group managing segments... Dec 8 14:25:22.746429339 (70h56m22.617512384s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.747165611 (70h56m22.618248646s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.753178531 (70h56m22.624261566s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.922057260 (70h56m22.793140305s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.951038517 (70h56m22.822121562s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.651296259 (70h56m23.522379304s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.653766143 (70h56m23.524849178s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.680671276 (70h56m23.551754321s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.700290564 (70h56m23.571373609s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.707126822 (70h56m23.578209868s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.721842694 (70h56m23.592925739s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.730554192 (70h56m23.601637237s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670621120 (70h56m24.541704185s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670647519 (70h56m24.541730564s) [seg_pos={0 0}, upload=62126]: Segment reached optimal threshold of 13 pieces. Dec 8 14:25:24.670967409 (70h56m24.542050454s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection Dec 8 14:25:24.670977318 (70h56m24.542060363s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671144947 (70h56m24.542227982s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: read tcp 188.166.225.170:39576->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 14:25:24.671168231 (70h56m24.542251306s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671490626 (70h56m24.542573661s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 8 14:25:24.671499091 (70h56m24.542582126s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671551481 (70h56m24.542634526s) [seg_pos={0 0}, upload=62126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 14:25:24.671640871 (70h56m24.542723906s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 14:25:24.671647103 (70h56m24.542730138s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 14:25:25.114013719 (70h56m24.985096764s) [upload=62126]: Done uploading segments... Dec 8 14:25:25.114042193 (70h56m24.985125258s) [upload=62126]: Upload finished. err: <nil> Duration: 9.122298974s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 1.054770314s remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4a31039498bdbd4f FAIL (2025-12-08 14:40:32 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 1.054795592s core satellite us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [.X.X..X.XXXX...XXXXXXX.X.X....X.XXX.X..XX....X.......X..X...] duration: 21.568085s trace: https://jaeger.dev.storj.io/trace/286e49a04bd90f74 FAIL (2025-12-08 14:40:23 UTC) orbiter warning: context deadline exceeded Duration: 21.568101131s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 2.870834671s trace: https://jaeger.dev.storj.io/trace/29e54bf3415d9356 FAIL (2025-12-08 14:40:18 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.870866353s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [..X.X..X.XXXX...XXXXXXX.X.X....X.XXX.X..XX....X.......X..X..] duration: 22.50383863s trace: https://jaeger.dev.storj.io/trace/286e49a04bd90f74 FAIL (2025-12-08 14:40:01 UTC) orbiter warning: context deadline exceeded Duration: 22.503860601s edge gateway us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 1.078425547s remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4a31039498bdbd4f FAIL (2025-12-08 14:39:42 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 1.078460693s core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X..........X....................] duration: 9.046749296s trace: https://jaeger.dev.storj.io/trace/5ce839c1456bbf8a FAIL (2025-12-08 14:39:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 25:20.182936263 (70h56m20.054019298s) [upload=62126]: Starting upload Dec 8 14:25:20.183037416 (70h56m20.054120461s) [upload=62126]: Done starting upload Dec 8 14:25:20.183136214 (70h56m20.054219259s) [upload=62126]: Uploading segments... Dec 8 14:25:20.183291900 (70h56m20.054374945s) [upload=62126]: Got next segment. Inline: false Dec 8 14:25:20.183343058 (70h56m20.054426103s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 14:25:20.183349700 (70h56m20.054432745s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 14:25:20.560968522 (70h56m20.432051567s) [seg_pos={0 0}, upload=62126]: Begin upload segment... Dec 8 14:25:20.561021322 (70h56m20.432104357s) [seg_pos={0 0}, upload=62126]: Done begin upload segment. Dec 8 14:25:20.561050337 (70h56m20.432133383s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.562561536 (70h56m20.433644581s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563077198 (70h56m20.434160243s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563560729 (70h56m20.434643774s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564043677 (70h56m20.435126732s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564505988 (70h56m20.435589043s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564975802 (70h56m20.436058848s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.566789025 (70h56m20.43787207s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.568723158 (70h56m20.439806204s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.570985847 (70h56m20.442068892s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573462843 (70h56m20.444545889s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573987674 (70h56m20.445070719s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.575740802 (70h56m20.446823837s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.577275113 (70h56m20.448358158s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.578540883 (70h56m20.449623928s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.580403731 (70h56m20.451486776s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:22.334236681 (70h56m22.205319726s) [upload=62126]: Next returned nil segment Dec 8 14:25:22.334251700 (70h56m22.205334735s) [upload=62126]: Waiting for error group managing segments... Dec 8 14:25:22.746429339 (70h56m22.617512384s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.747165611 (70h56m22.618248646s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.753178531 (70h56m22.624261566s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.922057260 (70h56m22.793140305s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.951038517 (70h56m22.822121562s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.651296259 (70h56m23.522379304s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.653766143 (70h56m23.524849178s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.680671276 (70h56m23.551754321s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.700290564 (70h56m23.571373609s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.707126822 (70h56m23.578209868s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.721842694 (70h56m23.592925739s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.730554192 (70h56m23.601637237s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670621120 (70h56m24.541704185s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670647519 (70h56m24.541730564s) [seg_pos={0 0}, upload=62126]: Segment reached optimal threshold of 13 pieces. Dec 8 14:25:24.670967409 (70h56m24.542050454s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection Dec 8 14:25:24.670977318 (70h56m24.542060363s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671144947 (70h56m24.542227982s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: read tcp 188.166.225.170:39576->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 14:25:24.671168231 (70h56m24.542251306s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671490626 (70h56m24.542573661s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 8 14:25:24.671499091 (70h56m24.542582126s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671551481 (70h56m24.542634526s) [seg_pos={0 0}, upload=62126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 14:25:24.671640871 (70h56m24.542723906s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 14:25:24.671647103 (70h56m24.542730138s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 14:25:25.114013719 (70h56m24.985096764s) [upload=62126]: Done uploading segments... Dec 8 14:25:25.114042193 (70h56m24.985125258s) [upload=62126]: Upload finished. err: <nil> Duration: 9.046777931s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 21 out of 60 is failed (threshold=3) [X..XX....X..........X.XX.XXX.X..XXX.X.XXX.X...X.....X.......] duration: 15.133214615s trace: https://jaeger.dev.storj.io/trace/31cff760bc138e54 FAIL (2025-12-08 14:39:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.133239161s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXX.XXXXX] duration: 5.001679625s trace: https://jaeger.dev.storj.io/trace/3af1f03278e1f7c5 FAIL (2025-12-08 14:39:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001710294s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 2.954489749s trace: https://jaeger.dev.storj.io/trace/29e54bf3415d9356 FAIL (2025-12-08 14:39:20 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.954513585s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X..........X...................] duration: 9.251376278s trace: https://jaeger.dev.storj.io/trace/5ce839c1456bbf8a FAIL (2025-12-08 14:39:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 25:20.182936263 (70h56m20.054019298s) [upload=62126]: Starting upload Dec 8 14:25:20.183037416 (70h56m20.054120461s) [upload=62126]: Done starting upload Dec 8 14:25:20.183136214 (70h56m20.054219259s) [upload=62126]: Uploading segments... Dec 8 14:25:20.183291900 (70h56m20.054374945s) [upload=62126]: Got next segment. Inline: false Dec 8 14:25:20.183343058 (70h56m20.054426103s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 14:25:20.183349700 (70h56m20.054432745s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 14:25:20.560968522 (70h56m20.432051567s) [seg_pos={0 0}, upload=62126]: Begin upload segment... Dec 8 14:25:20.561021322 (70h56m20.432104357s) [seg_pos={0 0}, upload=62126]: Done begin upload segment. Dec 8 14:25:20.561050337 (70h56m20.432133383s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.562561536 (70h56m20.433644581s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563077198 (70h56m20.434160243s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563560729 (70h56m20.434643774s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564043677 (70h56m20.435126732s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564505988 (70h56m20.435589043s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564975802 (70h56m20.436058848s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.566789025 (70h56m20.43787207s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.568723158 (70h56m20.439806204s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.570985847 (70h56m20.442068892s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573462843 (70h56m20.444545889s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573987674 (70h56m20.445070719s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.575740802 (70h56m20.446823837s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.577275113 (70h56m20.448358158s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.578540883 (70h56m20.449623928s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.580403731 (70h56m20.451486776s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:22.334236681 (70h56m22.205319726s) [upload=62126]: Next returned nil segment Dec 8 14:25:22.334251700 (70h56m22.205334735s) [upload=62126]: Waiting for error group managing segments... Dec 8 14:25:22.746429339 (70h56m22.617512384s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.747165611 (70h56m22.618248646s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.753178531 (70h56m22.624261566s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.922057260 (70h56m22.793140305s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.951038517 (70h56m22.822121562s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.651296259 (70h56m23.522379304s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.653766143 (70h56m23.524849178s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.680671276 (70h56m23.551754321s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.700290564 (70h56m23.571373609s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.707126822 (70h56m23.578209868s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.721842694 (70h56m23.592925739s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.730554192 (70h56m23.601637237s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670621120 (70h56m24.541704185s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670647519 (70h56m24.541730564s) [seg_pos={0 0}, upload=62126]: Segment reached optimal threshold of 13 pieces. Dec 8 14:25:24.670967409 (70h56m24.542050454s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection Dec 8 14:25:24.670977318 (70h56m24.542060363s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671144947 (70h56m24.542227982s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: read tcp 188.166.225.170:39576->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 14:25:24.671168231 (70h56m24.542251306s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671490626 (70h56m24.542573661s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 8 14:25:24.671499091 (70h56m24.542582126s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671551481 (70h56m24.542634526s) [seg_pos={0 0}, upload=62126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 14:25:24.671640871 (70h56m24.542723906s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 14:25:24.671647103 (70h56m24.542730138s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 14:25:25.114013719 (70h56m24.985096764s) [upload=62126]: Done uploading segments... Dec 8 14:25:25.114042193 (70h56m24.985125258s) [upload=62126]: Upload finished. err: <nil> Duration: 9.251416855s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 1.002904139s remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4a31039498bdbd4f FAIL (2025-12-08 14:39:09 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 1.002959693s core satellite us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [...X.X..X.XXXX...XXXXXXX.X.X....X.XXX.X..XX....X.......X..X.] duration: 30.001671513s trace: https://jaeger.dev.storj.io/trace/286e49a04bd90f74 FAIL (2025-12-08 14:39:01 UTC) orbiter warning: context deadline exceeded Duration: 30.001692423s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXX.XXXX] duration: 5.001680053s trace: https://jaeger.dev.storj.io/trace/669c60b145d3b058 FAIL (2025-12-08 14:38:59 UTC) orbiter warning: context deadline exceeded Duration: 5.001696654s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 1.02124861s remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4a31039498bdbd4f FAIL (2025-12-08 14:38:43 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 1.02126978s core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 21 out of 60 is failed (threshold=3) [.X..XX....X..........X.XX.XXX.X..XXX.X.XXX.X...X.....X......] duration: 6.630214084s trace: https://jaeger.dev.storj.io/trace/31cff760bc138e54 FAIL (2025-12-08 14:38:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.630847701s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 3.016578025s trace: https://jaeger.dev.storj.io/trace/29e54bf3415d9356 FAIL (2025-12-08 14:38:30 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 3.016608233s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [....X.X..X.XXXX...XXXXXXX.X.X....X.XXX.X..XX....X.......X..X] duration: 30.00168718s trace: https://jaeger.dev.storj.io/trace/286e49a04bd90f74 FAIL (2025-12-08 14:38:30 UTC) orbiter warning: context deadline exceeded Duration: 30.0017132s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X..........X..................] duration: 9.546358953s trace: https://jaeger.dev.storj.io/trace/5ce839c1456bbf8a FAIL (2025-12-08 14:38:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 25:20.182936263 (70h56m20.054019298s) [upload=62126]: Starting upload Dec 8 14:25:20.183037416 (70h56m20.054120461s) [upload=62126]: Done starting upload Dec 8 14:25:20.183136214 (70h56m20.054219259s) [upload=62126]: Uploading segments... Dec 8 14:25:20.183291900 (70h56m20.054374945s) [upload=62126]: Got next segment. Inline: false Dec 8 14:25:20.183343058 (70h56m20.054426103s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 14:25:20.183349700 (70h56m20.054432745s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 14:25:20.560968522 (70h56m20.432051567s) [seg_pos={0 0}, upload=62126]: Begin upload segment... Dec 8 14:25:20.561021322 (70h56m20.432104357s) [seg_pos={0 0}, upload=62126]: Done begin upload segment. Dec 8 14:25:20.561050337 (70h56m20.432133383s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.562561536 (70h56m20.433644581s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563077198 (70h56m20.434160243s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563560729 (70h56m20.434643774s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564043677 (70h56m20.435126732s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564505988 (70h56m20.435589043s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564975802 (70h56m20.436058848s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.566789025 (70h56m20.43787207s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.568723158 (70h56m20.439806204s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.570985847 (70h56m20.442068892s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573462843 (70h56m20.444545889s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573987674 (70h56m20.445070719s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.575740802 (70h56m20.446823837s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.577275113 (70h56m20.448358158s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.578540883 (70h56m20.449623928s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.580403731 (70h56m20.451486776s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:22.334236681 (70h56m22.205319726s) [upload=62126]: Next returned nil segment Dec 8 14:25:22.334251700 (70h56m22.205334735s) [upload=62126]: Waiting for error group managing segments... Dec 8 14:25:22.746429339 (70h56m22.617512384s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.747165611 (70h56m22.618248646s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.753178531 (70h56m22.624261566s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.922057260 (70h56m22.793140305s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.951038517 (70h56m22.822121562s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.651296259 (70h56m23.522379304s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.653766143 (70h56m23.524849178s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.680671276 (70h56m23.551754321s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.700290564 (70h56m23.571373609s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.707126822 (70h56m23.578209868s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.721842694 (70h56m23.592925739s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.730554192 (70h56m23.601637237s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670621120 (70h56m24.541704185s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670647519 (70h56m24.541730564s) [seg_pos={0 0}, upload=62126]: Segment reached optimal threshold of 13 pieces. Dec 8 14:25:24.670967409 (70h56m24.542050454s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection Dec 8 14:25:24.670977318 (70h56m24.542060363s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671144947 (70h56m24.542227982s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: read tcp 188.166.225.170:39576->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 14:25:24.671168231 (70h56m24.542251306s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671490626 (70h56m24.542573661s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 8 14:25:24.671499091 (70h56m24.542582126s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671551481 (70h56m24.542634526s) [seg_pos={0 0}, upload=62126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 14:25:24.671640871 (70h56m24.542723906s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 14:25:24.671647103 (70h56m24.542730138s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 14:25:25.114013719 (70h56m24.985096764s) [upload=62126]: Done uploading segments... Dec 8 14:25:25.114042193 (70h56m24.985125258s) [upload=62126]: Upload finished. err: <nil> Duration: 9.546386838s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 4.754012082s trace: https://jaeger.dev.storj.io/trace/29e54bf3415d9356 FAIL (2025-12-08 14:38:09 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.754042681s core uplink us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 1.053000694s remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4a31039498bdbd4f FAIL (2025-12-08 14:38:03 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 1.053037604s core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 7.800611155s trace: https://jaeger.dev.storj.io/trace/406d13f34dbd7740 FAIL (2025-12-08 14:37:58 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 7.800644409s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXX.XXX] duration: 5.001645317s trace: https://jaeger.dev.storj.io/trace/1ad111aa5c4e1e33 FAIL (2025-12-08 14:37:58 UTC) orbiter warning: context deadline exceeded Duration: 5.001668411s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X..........X.................] duration: 19.866623792s trace: https://jaeger.dev.storj.io/trace/5ce839c1456bbf8a FAIL (2025-12-08 14:37:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 25:20.182936263 (70h56m20.054019298s) [upload=62126]: Starting upload Dec 8 14:25:20.183037416 (70h56m20.054120461s) [upload=62126]: Done starting upload Dec 8 14:25:20.183136214 (70h56m20.054219259s) [upload=62126]: Uploading segments... Dec 8 14:25:20.183291900 (70h56m20.054374945s) [upload=62126]: Got next segment. Inline: false Dec 8 14:25:20.183343058 (70h56m20.054426103s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 14:25:20.183349700 (70h56m20.054432745s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 14:25:20.560968522 (70h56m20.432051567s) [seg_pos={0 0}, upload=62126]: Begin upload segment... Dec 8 14:25:20.561021322 (70h56m20.432104357s) [seg_pos={0 0}, upload=62126]: Done begin upload segment. Dec 8 14:25:20.561050337 (70h56m20.432133383s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.562561536 (70h56m20.433644581s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563077198 (70h56m20.434160243s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563560729 (70h56m20.434643774s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564043677 (70h56m20.435126732s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564505988 (70h56m20.435589043s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564975802 (70h56m20.436058848s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.566789025 (70h56m20.43787207s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.568723158 (70h56m20.439806204s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.570985847 (70h56m20.442068892s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573462843 (70h56m20.444545889s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573987674 (70h56m20.445070719s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.575740802 (70h56m20.446823837s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.577275113 (70h56m20.448358158s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.578540883 (70h56m20.449623928s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.580403731 (70h56m20.451486776s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:22.334236681 (70h56m22.205319726s) [upload=62126]: Next returned nil segment Dec 8 14:25:22.334251700 (70h56m22.205334735s) [upload=62126]: Waiting for error group managing segments... Dec 8 14:25:22.746429339 (70h56m22.617512384s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.747165611 (70h56m22.618248646s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.753178531 (70h56m22.624261566s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.922057260 (70h56m22.793140305s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.951038517 (70h56m22.822121562s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.651296259 (70h56m23.522379304s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.653766143 (70h56m23.524849178s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.680671276 (70h56m23.551754321s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.700290564 (70h56m23.571373609s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.707126822 (70h56m23.578209868s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.721842694 (70h56m23.592925739s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.730554192 (70h56m23.601637237s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670621120 (70h56m24.541704185s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670647519 (70h56m24.541730564s) [seg_pos={0 0}, upload=62126]: Segment reached optimal threshold of 13 pieces. Dec 8 14:25:24.670967409 (70h56m24.542050454s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection Dec 8 14:25:24.670977318 (70h56m24.542060363s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671144947 (70h56m24.542227982s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: read tcp 188.166.225.170:39576->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 14:25:24.671168231 (70h56m24.542251306s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671490626 (70h56m24.542573661s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 8 14:25:24.671499091 (70h56m24.542582126s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671551481 (70h56m24.542634526s) [seg_pos={0 0}, upload=62126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 14:25:24.671640871 (70h56m24.542723906s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 14:25:24.671647103 (70h56m24.542730138s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 14:25:25.114013719 (70h56m24.985096764s) [upload=62126]: Done uploading segments... Dec 8 14:25:25.114042193 (70h56m24.985125258s) [upload=62126]: Upload finished. err: <nil> Duration: 19.866652506s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 22 out of 60 is failed (threshold=3) [X.X..XX....X..........X.XX.XXX.X..XXX.X.XXX.X...X.....X.....] duration: 5.887747987s trace: https://jaeger.dev.storj.io/trace/31cff760bc138e54 FAIL (2025-12-08 14:37:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.887799374s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.....X.X..X.XXXX...XXXXXXX.X.X....X.XXX.X..XX....X.......X..] duration: 17.049547748s trace: https://jaeger.dev.storj.io/trace/723baea6c03c60d5 FAIL (2025-12-08 14:37:44 UTC) orbiter warning: context deadline exceeded Duration: 17.049566594s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 9.632012732s trace: https://jaeger.dev.storj.io/trace/406d13f34dbd7740 FAIL (2025-12-08 14:37:34 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 9.632040004s core uplink us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 988.741671ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4a31039498bdbd4f FAIL (2025-12-08 14:37:27 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 988.772099ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X..........X................] duration: 13.31771234s trace: https://jaeger.dev.storj.io/trace/5ce839c1456bbf8a FAIL (2025-12-08 14:37:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 25:20.182936263 (70h56m20.054019298s) [upload=62126]: Starting upload Dec 8 14:25:20.183037416 (70h56m20.054120461s) [upload=62126]: Done starting upload Dec 8 14:25:20.183136214 (70h56m20.054219259s) [upload=62126]: Uploading segments... Dec 8 14:25:20.183291900 (70h56m20.054374945s) [upload=62126]: Got next segment. Inline: false Dec 8 14:25:20.183343058 (70h56m20.054426103s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 14:25:20.183349700 (70h56m20.054432745s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 14:25:20.560968522 (70h56m20.432051567s) [seg_pos={0 0}, upload=62126]: Begin upload segment... Dec 8 14:25:20.561021322 (70h56m20.432104357s) [seg_pos={0 0}, upload=62126]: Done begin upload segment. Dec 8 14:25:20.561050337 (70h56m20.432133383s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.562561536 (70h56m20.433644581s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563077198 (70h56m20.434160243s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.563560729 (70h56m20.434643774s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564043677 (70h56m20.435126732s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564505988 (70h56m20.435589043s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.564975802 (70h56m20.436058848s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.566789025 (70h56m20.43787207s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.568723158 (70h56m20.439806204s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.570985847 (70h56m20.442068892s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573462843 (70h56m20.444545889s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.573987674 (70h56m20.445070719s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.575740802 (70h56m20.446823837s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.577275113 (70h56m20.448358158s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.578540883 (70h56m20.449623928s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:20.580403731 (70h56m20.451486776s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Uploading piece... Dec 8 14:25:22.334236681 (70h56m22.205319726s) [upload=62126]: Next returned nil segment Dec 8 14:25:22.334251700 (70h56m22.205334735s) [upload=62126]: Waiting for error group managing segments... Dec 8 14:25:22.746429339 (70h56m22.617512384s) [noise=true , address=77.90.31.11:4555 , piece_id=MAM7UKIK4FXUSU2NXF7ISKYBT7CJVH3GNLPUCDURX2QHEVTU2X6A, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.747165611 (70h56m22.618248646s) [noise=true , address=77.90.31.17:4772 , piece_id=OWFG5DPNGQ423VEESPNX6M3YWD65IPFRYF2ZDHIYRBHEYQKANDLQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.753178531 (70h56m22.624261566s) [noise=true , address=77.90.31.11:4554 , piece_id=XQEK7QDM5AHZ62KTNY3PUOH5DK3SM4ZRHYVDQKTGML3EAWVV4IZA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.922057260 (70h56m22.793140305s) [noise=true , address=45.91.116.8:4045 , piece_id=WWJ6WLJ2MEPXAS23J4YVW4M7HEZ46JUKLCGCPXYKKRDMXNBBIHTA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:22.951038517 (70h56m22.822121562s) [noise=true , address=45.91.116.8:4048 , piece_id=LH67K3K5CAJHXBQG4CQABBYLXSBJ2AID7JKHOM5MC57DU7C4EIRA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.651296259 (70h56m23.522379304s) [noise=true , address=145.79.227.34:4906 , piece_id=D7WVXKXAS7NNPHOXOUS7MOZWBQ2BV5R4E7EFOJ24GIBC7OUC3WHA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.653766143 (70h56m23.524849178s) [noise=true , address=145.79.227.38:4979 , piece_id=FHG2EXMZKWTLV3CB24LBWFPQETM35A3UKG2DHGQHOM2QLZ7DWAGQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.680671276 (70h56m23.551754321s) [noise=true , address=145.79.227.43:4158 , piece_id=6P6V3NYFJFS6PZ45QJLPBLUL3DRMP7EF6K3TIRUFALCEPVCC3BIQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.700290564 (70h56m23.571373609s) [noise=true , address=145.79.227.24:4582 , piece_id=OXXHMNMMXX523ORM2F4BVRR6ALDTUYR4WUUALUJHYDOOWWDSPJTQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.707126822 (70h56m23.578209868s) [noise=true , address=145.79.227.22:4510 , piece_id=SAO6ZGRIDQEL3GZDENC6ZXHCLIHRWEU7UTU6DFAL7WBPEDLWT7GA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.721842694 (70h56m23.592925739s) [noise=true , address=145.79.227.39:4014 , piece_id=GW2CAOTIFZ65MSTRQ3R42TGWBUMS3HCB5VLF34D2E2BJOX76SMYQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:23.730554192 (70h56m23.601637237s) [noise=true , address=145.79.227.32:4836 , piece_id=F53BEOIRAO4X6VVXAKPK57UXAU3LLAOI2WYMDOOV2T3342VKW7RQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670621120 (70h56m24.541704185s) [noise=true , address=145.79.227.40:4051 , piece_id=R5SK24KL2MVQ5BIANSX4L6J3Z6HA4XJHP4IDXEH4XNTYWFVMOGNQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: <nil> Dec 8 14:25:24.670647519 (70h56m24.541730564s) [seg_pos={0 0}, upload=62126]: Segment reached optimal threshold of 13 pieces. Dec 8 14:25:24.670967409 (70h56m24.542050454s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55430->77.90.31.16:4734: use of closed network connection Dec 8 14:25:24.670977318 (70h56m24.542060363s) [noise=true , address=77.90.31.16:4734 , piece_id=ACPSX5F6ALOQAFBM4GOAKXAPGVXDROEXHWH3GLLBB4RXOIQHTMAA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671144947 (70h56m24.542227982s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: read tcp 188.166.225.170:39576->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 14:25:24.671168231 (70h56m24.542251306s) [noise=true , address=45.91.116.12:4225 , piece_id=DBFTR6VSOVT7AXZGZIERSC4AON3LDHZ6HV2MGR5X52MHZ36KWHJQ, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671490626 (70h56m24.542573661s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 8 14:25:24.671499091 (70h56m24.542582126s) [noise=true , address=145.79.227.30:4761 , piece_id=HFZUKWJWQRTORCPO33PI4OTRSO2YN6QQ2ZDBGBLMKU3UDJ3UBQIA, seg_pos={0 0}, upload=62126]: Long tail cancellation due to optimal threshold Dec 8 14:25:24.671551481 (70h56m24.542634526s) [seg_pos={0 0}, upload=62126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 14:25:24.671640871 (70h56m24.542723906s) [upload=62126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 14:25:24.671647103 (70h56m24.542730138s) [upload=62126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 14:25:25.114013719 (70h56m24.985096764s) [upload=62126]: Done uploading segments... Dec 8 14:25:25.114042193 (70h56m24.985125258s) [upload=62126]: Upload finished. err: <nil> Duration: 13.317742726s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [......X.X..X.XXXX...XXXXXXX.X.X....X.XXX.X..XX....X.......X.] duration: 25.955886565s trace: https://jaeger.dev.storj.io/trace/723baea6c03c60d5 FAIL (2025-12-08 14:37:26 UTC) orbiter warning: context deadline exceeded Duration: 25.955901363s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXX.XX] duration: 5.000960487s trace: https://jaeger.dev.storj.io/trace/5b2eeebe78789302 FAIL (2025-12-08 14:37:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000980946s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.......X.X..X.XXXX...XXXXXXX.X.X....X.XXX.X..XX....X.......X] duration: 30.000891009s trace: https://jaeger.dev.storj.io/trace/723baea6c03c60d5 FAIL (2025-12-08 14:37:00 UTC) orbiter warning: context deadline exceeded Duration: 30.000918913s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 2.905732286s trace: https://jaeger.dev.storj.io/trace/29e54bf3415d9356 FAIL (2025-12-08 14:36:54 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.905764708s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 22 out of 60 is failed (threshold=3) [.X.X..XX....X..........X.XX.XXX.X..XXX.X.XXX.X...X.....X....] duration: 17.112995134s trace: https://jaeger.dev.storj.io/trace/31cff760bc138e54 FAIL (2025-12-08 14:36:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.113020272s core uplink us-select-1 us1-placement-20

1
...
11713
11714
11715
11716
11717
...
17280