Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.........X...X..............................................] duration:
5.781917766s trace:
https://jaeger.dev.storj.io/trace/6ea66b1ed6f4c6e5
FAIL (2025-12-30 17:40:26 UTC)
orbiter warning: context deadline exceeded
Duration: 5.781941031s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.............X...X..X............................XX.........] duration:
6.790015531s trace:
https://jaeger.dev.storj.io/trace/4173ea2fcd946b89
FAIL (2025-12-30 17:40:22 UTC)
orbiter warning: context deadline exceeded
Duration: 6.790062371s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
19.058930608s trace:
https://jaeger.dev.storj.io/trace/5ea8d7d4f61e13a2
FAIL (2025-12-30 17:40:14 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK, Piece ID: HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ) context deadline exceeded; piecestore: (Node ID: 12eLKUg2vTnZMrLaraSPEoGXe3jDVBJYyURdcn4f2gCN5pvvCQ8, Piece ID: KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ) context deadline exceeded; piecestore: (Node ID: 12W4n6uf6azQk1QgeR8Xw7UkpnZm19zdyFKwojCsqZt2GFHrpce, Piece ID: DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ) context deadline exceeded
======LOGS======
17.114181822 (601h42m16.985264867s) [noise=true , address=145.79.227.44:4208 , piece_id=3OXDCHZSOIFS5VQCI4SXSAQFICDCYHRE5NGFBERR33KOTGSZWLDQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.114641387 (601h42m16.985724432s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.115652532 (601h42m16.986735578s) [noise=true , address=77.90.31.17:4790 , piece_id=5RSRESKGDA2YF4NBRF4ZFABBKWJK4NO6VLEXW2Z6EY3OPXH2WPNA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.115937134 (601h42m16.987020179s) [noise=true , address=64.49.13.155:4923 , piece_id=MUJD7XLM2YRLRCHYVSDPBE6VBYLHCOA7HJQP22ZSET4N574XQMJA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.116814986 (601h42m16.987898031s) [noise=true , address=45.91.116.11:4139 , piece_id=72DPEWZ7ZPZZ53UGYV2VCUKLCK7HCM22CDBRHWVLJGDGO5B5JADQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.117501182 (601h42m16.988584228s) [noise=true , address=77.90.31.14:4665 , piece_id=DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.118263334 (601h42m16.989346379s) [noise=true , address=145.79.227.40:4084 , piece_id=ZADZDRKBIILLAI4IT335CYETOSFOFUUPQIH6HZJCTHKRTZSPDJUQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.118555850 (601h42m16.989638886s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.118765930 (601h42m16.989849036s) [noise=true , address=77.90.31.21:4905 , piece_id=HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119029533 (601h42m16.990112568s) [noise=true , address=77.90.31.10:4533 , piece_id=QFBBYNQHQL2G7PEREK43FZBRUMKMVNBC2XKTE2NWTVTKN2452LPA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119417672 (601h42m16.990500707s) [noise=true , address=45.91.116.12:4258 , piece_id=KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119441588 (601h42m16.990524643s) [noise=true , address=145.79.227.40:4053 , piece_id=JCSIUMWKEAQNQXTNSJTN6VSB3V2SIDRQK67QMK46B4GOUWEMQE2A, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119763470 (601h42m16.990846505s) [noise=true , address=77.90.31.18:4821 , piece_id=MXZFKXM3JV2HQURDLIU7BKWLAKDZ27O4ER3SGC7QBZV5I3J2EHKA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.120028685 (601h42m16.99111172s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.120086986 (601h42m16.991170031s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.120586507 (601h42m16.991669562s) [noise=true , address=45.91.116.13:4168 , piece_id=AKUE5H3A2UECQOZY3GV3MWZKIHSNKCDOG32DKCNHIRTJROJ74M3A, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.121258768 (601h42m16.992341813s) [noise=true , address=145.79.227.9:4379 , piece_id=QTFXFSGGKYBQSPWJYBBJMQUL3D5EEO3IX4C53ECD7FXMUOGUBC7Q, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:18.568739165 (601h42m18.43982221s) [upload=527370]: Next returned nil segment
Dec 30 17:11:18.568751328 (601h42m18.439834363s) [upload=527370]: Waiting for error group managing segments...
Dec 30 17:11:18.978424178 (601h42m18.849507213s) [noise=true , address=64.49.13.155:4923 , piece_id=MUJD7XLM2YRLRCHYVSDPBE6VBYLHCOA7HJQP22ZSET4N574XQMJA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:18.986258125 (601h42m18.85734116s) [noise=true , address=145.79.227.9:4379 , piece_id=QTFXFSGGKYBQSPWJYBBJMQUL3D5EEO3IX4C53ECD7FXMUOGUBC7Q, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.274519421 (601h42m19.145602456s) [noise=true , address=77.90.31.14:4665 , piece_id=DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.315775438 (601h42m19.186858473s) [noise=true , address=45.91.116.13:4168 , piece_id=AKUE5H3A2UECQOZY3GV3MWZKIHSNKCDOG32DKCNHIRTJROJ74M3A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.335765371 (601h42m19.206848406s) [noise=true , address=77.90.31.21:4905 , piece_id=HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.354812129 (601h42m19.225895164s) [noise=true , address=77.90.31.10:4533 , piece_id=QFBBYNQHQL2G7PEREK43FZBRUMKMVNBC2XKTE2NWTVTKN2452LPA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.392024917 (601h42m19.263107952s) [noise=true , address=45.91.116.11:4139 , piece_id=72DPEWZ7ZPZZ53UGYV2VCUKLCK7HCM22CDBRHWVLJGDGO5B5JADQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.396084327 (601h42m19.267167372s) [noise=true , address=77.90.31.18:4821 , piece_id=MXZFKXM3JV2HQURDLIU7BKWLAKDZ27O4ER3SGC7QBZV5I3J2EHKA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.862099099 (601h42m19.733182134s) [noise=true , address=77.90.31.17:4790 , piece_id=5RSRESKGDA2YF4NBRF4ZFABBKWJK4NO6VLEXW2Z6EY3OPXH2WPNA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.905412913 (601h42m19.776495948s) [noise=true , address=145.79.227.44:4208 , piece_id=3OXDCHZSOIFS5VQCI4SXSAQFICDCYHRE5NGFBERR33KOTGSZWLDQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.910309688 (601h42m19.781392743s) [noise=true , address=145.79.227.30:4794 , piece_id=EJH6Q3T2BMPVLG6F7CW7N4GYOA3EZFPADL6VPT7WKKG65B2WZVXQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.934233361 (601h42m19.805316397s) [noise=true , address=145.79.227.40:4084 , piece_id=ZADZDRKBIILLAI4IT335CYETOSFOFUUPQIH6HZJCTHKRTZSPDJUQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:20.074011527 (601h42m19.945094572s) [noise=true , address=45.91.116.12:4258 , piece_id=KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:20.106010337 (601h42m19.977093372s) [noise=true , address=145.79.227.40:4053 , piece_id=JCSIUMWKEAQNQXTNSJTN6VSB3V2SIDRQK67QMK46B4GOUWEMQE2A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:20.106080911 (601h42m19.977163946s) [seg_pos={0 0}, upload=527370]: Segment reached optimal threshold of 14 pieces.
Dec 30 17:11:20.106274520 (601h42m19.977357565s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106281644 (601h42m19.977364679s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106351607 (601h42m19.977434642s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106367767 (601h42m19.977450802s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106424967 (601h42m19.977508002s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106432451 (601h42m19.977515486s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106540666 (601h42m19.977623702s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106546207 (601h42m19.977629242s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106563380 (601h42m19.977646415s) [seg_pos={0 0}, upload=527370]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 17:11:20.106625529 (601h42m19.977708564s) [upload=527370]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 17:11:20.106629837 (601h42m19.977712882s) [upload=527370]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 17:11:20.406465452 (601h42m20.277548507s) [upload=527370]: Done uploading segments...
Dec 30 17:11:20.406504206 (601h42m20.277587251s) [upload=527370]: Upload finished. err: <nil>
Duration: 19.058943703s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
12 out of 60 is failed (threshold=6) [..........X..X....X.XX..X..X......X.........X..XXX..........] duration:
3.603905803s trace:
https://jaeger.dev.storj.io/trace/56692ebd5ab65c7c
FAIL (2025-12-30 17:40:11 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.6039998s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
49.777550094s trace:
https://jaeger.dev.storj.io/trace/5ea8d7d4f61e13a2
FAIL (2025-12-30 17:39:54 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK, Piece ID: HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ) context deadline exceeded; piecestore: (Node ID: 12eLKUg2vTnZMrLaraSPEoGXe3jDVBJYyURdcn4f2gCN5pvvCQ8, Piece ID: KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ) context deadline exceeded; piecestore: (Node ID: 12W4n6uf6azQk1QgeR8Xw7UkpnZm19zdyFKwojCsqZt2GFHrpce, Piece ID: DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ) context deadline exceeded
======LOGS======
17.114181822 (601h42m16.985264867s) [noise=true , address=145.79.227.44:4208 , piece_id=3OXDCHZSOIFS5VQCI4SXSAQFICDCYHRE5NGFBERR33KOTGSZWLDQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.114641387 (601h42m16.985724432s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.115652532 (601h42m16.986735578s) [noise=true , address=77.90.31.17:4790 , piece_id=5RSRESKGDA2YF4NBRF4ZFABBKWJK4NO6VLEXW2Z6EY3OPXH2WPNA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.115937134 (601h42m16.987020179s) [noise=true , address=64.49.13.155:4923 , piece_id=MUJD7XLM2YRLRCHYVSDPBE6VBYLHCOA7HJQP22ZSET4N574XQMJA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.116814986 (601h42m16.987898031s) [noise=true , address=45.91.116.11:4139 , piece_id=72DPEWZ7ZPZZ53UGYV2VCUKLCK7HCM22CDBRHWVLJGDGO5B5JADQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.117501182 (601h42m16.988584228s) [noise=true , address=77.90.31.14:4665 , piece_id=DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.118263334 (601h42m16.989346379s) [noise=true , address=145.79.227.40:4084 , piece_id=ZADZDRKBIILLAI4IT335CYETOSFOFUUPQIH6HZJCTHKRTZSPDJUQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.118555850 (601h42m16.989638886s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.118765930 (601h42m16.989849036s) [noise=true , address=77.90.31.21:4905 , piece_id=HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119029533 (601h42m16.990112568s) [noise=true , address=77.90.31.10:4533 , piece_id=QFBBYNQHQL2G7PEREK43FZBRUMKMVNBC2XKTE2NWTVTKN2452LPA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119417672 (601h42m16.990500707s) [noise=true , address=45.91.116.12:4258 , piece_id=KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119441588 (601h42m16.990524643s) [noise=true , address=145.79.227.40:4053 , piece_id=JCSIUMWKEAQNQXTNSJTN6VSB3V2SIDRQK67QMK46B4GOUWEMQE2A, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119763470 (601h42m16.990846505s) [noise=true , address=77.90.31.18:4821 , piece_id=MXZFKXM3JV2HQURDLIU7BKWLAKDZ27O4ER3SGC7QBZV5I3J2EHKA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.120028685 (601h42m16.99111172s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.120086986 (601h42m16.991170031s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.120586507 (601h42m16.991669562s) [noise=true , address=45.91.116.13:4168 , piece_id=AKUE5H3A2UECQOZY3GV3MWZKIHSNKCDOG32DKCNHIRTJROJ74M3A, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.121258768 (601h42m16.992341813s) [noise=true , address=145.79.227.9:4379 , piece_id=QTFXFSGGKYBQSPWJYBBJMQUL3D5EEO3IX4C53ECD7FXMUOGUBC7Q, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:18.568739165 (601h42m18.43982221s) [upload=527370]: Next returned nil segment
Dec 30 17:11:18.568751328 (601h42m18.439834363s) [upload=527370]: Waiting for error group managing segments...
Dec 30 17:11:18.978424178 (601h42m18.849507213s) [noise=true , address=64.49.13.155:4923 , piece_id=MUJD7XLM2YRLRCHYVSDPBE6VBYLHCOA7HJQP22ZSET4N574XQMJA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:18.986258125 (601h42m18.85734116s) [noise=true , address=145.79.227.9:4379 , piece_id=QTFXFSGGKYBQSPWJYBBJMQUL3D5EEO3IX4C53ECD7FXMUOGUBC7Q, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.274519421 (601h42m19.145602456s) [noise=true , address=77.90.31.14:4665 , piece_id=DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.315775438 (601h42m19.186858473s) [noise=true , address=45.91.116.13:4168 , piece_id=AKUE5H3A2UECQOZY3GV3MWZKIHSNKCDOG32DKCNHIRTJROJ74M3A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.335765371 (601h42m19.206848406s) [noise=true , address=77.90.31.21:4905 , piece_id=HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.354812129 (601h42m19.225895164s) [noise=true , address=77.90.31.10:4533 , piece_id=QFBBYNQHQL2G7PEREK43FZBRUMKMVNBC2XKTE2NWTVTKN2452LPA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.392024917 (601h42m19.263107952s) [noise=true , address=45.91.116.11:4139 , piece_id=72DPEWZ7ZPZZ53UGYV2VCUKLCK7HCM22CDBRHWVLJGDGO5B5JADQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.396084327 (601h42m19.267167372s) [noise=true , address=77.90.31.18:4821 , piece_id=MXZFKXM3JV2HQURDLIU7BKWLAKDZ27O4ER3SGC7QBZV5I3J2EHKA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.862099099 (601h42m19.733182134s) [noise=true , address=77.90.31.17:4790 , piece_id=5RSRESKGDA2YF4NBRF4ZFABBKWJK4NO6VLEXW2Z6EY3OPXH2WPNA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.905412913 (601h42m19.776495948s) [noise=true , address=145.79.227.44:4208 , piece_id=3OXDCHZSOIFS5VQCI4SXSAQFICDCYHRE5NGFBERR33KOTGSZWLDQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.910309688 (601h42m19.781392743s) [noise=true , address=145.79.227.30:4794 , piece_id=EJH6Q3T2BMPVLG6F7CW7N4GYOA3EZFPADL6VPT7WKKG65B2WZVXQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.934233361 (601h42m19.805316397s) [noise=true , address=145.79.227.40:4084 , piece_id=ZADZDRKBIILLAI4IT335CYETOSFOFUUPQIH6HZJCTHKRTZSPDJUQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:20.074011527 (601h42m19.945094572s) [noise=true , address=45.91.116.12:4258 , piece_id=KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:20.106010337 (601h42m19.977093372s) [noise=true , address=145.79.227.40:4053 , piece_id=JCSIUMWKEAQNQXTNSJTN6VSB3V2SIDRQK67QMK46B4GOUWEMQE2A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:20.106080911 (601h42m19.977163946s) [seg_pos={0 0}, upload=527370]: Segment reached optimal threshold of 14 pieces.
Dec 30 17:11:20.106274520 (601h42m19.977357565s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106281644 (601h42m19.977364679s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106351607 (601h42m19.977434642s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106367767 (601h42m19.977450802s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106424967 (601h42m19.977508002s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106432451 (601h42m19.977515486s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106540666 (601h42m19.977623702s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106546207 (601h42m19.977629242s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106563380 (601h42m19.977646415s) [seg_pos={0 0}, upload=527370]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 17:11:20.106625529 (601h42m19.977708564s) [upload=527370]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 17:11:20.106629837 (601h42m19.977712882s) [upload=527370]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 17:11:20.406465452 (601h42m20.277548507s) [upload=527370]: Done uploading segments...
Dec 30 17:11:20.406504206 (601h42m20.277587251s) [upload=527370]: Upload finished. err: <nil>
Duration: 49.777577226s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
28 out of 60 is failed (threshold=3) [X.X..XX.X.X.X....X.XXXXX.XX.X.XXX......XX.XXX...XX..X.....X.] duration:
6.527885279s trace:
https://jaeger.dev.storj.io/trace/19ecb93e18652e4d
FAIL (2025-12-30 17:39:48 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: ATM3LTIHV5VZJ6NOEYYWCXN7YL5P4KAAID6DGYIWTTHUI4Y6TPQQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: 6P4IN6T5FZKXOWB5FE7MLXTNRHHLV5RLNM4K3D7XAFALS5VMVA5Q) context deadline exceeded
======LOGS======
Dec 30 17:39:12.914699379 (602h10m12.785782424s) [upload=527786]: Starting upload
Dec 30 17:39:12.914760154 (602h10m12.785843189s) [upload=527786]: Done starting upload
Dec 30 17:39:12.914828985 (602h10m12.78591202s) [upload=527786]: Uploading segments...
Dec 30 17:39:12.914938964 (602h10m12.786021999s) [upload=527786]: Got next segment. Inline: false
Dec 30 17:39:12.914971557 (602h10m12.786054592s) [upload=527786]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 17:39:12.914975915 (602h10m12.78605896s) [upload=527786]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 17:39:13.164433145 (602h10m13.03551618s) [seg_pos={0 0}, upload=527786]: Begin upload segment...
Dec 30 17:39:13.164476629 (602h10m13.035559664s) [seg_pos={0 0}, upload=527786]: Done begin upload segment.
Dec 30 17:39:13.164565588 (602h10m13.035648633s) [noise=true , address=45.91.116.12:4226 , piece_id=57NRG5PIILZNLNSGHOSKSANQNEYC7YUFKEELRLPOEJMRNHQPAXKA, seg_pos={0 0}, upload=527786]: Uploading piece...
Dec 30 17:39:13.164571479 (602h10m13.035654534s) [noise=true , address=77.90.31.18:4805 , piece_id=Y5EFDVXJBD4C2VWW7LXNDPMPGHXUHNWHM7NUHFW3UNR3UCKK3KWA, seg_pos={0 0}, upload=527786]: Uploading piece...
Dec 30 17:39:13.165154128 (602h10m13.036237183s) [noise=true , address=45.91.116.9:4261 , piece_id=RF7IK6JXHDF7KSSEE2JI6HGL4I6HMSBFL5WS7MCT3Q2EX6GITXDQ, seg_pos={0 0}, upload=527786]: Uploading piece...
Dec 30 17:39:13.165666564 (602h10m13.03674961s) [noise=true , address=145.79.227.28:4725 , piece_id=ZIYPJOGMRSAHCVL4G46R2XETUBVQU4RDKLHUDBDRU2C77VPEWMQA, seg_pos={0 0}, upload=527786]: Uploading piece...
Dec 30 17:39:13.165821549 (602h10m13.036904584s) [noise=true , address=77.90.31.15:4700 , piece_id=MP4P7HYHUNUCRTNAOLPVYPSP3WAIFWEW77MFLHJKFBA653I2BNCQ, seg_pos={0 0}, upload=527786]: Uploading piece...
Dec 30 17:39:13.166088718 (602h10m13.037171763s) [noise=true , address=45.91.116.5:4299 , piece_id=ATM3LTIHV5VZJ6NOEYYWCXN7YL5P4KAAID6DGYIWTTHUI4Y6TPQQ, seg_pos={0 0}, upload=527786]: Uploading piece...
Dec 30 17:39:13.166292867 (602h10m13.037375912s) [noise=true , address=145.79.227.21:4475 , piece_id=YLOFJZOZ2SWTO5JSRNAUOV2VN3VPGOEXI6ICMKVP4C26UST4VIKQ, seg_pos={0 0}, upload=527786]: Uploading piece...
Dec 30 17:39:13.166487277 (602h10m13.037570312s) [noise=true , address=77.90.31.19:4844 , piece_id=FO6K3Z2VRYWGVZBDI63BK6XYLNTG7I5SGNWWWJ265CJX6DGIJZ3Q, seg_pos={0 0}, upload=527786]: Uploading piece...
Dec 30 17:39:13.166610572 (602h10m13.037693607s) [noise=true , address=145.79.227.50:4374 , piece_id=UT7E4XGTZYW3BVYC5AL5UJJ3Z6OHY3IJ4M4EZQRAEGU5S3QY63KQ, seg_pos={0 0}, upload=527786]: Uploading piece...
Dec 30 17:39:13.166876087 (602h10m13.037959122s) [noise=true , address=145.79.227.24:4581 , piece_id=NWJ4OVPD4RP5SRN7G73MKFS6NOXAXCZZB5GWQ6Z4HOK3ZYI2MKZA, seg_pos={0 0}, upload=527786]: Uploading piece...
Dec 30 17:39:13.166953104 (602h10m13.038036149s) [noise=true , address=145.79.227.15:4873 , piece_id=ZUM43WBALO47UUW27GNXLPNDAWTKCTB5VGVLUFSCLDA2Y3IC2Z3Q, seg_pos={0 0}, upload=527786]: Uploading piece...
Dec 30 17:39:13.167252384 (602h10m13.038335429s) [noise=true , address=145.79.227.27:4691 , piece_id=2F4YLQ7HN6TE5KPBYOLTKJ5AE7S7TUA5ZR7RHOISGLMLCP3NTIZQ, seg_pos={0 0}, upload=527786]: Uploading piece...
Dec 30 17:39:13.167367624 (602h10m13.038450669s) [noise=true , address=145.79.227.9:4367 , piece_id=MXRB67UFZSHPMPDMF3ASRTIDLYFBT6KRXKN2Z7WC3R5Q6GRA2NXQ, seg_pos={0 0}, upload=527786]: Uploading piece...
Dec 30 17:39:13.167654219 (602h10m13.038737264s) [noise=true , address=145.79.227.16:4910 , piece_id=4EG3OG6BBZIUNEZC6UCLMR7GULAWFKA34ANNENSTU4ASHECVQ2MA, seg_pos={0 0}, upload=527786]: Uploading piece...
Dec 30 17:39:13.167981272 (602h10m13.039064307s) [noise=true , address=77.90.31.21:4880 , piece_id=MGN77JMCIRUTR3ZHNVQKOANSBL63NECYLFJEABELE5ERWDPPV64Q, seg_pos={0 0}, upload=527786]: Uploading piece...
Dec 30 17:39:13.172782584 (602h10m13.043865629s) [noise=true , address=45.91.116.7:4012 , piece_id=6P4IN6T5FZKXOWB5FE7MLXTNRHHLV5RLNM4K3D7XAFALS5VMVA5Q, seg_pos={0 0}, upload=527786]: Uploading piece...
Dec 30 17:39:14.871218946 (602h10m14.742301981s) [upload=527786]: Next returned nil segment
Dec 30 17:39:14.871234014 (602h10m14.742317049s) [upload=527786]: Waiting for error group managing segments...
Dec 30 17:39:15.148769228 (602h10m15.019852273s) [noise=true , address=145.79.227.21:4475 , piece_id=YLOFJZOZ2SWTO5JSRNAUOV2VN3VPGOEXI6ICMKVP4C26UST4VIKQ, seg_pos={0 0}, upload=527786]: Done uploading piece. err: <nil>
Dec 30 17:39:15.162197066 (602h10m15.033280102s) [noise=true , address=145.79.227.50:4374 , piece_id=UT7E4XGTZYW3BVYC5AL5UJJ3Z6OHY3IJ4M4EZQRAEGU5S3QY63KQ, seg_pos={0 0}, upload=527786]: Done uploading piece. err: <nil>
Dec 30 17:39:15.165550842 (602h10m15.036633887s) [noise=true , address=145.79.227.9:4367 , piece_id=MXRB67UFZSHPMPDMF3ASRTIDLYFBT6KRXKN2Z7WC3R5Q6GRA2NXQ, seg_pos={0 0}, upload=527786]: Done uploading piece. err: <nil>
Dec 30 17:39:15.166450375 (602h10m15.03753341s) [noise=true , address=145.79.227.24:4581 , piece_id=NWJ4OVPD4RP5SRN7G73MKFS6NOXAXCZZB5GWQ6Z4HOK3ZYI2MKZA, seg_pos={0 0}, upload=527786]: Done uploading piece. err: <nil>
Dec 30 17:39:15.176632624 (602h10m15.047715659s) [noise=true , address=145.79.227.28:4725 , piece_id=ZIYPJOGMRSAHCVL4G46R2XETUBVQU4RDKLHUDBDRU2C77VPEWMQA, seg_pos={0 0}, upload=527786]: Done uploading piece. err: <nil>
Dec 30 17:39:15.178592306 (602h10m15.049675351s) [noise=true , address=145.79.227.15:4873 , piece_id=ZUM43WBALO47UUW27GNXLPNDAWTKCTB5VGVLUFSCLDA2Y3IC2Z3Q, seg_pos={0 0}, upload=527786]: Done uploading piece. err: <nil>
Dec 30 17:39:15.193202606 (602h10m15.064285641s) [noise=true , address=145.79.227.27:4691 , piece_id=2F4YLQ7HN6TE5KPBYOLTKJ5AE7S7TUA5ZR7RHOISGLMLCP3NTIZQ, seg_pos={0 0}, upload=527786]: Done uploading piece. err: <nil>
Dec 30 17:39:15.204478516 (602h10m15.075561551s) [noise=true , address=145.79.227.16:4910 , piece_id=4EG3OG6BBZIUNEZC6UCLMR7GULAWFKA34ANNENSTU4ASHECVQ2MA, seg_pos={0 0}, upload=527786]: Done uploading piece. err: <nil>
Dec 30 17:39:15.442079361 (602h10m15.313162396s) [noise=true , address=77.90.31.15:4700 , piece_id=MP4P7HYHUNUCRTNAOLPVYPSP3WAIFWEW77MFLHJKFBA653I2BNCQ, seg_pos={0 0}, upload=527786]: Done uploading piece. err: <nil>
Dec 30 17:39:15.567837782 (602h10m15.438920827s) [noise=true , address=45.91.116.9:4261 , piece_id=RF7IK6JXHDF7KSSEE2JI6HGL4I6HMSBFL5WS7MCT3Q2EX6GITXDQ, seg_pos={0 0}, upload=527786]: Done uploading piece. err: <nil>
Dec 30 17:39:15.590848016 (602h10m15.461931071s) [noise=true , address=45.91.116.5:4299 , piece_id=ATM3LTIHV5VZJ6NOEYYWCXN7YL5P4KAAID6DGYIWTTHUI4Y6TPQQ, seg_pos={0 0}, upload=527786]: Done uploading piece. err: <nil>
Dec 30 17:39:15.805493951 (602h10m15.676576986s) [noise=true , address=77.90.31.19:4844 , piece_id=FO6K3Z2VRYWGVZBDI63BK6XYLNTG7I5SGNWWWJ265CJX6DGIJZ3Q, seg_pos={0 0}, upload=527786]: Done uploading piece. err: <nil>
Dec 30 17:39:15.855110096 (602h10m15.726193131s) [noise=true , address=45.91.116.7:4012 , piece_id=6P4IN6T5FZKXOWB5FE7MLXTNRHHLV5RLNM4K3D7XAFALS5VMVA5Q, seg_pos={0 0}, upload=527786]: Done uploading piece. err: <nil>
Dec 30 17:39:15.855130065 (602h10m15.7262131s) [seg_pos={0 0}, upload=527786]: Segment reached optimal threshold of 13 pieces.
Dec 30 17:39:15.855446798 (602h10m15.726529833s) [noise=true , address=77.90.31.18:4805 , piece_id=Y5EFDVXJBD4C2VWW7LXNDPMPGHXUHNWHM7NUHFW3UNR3UCKK3KWA, seg_pos={0 0}, upload=527786]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF
Dec 30 17:39:15.855454031 (602h10m15.726537066s) [noise=true , address=77.90.31.18:4805 , piece_id=Y5EFDVXJBD4C2VWW7LXNDPMPGHXUHNWHM7NUHFW3UNR3UCKK3KWA, seg_pos={0 0}, upload=527786]: Long tail cancellation due to optimal threshold
Dec 30 17:39:15.855707033 (602h10m15.726790068s) [noise=true , address=77.90.31.21:4880 , piece_id=MGN77JMCIRUTR3ZHNVQKOANSBL63NECYLFJEABELE5ERWDPPV64Q, seg_pos={0 0}, upload=527786]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF
Dec 30 17:39:15.855744875 (602h10m15.72682792s) [noise=true , address=77.90.31.21:4880 , piece_id=MGN77JMCIRUTR3ZHNVQKOANSBL63NECYLFJEABELE5ERWDPPV64Q, seg_pos={0 0}, upload=527786]: Long tail cancellation due to optimal threshold
Dec 30 17:39:15.855904259 (602h10m15.726987294s) [noise=true , address=45.91.116.12:4226 , piece_id=57NRG5PIILZNLNSGHOSKSANQNEYC7YUFKEELRLPOEJMRNHQPAXKA, seg_pos={0 0}, upload=527786]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: read tcp 188.166.225.170:56104->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 30 17:39:15.855913817 (602h10m15.726996852s) [noise=true , address=45.91.116.12:4226 , piece_id=57NRG5PIILZNLNSGHOSKSANQNEYC7YUFKEELRLPOEJMRNHQPAXKA, seg_pos={0 0}, upload=527786]: Long tail cancellation due to optimal threshold
Dec 30 17:39:15.855931250 (602h10m15.727014285s) [seg_pos={0 0}, upload=527786]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 17:39:15.855998889 (602h10m15.727081924s) [upload=527786]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 17:39:15.856003207 (602h10m15.727086242s) [upload=527786]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 17:39:16.145556055 (602h10m16.016639331s) [upload=527786]: Done uploading segments...
Dec 30 17:39:16.145603636 (602h10m16.016686671s) [upload=527786]: Upload finished. err: <nil>
Duration: 6.527899697s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..........X...X.............................................] duration:
5.949712018s trace:
https://jaeger.dev.storj.io/trace/6ea66b1ed6f4c6e5
FAIL (2025-12-30 17:39:48 UTC)
orbiter warning: context deadline exceeded
Duration: 5.949746555s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
28 out of 60 is failed (threshold=3) [.X.X..XX.X.X.X....X.XXXXX.XX.X.XXX......XX.XXX...XX..X.....X] duration:
30.00367983s trace:
https://jaeger.dev.storj.io/trace/19ecb93e18652e4d
FAIL (2025-12-30 17:39:42 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: ATM3LTIHV5VZJ6NOEYYWCXN7YL5P4KAAID6DGYIWTTHUI4Y6TPQQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: 6P4IN6T5FZKXOWB5FE7MLXTNRHHLV5RLNM4K3D7XAFALS5VMVA5Q) context deadline exceeded
======LOGS======
Dec 30 17:39:12.914699379 (602h10m12.785782424s) [upload=527786]: Starting upload
Dec 30 17:39:12.914760154 (602h10m12.785843189s) [upload=527786]: Done starting upload
Dec 30 17:39:12.914828985 (602h10m12.78591202s) [upload=527786]: Uploading segments...
Dec 30 17:39:12.914938964 (602h10m12.786021999s) [upload=527786]: Got next segment. Inline: false
Dec 30 17:39:12.914971557 (602h10m12.786054592s) [upload=527786]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 17:39:12.914975915 (602h10m12.78605896s) [upload=527786]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 17:39:13.164433145 (602h10m13.03551618s) [seg_pos={0 0}, upload=527786]: Begin upload segment...
Dec 30 17:39:13.164476629 (602h10m13.035559664s) [seg_pos={0 0}, upload=527786]: Done begin upload segment.
Dec 30 17:39:13.164565588 (602h10m13.035648633s) [noise=true , address=45.91.116.12:4226 , piece_id=57NRG5PIILZNLNSGHOSKSANQNEYC7YUFKEELRLPOEJMRNHQPAXKA, seg_pos={0 0}, upload=527786]: Uploading piece...
Dec 30 17:39:13.164571479 (602h10m13.035654534s) [noise=true , address=77.90.31.18:4805 , piece_id=Y5EFDVXJBD4C2VWW7LXNDPMPGHXUHNWHM7NUHFW3UNR3UCKK3KWA, seg_pos={0 0}, upload=527786]: Uploading piece...
Dec 30 17:39:13.165154128 (602h10m13.036237183s) [noise=true , address=45.91.116.9:4261 , piece_id=RF7IK6JXHDF7KSSEE2JI6HGL4I6HMSBFL5WS7MCT3Q2EX6GITXDQ, seg_pos={0 0}, upload=527786]: Uploading piece...
Dec 30 17:39:13.165666564 (602h10m13.03674961s) [noise=true , address=145.79.227.28:4725 , piece_id=ZIYPJOGMRSAHCVL4G46R2XETUBVQU4RDKLHUDBDRU2C77VPEWMQA, seg_pos={0 0}, upload=527786]: Uploading piece...
Dec 30 17:39:13.165821549 (602h10m13.036904584s) [noise=true , address=77.90.31.15:4700 , piece_id=MP4P7HYHUNUCRTNAOLPVYPSP3WAIFWEW77MFLHJKFBA653I2BNCQ, seg_pos={0 0}, upload=527786]: Uploading piece...
Dec 30 17:39:13.166088718 (602h10m13.037171763s) [noise=true , address=45.91.116.5:4299 , piece_id=ATM3LTIHV5VZJ6NOEYYWCXN7YL5P4KAAID6DGYIWTTHUI4Y6TPQQ, seg_pos={0 0}, upload=527786]: Uploading piece...
Dec 30 17:39:13.166292867 (602h10m13.037375912s) [noise=true , address=145.79.227.21:4475 , piece_id=YLOFJZOZ2SWTO5JSRNAUOV2VN3VPGOEXI6ICMKVP4C26UST4VIKQ, seg_pos={0 0}, upload=527786]: Uploading piece...
Dec 30 17:39:13.166487277 (602h10m13.037570312s) [noise=true , address=77.90.31.19:4844 , piece_id=FO6K3Z2VRYWGVZBDI63BK6XYLNTG7I5SGNWWWJ265CJX6DGIJZ3Q, seg_pos={0 0}, upload=527786]: Uploading piece...
Dec 30 17:39:13.166610572 (602h10m13.037693607s) [noise=true , address=145.79.227.50:4374 , piece_id=UT7E4XGTZYW3BVYC5AL5UJJ3Z6OHY3IJ4M4EZQRAEGU5S3QY63KQ, seg_pos={0 0}, upload=527786]: Uploading piece...
Dec 30 17:39:13.166876087 (602h10m13.037959122s) [noise=true , address=145.79.227.24:4581 , piece_id=NWJ4OVPD4RP5SRN7G73MKFS6NOXAXCZZB5GWQ6Z4HOK3ZYI2MKZA, seg_pos={0 0}, upload=527786]: Uploading piece...
Dec 30 17:39:13.166953104 (602h10m13.038036149s) [noise=true , address=145.79.227.15:4873 , piece_id=ZUM43WBALO47UUW27GNXLPNDAWTKCTB5VGVLUFSCLDA2Y3IC2Z3Q, seg_pos={0 0}, upload=527786]: Uploading piece...
Dec 30 17:39:13.167252384 (602h10m13.038335429s) [noise=true , address=145.79.227.27:4691 , piece_id=2F4YLQ7HN6TE5KPBYOLTKJ5AE7S7TUA5ZR7RHOISGLMLCP3NTIZQ, seg_pos={0 0}, upload=527786]: Uploading piece...
Dec 30 17:39:13.167367624 (602h10m13.038450669s) [noise=true , address=145.79.227.9:4367 , piece_id=MXRB67UFZSHPMPDMF3ASRTIDLYFBT6KRXKN2Z7WC3R5Q6GRA2NXQ, seg_pos={0 0}, upload=527786]: Uploading piece...
Dec 30 17:39:13.167654219 (602h10m13.038737264s) [noise=true , address=145.79.227.16:4910 , piece_id=4EG3OG6BBZIUNEZC6UCLMR7GULAWFKA34ANNENSTU4ASHECVQ2MA, seg_pos={0 0}, upload=527786]: Uploading piece...
Dec 30 17:39:13.167981272 (602h10m13.039064307s) [noise=true , address=77.90.31.21:4880 , piece_id=MGN77JMCIRUTR3ZHNVQKOANSBL63NECYLFJEABELE5ERWDPPV64Q, seg_pos={0 0}, upload=527786]: Uploading piece...
Dec 30 17:39:13.172782584 (602h10m13.043865629s) [noise=true , address=45.91.116.7:4012 , piece_id=6P4IN6T5FZKXOWB5FE7MLXTNRHHLV5RLNM4K3D7XAFALS5VMVA5Q, seg_pos={0 0}, upload=527786]: Uploading piece...
Dec 30 17:39:14.871218946 (602h10m14.742301981s) [upload=527786]: Next returned nil segment
Dec 30 17:39:14.871234014 (602h10m14.742317049s) [upload=527786]: Waiting for error group managing segments...
Dec 30 17:39:15.148769228 (602h10m15.019852273s) [noise=true , address=145.79.227.21:4475 , piece_id=YLOFJZOZ2SWTO5JSRNAUOV2VN3VPGOEXI6ICMKVP4C26UST4VIKQ, seg_pos={0 0}, upload=527786]: Done uploading piece. err: <nil>
Dec 30 17:39:15.162197066 (602h10m15.033280102s) [noise=true , address=145.79.227.50:4374 , piece_id=UT7E4XGTZYW3BVYC5AL5UJJ3Z6OHY3IJ4M4EZQRAEGU5S3QY63KQ, seg_pos={0 0}, upload=527786]: Done uploading piece. err: <nil>
Dec 30 17:39:15.165550842 (602h10m15.036633887s) [noise=true , address=145.79.227.9:4367 , piece_id=MXRB67UFZSHPMPDMF3ASRTIDLYFBT6KRXKN2Z7WC3R5Q6GRA2NXQ, seg_pos={0 0}, upload=527786]: Done uploading piece. err: <nil>
Dec 30 17:39:15.166450375 (602h10m15.03753341s) [noise=true , address=145.79.227.24:4581 , piece_id=NWJ4OVPD4RP5SRN7G73MKFS6NOXAXCZZB5GWQ6Z4HOK3ZYI2MKZA, seg_pos={0 0}, upload=527786]: Done uploading piece. err: <nil>
Dec 30 17:39:15.176632624 (602h10m15.047715659s) [noise=true , address=145.79.227.28:4725 , piece_id=ZIYPJOGMRSAHCVL4G46R2XETUBVQU4RDKLHUDBDRU2C77VPEWMQA, seg_pos={0 0}, upload=527786]: Done uploading piece. err: <nil>
Dec 30 17:39:15.178592306 (602h10m15.049675351s) [noise=true , address=145.79.227.15:4873 , piece_id=ZUM43WBALO47UUW27GNXLPNDAWTKCTB5VGVLUFSCLDA2Y3IC2Z3Q, seg_pos={0 0}, upload=527786]: Done uploading piece. err: <nil>
Dec 30 17:39:15.193202606 (602h10m15.064285641s) [noise=true , address=145.79.227.27:4691 , piece_id=2F4YLQ7HN6TE5KPBYOLTKJ5AE7S7TUA5ZR7RHOISGLMLCP3NTIZQ, seg_pos={0 0}, upload=527786]: Done uploading piece. err: <nil>
Dec 30 17:39:15.204478516 (602h10m15.075561551s) [noise=true , address=145.79.227.16:4910 , piece_id=4EG3OG6BBZIUNEZC6UCLMR7GULAWFKA34ANNENSTU4ASHECVQ2MA, seg_pos={0 0}, upload=527786]: Done uploading piece. err: <nil>
Dec 30 17:39:15.442079361 (602h10m15.313162396s) [noise=true , address=77.90.31.15:4700 , piece_id=MP4P7HYHUNUCRTNAOLPVYPSP3WAIFWEW77MFLHJKFBA653I2BNCQ, seg_pos={0 0}, upload=527786]: Done uploading piece. err: <nil>
Dec 30 17:39:15.567837782 (602h10m15.438920827s) [noise=true , address=45.91.116.9:4261 , piece_id=RF7IK6JXHDF7KSSEE2JI6HGL4I6HMSBFL5WS7MCT3Q2EX6GITXDQ, seg_pos={0 0}, upload=527786]: Done uploading piece. err: <nil>
Dec 30 17:39:15.590848016 (602h10m15.461931071s) [noise=true , address=45.91.116.5:4299 , piece_id=ATM3LTIHV5VZJ6NOEYYWCXN7YL5P4KAAID6DGYIWTTHUI4Y6TPQQ, seg_pos={0 0}, upload=527786]: Done uploading piece. err: <nil>
Dec 30 17:39:15.805493951 (602h10m15.676576986s) [noise=true , address=77.90.31.19:4844 , piece_id=FO6K3Z2VRYWGVZBDI63BK6XYLNTG7I5SGNWWWJ265CJX6DGIJZ3Q, seg_pos={0 0}, upload=527786]: Done uploading piece. err: <nil>
Dec 30 17:39:15.855110096 (602h10m15.726193131s) [noise=true , address=45.91.116.7:4012 , piece_id=6P4IN6T5FZKXOWB5FE7MLXTNRHHLV5RLNM4K3D7XAFALS5VMVA5Q, seg_pos={0 0}, upload=527786]: Done uploading piece. err: <nil>
Dec 30 17:39:15.855130065 (602h10m15.7262131s) [seg_pos={0 0}, upload=527786]: Segment reached optimal threshold of 13 pieces.
Dec 30 17:39:15.855446798 (602h10m15.726529833s) [noise=true , address=77.90.31.18:4805 , piece_id=Y5EFDVXJBD4C2VWW7LXNDPMPGHXUHNWHM7NUHFW3UNR3UCKK3KWA, seg_pos={0 0}, upload=527786]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF
Dec 30 17:39:15.855454031 (602h10m15.726537066s) [noise=true , address=77.90.31.18:4805 , piece_id=Y5EFDVXJBD4C2VWW7LXNDPMPGHXUHNWHM7NUHFW3UNR3UCKK3KWA, seg_pos={0 0}, upload=527786]: Long tail cancellation due to optimal threshold
Dec 30 17:39:15.855707033 (602h10m15.726790068s) [noise=true , address=77.90.31.21:4880 , piece_id=MGN77JMCIRUTR3ZHNVQKOANSBL63NECYLFJEABELE5ERWDPPV64Q, seg_pos={0 0}, upload=527786]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF
Dec 30 17:39:15.855744875 (602h10m15.72682792s) [noise=true , address=77.90.31.21:4880 , piece_id=MGN77JMCIRUTR3ZHNVQKOANSBL63NECYLFJEABELE5ERWDPPV64Q, seg_pos={0 0}, upload=527786]: Long tail cancellation due to optimal threshold
Dec 30 17:39:15.855904259 (602h10m15.726987294s) [noise=true , address=45.91.116.12:4226 , piece_id=57NRG5PIILZNLNSGHOSKSANQNEYC7YUFKEELRLPOEJMRNHQPAXKA, seg_pos={0 0}, upload=527786]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: read tcp 188.166.225.170:56104->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 30 17:39:15.855913817 (602h10m15.726996852s) [noise=true , address=45.91.116.12:4226 , piece_id=57NRG5PIILZNLNSGHOSKSANQNEYC7YUFKEELRLPOEJMRNHQPAXKA, seg_pos={0 0}, upload=527786]: Long tail cancellation due to optimal threshold
Dec 30 17:39:15.855931250 (602h10m15.727014285s) [seg_pos={0 0}, upload=527786]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 17:39:15.855998889 (602h10m15.727081924s) [upload=527786]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 17:39:15.856003207 (602h10m15.727086242s) [upload=527786]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 17:39:16.145556055 (602h10m16.016639331s) [upload=527786]: Done uploading segments...
Dec 30 17:39:16.145603636 (602h10m16.016686671s) [upload=527786]: Upload finished. err: <nil>
Duration: 30.003700819s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
12 out of 60 is failed (threshold=6) [...........X..X....X.XX..X..X......X.........X..XXX.........] duration:
3.718258534s trace:
https://jaeger.dev.storj.io/trace/56692ebd5ab65c7c
FAIL (2025-12-30 17:39:29 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.718284173s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [..............X...X..X............................XX........] duration:
7.027862394s trace:
https://jaeger.dev.storj.io/trace/4173ea2fcd946b89
FAIL (2025-12-30 17:39:11 UTC)
orbiter warning: context deadline exceeded
Duration: 7.027915967s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
52.275572638s trace:
https://jaeger.dev.storj.io/trace/5ea8d7d4f61e13a2
FAIL (2025-12-30 17:38:44 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK, Piece ID: HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ) context deadline exceeded; piecestore: (Node ID: 12eLKUg2vTnZMrLaraSPEoGXe3jDVBJYyURdcn4f2gCN5pvvCQ8, Piece ID: KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ) context deadline exceeded; piecestore: (Node ID: 12W4n6uf6azQk1QgeR8Xw7UkpnZm19zdyFKwojCsqZt2GFHrpce, Piece ID: DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ) context deadline exceeded
======LOGS======
17.114181822 (601h42m16.985264867s) [noise=true , address=145.79.227.44:4208 , piece_id=3OXDCHZSOIFS5VQCI4SXSAQFICDCYHRE5NGFBERR33KOTGSZWLDQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.114641387 (601h42m16.985724432s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.115652532 (601h42m16.986735578s) [noise=true , address=77.90.31.17:4790 , piece_id=5RSRESKGDA2YF4NBRF4ZFABBKWJK4NO6VLEXW2Z6EY3OPXH2WPNA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.115937134 (601h42m16.987020179s) [noise=true , address=64.49.13.155:4923 , piece_id=MUJD7XLM2YRLRCHYVSDPBE6VBYLHCOA7HJQP22ZSET4N574XQMJA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.116814986 (601h42m16.987898031s) [noise=true , address=45.91.116.11:4139 , piece_id=72DPEWZ7ZPZZ53UGYV2VCUKLCK7HCM22CDBRHWVLJGDGO5B5JADQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.117501182 (601h42m16.988584228s) [noise=true , address=77.90.31.14:4665 , piece_id=DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.118263334 (601h42m16.989346379s) [noise=true , address=145.79.227.40:4084 , piece_id=ZADZDRKBIILLAI4IT335CYETOSFOFUUPQIH6HZJCTHKRTZSPDJUQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.118555850 (601h42m16.989638886s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.118765930 (601h42m16.989849036s) [noise=true , address=77.90.31.21:4905 , piece_id=HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119029533 (601h42m16.990112568s) [noise=true , address=77.90.31.10:4533 , piece_id=QFBBYNQHQL2G7PEREK43FZBRUMKMVNBC2XKTE2NWTVTKN2452LPA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119417672 (601h42m16.990500707s) [noise=true , address=45.91.116.12:4258 , piece_id=KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119441588 (601h42m16.990524643s) [noise=true , address=145.79.227.40:4053 , piece_id=JCSIUMWKEAQNQXTNSJTN6VSB3V2SIDRQK67QMK46B4GOUWEMQE2A, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119763470 (601h42m16.990846505s) [noise=true , address=77.90.31.18:4821 , piece_id=MXZFKXM3JV2HQURDLIU7BKWLAKDZ27O4ER3SGC7QBZV5I3J2EHKA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.120028685 (601h42m16.99111172s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.120086986 (601h42m16.991170031s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.120586507 (601h42m16.991669562s) [noise=true , address=45.91.116.13:4168 , piece_id=AKUE5H3A2UECQOZY3GV3MWZKIHSNKCDOG32DKCNHIRTJROJ74M3A, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.121258768 (601h42m16.992341813s) [noise=true , address=145.79.227.9:4379 , piece_id=QTFXFSGGKYBQSPWJYBBJMQUL3D5EEO3IX4C53ECD7FXMUOGUBC7Q, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:18.568739165 (601h42m18.43982221s) [upload=527370]: Next returned nil segment
Dec 30 17:11:18.568751328 (601h42m18.439834363s) [upload=527370]: Waiting for error group managing segments...
Dec 30 17:11:18.978424178 (601h42m18.849507213s) [noise=true , address=64.49.13.155:4923 , piece_id=MUJD7XLM2YRLRCHYVSDPBE6VBYLHCOA7HJQP22ZSET4N574XQMJA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:18.986258125 (601h42m18.85734116s) [noise=true , address=145.79.227.9:4379 , piece_id=QTFXFSGGKYBQSPWJYBBJMQUL3D5EEO3IX4C53ECD7FXMUOGUBC7Q, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.274519421 (601h42m19.145602456s) [noise=true , address=77.90.31.14:4665 , piece_id=DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.315775438 (601h42m19.186858473s) [noise=true , address=45.91.116.13:4168 , piece_id=AKUE5H3A2UECQOZY3GV3MWZKIHSNKCDOG32DKCNHIRTJROJ74M3A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.335765371 (601h42m19.206848406s) [noise=true , address=77.90.31.21:4905 , piece_id=HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.354812129 (601h42m19.225895164s) [noise=true , address=77.90.31.10:4533 , piece_id=QFBBYNQHQL2G7PEREK43FZBRUMKMVNBC2XKTE2NWTVTKN2452LPA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.392024917 (601h42m19.263107952s) [noise=true , address=45.91.116.11:4139 , piece_id=72DPEWZ7ZPZZ53UGYV2VCUKLCK7HCM22CDBRHWVLJGDGO5B5JADQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.396084327 (601h42m19.267167372s) [noise=true , address=77.90.31.18:4821 , piece_id=MXZFKXM3JV2HQURDLIU7BKWLAKDZ27O4ER3SGC7QBZV5I3J2EHKA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.862099099 (601h42m19.733182134s) [noise=true , address=77.90.31.17:4790 , piece_id=5RSRESKGDA2YF4NBRF4ZFABBKWJK4NO6VLEXW2Z6EY3OPXH2WPNA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.905412913 (601h42m19.776495948s) [noise=true , address=145.79.227.44:4208 , piece_id=3OXDCHZSOIFS5VQCI4SXSAQFICDCYHRE5NGFBERR33KOTGSZWLDQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.910309688 (601h42m19.781392743s) [noise=true , address=145.79.227.30:4794 , piece_id=EJH6Q3T2BMPVLG6F7CW7N4GYOA3EZFPADL6VPT7WKKG65B2WZVXQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.934233361 (601h42m19.805316397s) [noise=true , address=145.79.227.40:4084 , piece_id=ZADZDRKBIILLAI4IT335CYETOSFOFUUPQIH6HZJCTHKRTZSPDJUQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:20.074011527 (601h42m19.945094572s) [noise=true , address=45.91.116.12:4258 , piece_id=KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:20.106010337 (601h42m19.977093372s) [noise=true , address=145.79.227.40:4053 , piece_id=JCSIUMWKEAQNQXTNSJTN6VSB3V2SIDRQK67QMK46B4GOUWEMQE2A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:20.106080911 (601h42m19.977163946s) [seg_pos={0 0}, upload=527370]: Segment reached optimal threshold of 14 pieces.
Dec 30 17:11:20.106274520 (601h42m19.977357565s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106281644 (601h42m19.977364679s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106351607 (601h42m19.977434642s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106367767 (601h42m19.977450802s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106424967 (601h42m19.977508002s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106432451 (601h42m19.977515486s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106540666 (601h42m19.977623702s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106546207 (601h42m19.977629242s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106563380 (601h42m19.977646415s) [seg_pos={0 0}, upload=527370]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 17:11:20.106625529 (601h42m19.977708564s) [upload=527370]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 17:11:20.106629837 (601h42m19.977712882s) [upload=527370]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 17:11:20.406465452 (601h42m20.277548507s) [upload=527370]: Done uploading segments...
Dec 30 17:11:20.406504206 (601h42m20.277587251s) [upload=527370]: Upload finished. err: <nil>
Duration: 52.275599469s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [...............X...X..X............................XX.......] duration:
5.877704675s trace:
https://jaeger.dev.storj.io/trace/4173ea2fcd946b89
FAIL (2025-12-30 17:38:44 UTC)
orbiter warning: context deadline exceeded
Duration: 5.877728829s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...........X...X............................................] duration:
5.625041471s trace:
https://jaeger.dev.storj.io/trace/6ea66b1ed6f4c6e5
FAIL (2025-12-30 17:38:44 UTC)
orbiter warning: context deadline exceeded
Duration: 5.625069313s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
27 out of 60 is failed (threshold=3) [..X.X..XX.X.X.X....X.XXXXX.XX.X.XXX......XX.XXX...XX..X.....] duration:
6.002874088s trace:
https://jaeger.dev.storj.io/trace/3a1f7d23a1ef3354
FAIL (2025-12-30 17:38:35 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 30 17:34:45.008933692 (602h5m44.880016738s) [upload=527721]: Starting upload
Dec 30 17:34:45.009013564 (602h5m44.880096609s) [upload=527721]: Done starting upload
Dec 30 17:34:45.009131229 (602h5m44.880214274s) [upload=527721]: Uploading segments...
Dec 30 17:34:45.009254303 (602h5m44.880337348s) [upload=527721]: Got next segment. Inline: false
Dec 30 17:34:45.009307504 (602h5m44.88039056s) [upload=527721]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 17:34:45.009314247 (602h5m44.880397292s) [upload=527721]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 17:34:45.260909008 (602h5m45.131992073s) [seg_pos={0 0}, upload=527721]: Begin upload segment...
Dec 30 17:34:45.260991535 (602h5m45.13207458s) [seg_pos={0 0}, upload=527721]: Done begin upload segment.
Dec 30 17:34:45.261025630 (602h5m45.132108675s) [noise=true , address=45.91.116.9:4262 , piece_id=GXQMXHK6DSHZ6NBITKSPWLCCCRF5SUYI5VB7IO6FIMAD2FKYNVWA, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.261066187 (602h5m45.132149232s) [noise=true , address=145.79.227.50:4373 , piece_id=G37TCLHWAN4YWBPADLY3SBOTCCYAICFDQGK7WIYTGWZIKRSAJM3A, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.261501726 (602h5m45.132584761s) [noise=true , address=145.79.227.45:4230 , piece_id=X2ROAPLIHYN63DNINPSWV2YLZILWFMJV3P3ASEI5PDIFRXOJJJEQ, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.261952074 (602h5m45.133035119s) [noise=true , address=45.91.116.8:4045 , piece_id=Q3RGXD2UR4ERW4KELMX4N7HSD5AP4IVLIVWS7UYL72WM3AIS2UIA, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.262315346 (602h5m45.133398381s) [noise=true , address=145.79.227.27:4691 , piece_id=IGC6IL4673U7NUHOH26UKY6TZUTOQZLBDSKL4JXV3MNTLCXCOK7Q, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.262584518 (602h5m45.133667563s) [noise=true , address=145.79.227.9:4366 , piece_id=65KRZWTXUK5CT6Q5334FXBIKDVYZ6GNF4AWIUKFFSS5MFNPHOWFQ, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.262907303 (602h5m45.133990348s) [noise=true , address=145.79.227.31:4799 , piece_id=KUELWFEJRZ5AKH4ANMBA4VWSABRKKHEJ5QF62LMPDX5PIO44Z5DA, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.263203096 (602h5m45.134286141s) [noise=true , address=145.79.227.23:4546 , piece_id=D2P5BLURS4DA47VFEDTIJ3EVNVXKC5YMDBDOHVBWMAICAOVDXMKQ, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.263626272 (602h5m45.134709317s) [noise=true , address=145.79.227.11:4731 , piece_id=PFDYMYBZVHSHRVX63RJXKDCKGWNSXKDOFMORFPXO564HQ2SU4JPQ, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.265075592 (602h5m45.136158637s) [noise=true , address=45.91.116.12:4225 , piece_id=SUUGK5PEBWJUTZOWYIYMSJRFBWGT3D7VWNYG76T6HUQF6F2YB2IQ, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.266166759 (602h5m45.137249804s) [noise=true , address=145.79.227.22:4510 , piece_id=XWR5JQBOOU5FF226G6JY266ZE5OENBAJJ2P353UOI3QSZGRBQUIQ, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.267379789 (602h5m45.138462824s) [noise=true , address=145.79.227.21:4473 , piece_id=2FH5XZNFRXSWKQOLJ2ZAL26Y3KMWZZBQL2RTEBQSARKJFSEXQXFA, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.268484482 (602h5m45.139567527s) [noise=true , address=145.79.227.16:4910 , piece_id=XSJ4YM7I3BIIG3O3X3LK7HSVQKL5AA3Q4ZLWMJYQDM223AFQUFWA, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.269628030 (602h5m45.140711075s) [noise=true , address=45.91.116.11:4117 , piece_id=CWPARDQZXXV46MFGPDYLAFQTPPYXD25IEQ4FIZQLEPJQK6HVKB3A, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.272821340 (602h5m45.143904386s) [noise=true , address=45.91.116.14:4189 , piece_id=EN6UQV4BZ4M635TZ6HSSN45UW65LLBKMPU6EMZXGNQLHR3MARUGQ, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.274412831 (602h5m45.145495876s) [noise=true , address=145.79.227.35:4482 , piece_id=WWZLMRQNPXAW53HCZZP5I2NAQM66OXRE2IATP366UJFZHCJMI2VQ, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:46.959387699 (602h5m46.830470735s) [upload=527721]: Next returned nil segment
Dec 30 17:34:46.959401716 (602h5m46.830484751s) [upload=527721]: Waiting for error group managing segments...
Dec 30 17:34:47.233771963 (602h5m47.104854998s) [noise=true , address=145.79.227.45:4230 , piece_id=X2ROAPLIHYN63DNINPSWV2YLZILWFMJV3P3ASEI5PDIFRXOJJJEQ, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.248691974 (602h5m47.119775009s) [noise=true , address=145.79.227.11:4731 , piece_id=PFDYMYBZVHSHRVX63RJXKDCKGWNSXKDOFMORFPXO564HQ2SU4JPQ, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.252485076 (602h5m47.123568111s) [noise=true , address=145.79.227.27:4691 , piece_id=IGC6IL4673U7NUHOH26UKY6TZUTOQZLBDSKL4JXV3MNTLCXCOK7Q, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.261360666 (602h5m47.132443702s) [noise=true , address=145.79.227.16:4910 , piece_id=XSJ4YM7I3BIIG3O3X3LK7HSVQKL5AA3Q4ZLWMJYQDM223AFQUFWA, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.265225967 (602h5m47.136309012s) [noise=true , address=145.79.227.23:4546 , piece_id=D2P5BLURS4DA47VFEDTIJ3EVNVXKC5YMDBDOHVBWMAICAOVDXMKQ, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.272045612 (602h5m47.143128648s) [noise=true , address=145.79.227.50:4373 , piece_id=G37TCLHWAN4YWBPADLY3SBOTCCYAICFDQGK7WIYTGWZIKRSAJM3A, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.279425644 (602h5m47.150508679s) [noise=true , address=145.79.227.9:4366 , piece_id=65KRZWTXUK5CT6Q5334FXBIKDVYZ6GNF4AWIUKFFSS5MFNPHOWFQ, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.284168746 (602h5m47.155251781s) [noise=true , address=145.79.227.21:4473 , piece_id=2FH5XZNFRXSWKQOLJ2ZAL26Y3KMWZZBQL2RTEBQSARKJFSEXQXFA, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.284835917 (602h5m47.155918962s) [noise=true , address=145.79.227.31:4799 , piece_id=KUELWFEJRZ5AKH4ANMBA4VWSABRKKHEJ5QF62LMPDX5PIO44Z5DA, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.288884154 (602h5m47.159967189s) [noise=true , address=145.79.227.22:4510 , piece_id=XWR5JQBOOU5FF226G6JY266ZE5OENBAJJ2P353UOI3QSZGRBQUIQ, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.326381835 (602h5m47.19746487s) [noise=true , address=145.79.227.35:4482 , piece_id=WWZLMRQNPXAW53HCZZP5I2NAQM66OXRE2IATP366UJFZHCJMI2VQ, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.583738587 (602h5m47.454821632s) [noise=true , address=45.91.116.9:4262 , piece_id=GXQMXHK6DSHZ6NBITKSPWLCCCRF5SUYI5VB7IO6FIMAD2FKYNVWA, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.625677305 (602h5m47.49676035s) [noise=true , address=45.91.116.8:4045 , piece_id=Q3RGXD2UR4ERW4KELMX4N7HSD5AP4IVLIVWS7UYL72WM3AIS2UIA, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.625694757 (602h5m47.496777792s) [seg_pos={0 0}, upload=527721]: Segment reached optimal threshold of 13 pieces.
Dec 30 17:34:47.625981503 (602h5m47.497064548s) [noise=true , address=45.91.116.14:4189 , piece_id=EN6UQV4BZ4M635TZ6HSSN45UW65LLBKMPU6EMZXGNQLHR3MARUGQ, seg_pos={0 0}, upload=527721]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF
Dec 30 17:34:47.625991513 (602h5m47.497074548s) [noise=true , address=45.91.116.14:4189 , piece_id=EN6UQV4BZ4M635TZ6HSSN45UW65LLBKMPU6EMZXGNQLHR3MARUGQ, seg_pos={0 0}, upload=527721]: Long tail cancellation due to optimal threshold
Dec 30 17:34:47.626166286 (602h5m47.497249331s) [noise=true , address=45.91.116.12:4225 , piece_id=SUUGK5PEBWJUTZOWYIYMSJRFBWGT3D7VWNYG76T6HUQF6F2YB2IQ, seg_pos={0 0}, upload=527721]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF
Dec 30 17:34:47.626174451 (602h5m47.497257486s) [noise=true , address=45.91.116.12:4225 , piece_id=SUUGK5PEBWJUTZOWYIYMSJRFBWGT3D7VWNYG76T6HUQF6F2YB2IQ, seg_pos={0 0}, upload=527721]: Long tail cancellation due to optimal threshold
Dec 30 17:34:47.626257829 (602h5m47.497340864s) [noise=true , address=45.91.116.11:4117 , piece_id=CWPARDQZXXV46MFGPDYLAFQTPPYXD25IEQ4FIZQLEPJQK6HVKB3A, seg_pos={0 0}, upload=527721]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Dec 30 17:34:47.626267428 (602h5m47.497350463s) [noise=true , address=45.91.116.11:4117 , piece_id=CWPARDQZXXV46MFGPDYLAFQTPPYXD25IEQ4FIZQLEPJQK6HVKB3A, seg_pos={0 0}, upload=527721]: Long tail cancellation due to optimal threshold
Dec 30 17:34:47.626304649 (602h5m47.497387694s) [seg_pos={0 0}, upload=527721]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 17:34:47.626426300 (602h5m47.497509335s) [upload=527721]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 17:34:47.626433895 (602h5m47.49751693s) [upload=527721]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 17:34:47.916655325 (602h5m47.78773838s) [upload=527721]: Done uploading segments...
Dec 30 17:34:47.916680373 (602h5m47.787763408s) [upload=527721]: Upload finished. err: <nil>
Duration: 6.002899836s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
12 out of 60 is failed (threshold=6) [............X..X....X.XX..X..X......X.........X..XXX........] duration:
3.549827224s trace:
https://jaeger.dev.storj.io/trace/56692ebd5ab65c7c
FAIL (2025-12-30 17:38:25 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.549853785s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [............X...X...........................................] duration:
5.93139182s trace:
https://jaeger.dev.storj.io/trace/6ea66b1ed6f4c6e5
FAIL (2025-12-30 17:38:20 UTC)
orbiter warning: context deadline exceeded
Duration: 5.931419832s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
27 out of 60 is failed (threshold=3) [...X.X..XX.X.X.X....X.XXXXX.XX.X.XXX......XX.XXX...XX..X....] duration:
5.927729949s trace:
https://jaeger.dev.storj.io/trace/3a1f7d23a1ef3354
FAIL (2025-12-30 17:38:15 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 30 17:34:45.008933692 (602h5m44.880016738s) [upload=527721]: Starting upload
Dec 30 17:34:45.009013564 (602h5m44.880096609s) [upload=527721]: Done starting upload
Dec 30 17:34:45.009131229 (602h5m44.880214274s) [upload=527721]: Uploading segments...
Dec 30 17:34:45.009254303 (602h5m44.880337348s) [upload=527721]: Got next segment. Inline: false
Dec 30 17:34:45.009307504 (602h5m44.88039056s) [upload=527721]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 17:34:45.009314247 (602h5m44.880397292s) [upload=527721]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 17:34:45.260909008 (602h5m45.131992073s) [seg_pos={0 0}, upload=527721]: Begin upload segment...
Dec 30 17:34:45.260991535 (602h5m45.13207458s) [seg_pos={0 0}, upload=527721]: Done begin upload segment.
Dec 30 17:34:45.261025630 (602h5m45.132108675s) [noise=true , address=45.91.116.9:4262 , piece_id=GXQMXHK6DSHZ6NBITKSPWLCCCRF5SUYI5VB7IO6FIMAD2FKYNVWA, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.261066187 (602h5m45.132149232s) [noise=true , address=145.79.227.50:4373 , piece_id=G37TCLHWAN4YWBPADLY3SBOTCCYAICFDQGK7WIYTGWZIKRSAJM3A, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.261501726 (602h5m45.132584761s) [noise=true , address=145.79.227.45:4230 , piece_id=X2ROAPLIHYN63DNINPSWV2YLZILWFMJV3P3ASEI5PDIFRXOJJJEQ, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.261952074 (602h5m45.133035119s) [noise=true , address=45.91.116.8:4045 , piece_id=Q3RGXD2UR4ERW4KELMX4N7HSD5AP4IVLIVWS7UYL72WM3AIS2UIA, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.262315346 (602h5m45.133398381s) [noise=true , address=145.79.227.27:4691 , piece_id=IGC6IL4673U7NUHOH26UKY6TZUTOQZLBDSKL4JXV3MNTLCXCOK7Q, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.262584518 (602h5m45.133667563s) [noise=true , address=145.79.227.9:4366 , piece_id=65KRZWTXUK5CT6Q5334FXBIKDVYZ6GNF4AWIUKFFSS5MFNPHOWFQ, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.262907303 (602h5m45.133990348s) [noise=true , address=145.79.227.31:4799 , piece_id=KUELWFEJRZ5AKH4ANMBA4VWSABRKKHEJ5QF62LMPDX5PIO44Z5DA, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.263203096 (602h5m45.134286141s) [noise=true , address=145.79.227.23:4546 , piece_id=D2P5BLURS4DA47VFEDTIJ3EVNVXKC5YMDBDOHVBWMAICAOVDXMKQ, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.263626272 (602h5m45.134709317s) [noise=true , address=145.79.227.11:4731 , piece_id=PFDYMYBZVHSHRVX63RJXKDCKGWNSXKDOFMORFPXO564HQ2SU4JPQ, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.265075592 (602h5m45.136158637s) [noise=true , address=45.91.116.12:4225 , piece_id=SUUGK5PEBWJUTZOWYIYMSJRFBWGT3D7VWNYG76T6HUQF6F2YB2IQ, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.266166759 (602h5m45.137249804s) [noise=true , address=145.79.227.22:4510 , piece_id=XWR5JQBOOU5FF226G6JY266ZE5OENBAJJ2P353UOI3QSZGRBQUIQ, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.267379789 (602h5m45.138462824s) [noise=true , address=145.79.227.21:4473 , piece_id=2FH5XZNFRXSWKQOLJ2ZAL26Y3KMWZZBQL2RTEBQSARKJFSEXQXFA, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.268484482 (602h5m45.139567527s) [noise=true , address=145.79.227.16:4910 , piece_id=XSJ4YM7I3BIIG3O3X3LK7HSVQKL5AA3Q4ZLWMJYQDM223AFQUFWA, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.269628030 (602h5m45.140711075s) [noise=true , address=45.91.116.11:4117 , piece_id=CWPARDQZXXV46MFGPDYLAFQTPPYXD25IEQ4FIZQLEPJQK6HVKB3A, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.272821340 (602h5m45.143904386s) [noise=true , address=45.91.116.14:4189 , piece_id=EN6UQV4BZ4M635TZ6HSSN45UW65LLBKMPU6EMZXGNQLHR3MARUGQ, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.274412831 (602h5m45.145495876s) [noise=true , address=145.79.227.35:4482 , piece_id=WWZLMRQNPXAW53HCZZP5I2NAQM66OXRE2IATP366UJFZHCJMI2VQ, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:46.959387699 (602h5m46.830470735s) [upload=527721]: Next returned nil segment
Dec 30 17:34:46.959401716 (602h5m46.830484751s) [upload=527721]: Waiting for error group managing segments...
Dec 30 17:34:47.233771963 (602h5m47.104854998s) [noise=true , address=145.79.227.45:4230 , piece_id=X2ROAPLIHYN63DNINPSWV2YLZILWFMJV3P3ASEI5PDIFRXOJJJEQ, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.248691974 (602h5m47.119775009s) [noise=true , address=145.79.227.11:4731 , piece_id=PFDYMYBZVHSHRVX63RJXKDCKGWNSXKDOFMORFPXO564HQ2SU4JPQ, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.252485076 (602h5m47.123568111s) [noise=true , address=145.79.227.27:4691 , piece_id=IGC6IL4673U7NUHOH26UKY6TZUTOQZLBDSKL4JXV3MNTLCXCOK7Q, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.261360666 (602h5m47.132443702s) [noise=true , address=145.79.227.16:4910 , piece_id=XSJ4YM7I3BIIG3O3X3LK7HSVQKL5AA3Q4ZLWMJYQDM223AFQUFWA, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.265225967 (602h5m47.136309012s) [noise=true , address=145.79.227.23:4546 , piece_id=D2P5BLURS4DA47VFEDTIJ3EVNVXKC5YMDBDOHVBWMAICAOVDXMKQ, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.272045612 (602h5m47.143128648s) [noise=true , address=145.79.227.50:4373 , piece_id=G37TCLHWAN4YWBPADLY3SBOTCCYAICFDQGK7WIYTGWZIKRSAJM3A, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.279425644 (602h5m47.150508679s) [noise=true , address=145.79.227.9:4366 , piece_id=65KRZWTXUK5CT6Q5334FXBIKDVYZ6GNF4AWIUKFFSS5MFNPHOWFQ, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.284168746 (602h5m47.155251781s) [noise=true , address=145.79.227.21:4473 , piece_id=2FH5XZNFRXSWKQOLJ2ZAL26Y3KMWZZBQL2RTEBQSARKJFSEXQXFA, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.284835917 (602h5m47.155918962s) [noise=true , address=145.79.227.31:4799 , piece_id=KUELWFEJRZ5AKH4ANMBA4VWSABRKKHEJ5QF62LMPDX5PIO44Z5DA, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.288884154 (602h5m47.159967189s) [noise=true , address=145.79.227.22:4510 , piece_id=XWR5JQBOOU5FF226G6JY266ZE5OENBAJJ2P353UOI3QSZGRBQUIQ, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.326381835 (602h5m47.19746487s) [noise=true , address=145.79.227.35:4482 , piece_id=WWZLMRQNPXAW53HCZZP5I2NAQM66OXRE2IATP366UJFZHCJMI2VQ, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.583738587 (602h5m47.454821632s) [noise=true , address=45.91.116.9:4262 , piece_id=GXQMXHK6DSHZ6NBITKSPWLCCCRF5SUYI5VB7IO6FIMAD2FKYNVWA, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.625677305 (602h5m47.49676035s) [noise=true , address=45.91.116.8:4045 , piece_id=Q3RGXD2UR4ERW4KELMX4N7HSD5AP4IVLIVWS7UYL72WM3AIS2UIA, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.625694757 (602h5m47.496777792s) [seg_pos={0 0}, upload=527721]: Segment reached optimal threshold of 13 pieces.
Dec 30 17:34:47.625981503 (602h5m47.497064548s) [noise=true , address=45.91.116.14:4189 , piece_id=EN6UQV4BZ4M635TZ6HSSN45UW65LLBKMPU6EMZXGNQLHR3MARUGQ, seg_pos={0 0}, upload=527721]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF
Dec 30 17:34:47.625991513 (602h5m47.497074548s) [noise=true , address=45.91.116.14:4189 , piece_id=EN6UQV4BZ4M635TZ6HSSN45UW65LLBKMPU6EMZXGNQLHR3MARUGQ, seg_pos={0 0}, upload=527721]: Long tail cancellation due to optimal threshold
Dec 30 17:34:47.626166286 (602h5m47.497249331s) [noise=true , address=45.91.116.12:4225 , piece_id=SUUGK5PEBWJUTZOWYIYMSJRFBWGT3D7VWNYG76T6HUQF6F2YB2IQ, seg_pos={0 0}, upload=527721]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF
Dec 30 17:34:47.626174451 (602h5m47.497257486s) [noise=true , address=45.91.116.12:4225 , piece_id=SUUGK5PEBWJUTZOWYIYMSJRFBWGT3D7VWNYG76T6HUQF6F2YB2IQ, seg_pos={0 0}, upload=527721]: Long tail cancellation due to optimal threshold
Dec 30 17:34:47.626257829 (602h5m47.497340864s) [noise=true , address=45.91.116.11:4117 , piece_id=CWPARDQZXXV46MFGPDYLAFQTPPYXD25IEQ4FIZQLEPJQK6HVKB3A, seg_pos={0 0}, upload=527721]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Dec 30 17:34:47.626267428 (602h5m47.497350463s) [noise=true , address=45.91.116.11:4117 , piece_id=CWPARDQZXXV46MFGPDYLAFQTPPYXD25IEQ4FIZQLEPJQK6HVKB3A, seg_pos={0 0}, upload=527721]: Long tail cancellation due to optimal threshold
Dec 30 17:34:47.626304649 (602h5m47.497387694s) [seg_pos={0 0}, upload=527721]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 17:34:47.626426300 (602h5m47.497509335s) [upload=527721]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 17:34:47.626433895 (602h5m47.49751693s) [upload=527721]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 17:34:47.916655325 (602h5m47.78773838s) [upload=527721]: Done uploading segments...
Dec 30 17:34:47.916680373 (602h5m47.787763408s) [upload=527721]: Upload finished. err: <nil>
Duration: 5.927758462s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [................X...X..X............................XX......] duration:
6.757281402s trace:
https://jaeger.dev.storj.io/trace/4173ea2fcd946b89
FAIL (2025-12-30 17:38:01 UTC)
orbiter warning: context deadline exceeded
Duration: 6.757368096s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
12 out of 60 is failed (threshold=6) [.............X..X....X.XX..X..X......X.........X..XXX.......] duration:
3.587248613s trace:
https://jaeger.dev.storj.io/trace/56692ebd5ab65c7c
FAIL (2025-12-30 17:38:00 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.587278912s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
27 out of 60 is failed (threshold=3) [....X.X..XX.X.X.X....X.XXXXX.XX.X.XXX......XX.XXX...XX..X...] duration:
5.822295424s trace:
https://jaeger.dev.storj.io/trace/3a1f7d23a1ef3354
FAIL (2025-12-30 17:37:47 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 30 17:34:45.008933692 (602h5m44.880016738s) [upload=527721]: Starting upload
Dec 30 17:34:45.009013564 (602h5m44.880096609s) [upload=527721]: Done starting upload
Dec 30 17:34:45.009131229 (602h5m44.880214274s) [upload=527721]: Uploading segments...
Dec 30 17:34:45.009254303 (602h5m44.880337348s) [upload=527721]: Got next segment. Inline: false
Dec 30 17:34:45.009307504 (602h5m44.88039056s) [upload=527721]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 17:34:45.009314247 (602h5m44.880397292s) [upload=527721]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 17:34:45.260909008 (602h5m45.131992073s) [seg_pos={0 0}, upload=527721]: Begin upload segment...
Dec 30 17:34:45.260991535 (602h5m45.13207458s) [seg_pos={0 0}, upload=527721]: Done begin upload segment.
Dec 30 17:34:45.261025630 (602h5m45.132108675s) [noise=true , address=45.91.116.9:4262 , piece_id=GXQMXHK6DSHZ6NBITKSPWLCCCRF5SUYI5VB7IO6FIMAD2FKYNVWA, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.261066187 (602h5m45.132149232s) [noise=true , address=145.79.227.50:4373 , piece_id=G37TCLHWAN4YWBPADLY3SBOTCCYAICFDQGK7WIYTGWZIKRSAJM3A, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.261501726 (602h5m45.132584761s) [noise=true , address=145.79.227.45:4230 , piece_id=X2ROAPLIHYN63DNINPSWV2YLZILWFMJV3P3ASEI5PDIFRXOJJJEQ, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.261952074 (602h5m45.133035119s) [noise=true , address=45.91.116.8:4045 , piece_id=Q3RGXD2UR4ERW4KELMX4N7HSD5AP4IVLIVWS7UYL72WM3AIS2UIA, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.262315346 (602h5m45.133398381s) [noise=true , address=145.79.227.27:4691 , piece_id=IGC6IL4673U7NUHOH26UKY6TZUTOQZLBDSKL4JXV3MNTLCXCOK7Q, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.262584518 (602h5m45.133667563s) [noise=true , address=145.79.227.9:4366 , piece_id=65KRZWTXUK5CT6Q5334FXBIKDVYZ6GNF4AWIUKFFSS5MFNPHOWFQ, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.262907303 (602h5m45.133990348s) [noise=true , address=145.79.227.31:4799 , piece_id=KUELWFEJRZ5AKH4ANMBA4VWSABRKKHEJ5QF62LMPDX5PIO44Z5DA, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.263203096 (602h5m45.134286141s) [noise=true , address=145.79.227.23:4546 , piece_id=D2P5BLURS4DA47VFEDTIJ3EVNVXKC5YMDBDOHVBWMAICAOVDXMKQ, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.263626272 (602h5m45.134709317s) [noise=true , address=145.79.227.11:4731 , piece_id=PFDYMYBZVHSHRVX63RJXKDCKGWNSXKDOFMORFPXO564HQ2SU4JPQ, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.265075592 (602h5m45.136158637s) [noise=true , address=45.91.116.12:4225 , piece_id=SUUGK5PEBWJUTZOWYIYMSJRFBWGT3D7VWNYG76T6HUQF6F2YB2IQ, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.266166759 (602h5m45.137249804s) [noise=true , address=145.79.227.22:4510 , piece_id=XWR5JQBOOU5FF226G6JY266ZE5OENBAJJ2P353UOI3QSZGRBQUIQ, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.267379789 (602h5m45.138462824s) [noise=true , address=145.79.227.21:4473 , piece_id=2FH5XZNFRXSWKQOLJ2ZAL26Y3KMWZZBQL2RTEBQSARKJFSEXQXFA, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.268484482 (602h5m45.139567527s) [noise=true , address=145.79.227.16:4910 , piece_id=XSJ4YM7I3BIIG3O3X3LK7HSVQKL5AA3Q4ZLWMJYQDM223AFQUFWA, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.269628030 (602h5m45.140711075s) [noise=true , address=45.91.116.11:4117 , piece_id=CWPARDQZXXV46MFGPDYLAFQTPPYXD25IEQ4FIZQLEPJQK6HVKB3A, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.272821340 (602h5m45.143904386s) [noise=true , address=45.91.116.14:4189 , piece_id=EN6UQV4BZ4M635TZ6HSSN45UW65LLBKMPU6EMZXGNQLHR3MARUGQ, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.274412831 (602h5m45.145495876s) [noise=true , address=145.79.227.35:4482 , piece_id=WWZLMRQNPXAW53HCZZP5I2NAQM66OXRE2IATP366UJFZHCJMI2VQ, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:46.959387699 (602h5m46.830470735s) [upload=527721]: Next returned nil segment
Dec 30 17:34:46.959401716 (602h5m46.830484751s) [upload=527721]: Waiting for error group managing segments...
Dec 30 17:34:47.233771963 (602h5m47.104854998s) [noise=true , address=145.79.227.45:4230 , piece_id=X2ROAPLIHYN63DNINPSWV2YLZILWFMJV3P3ASEI5PDIFRXOJJJEQ, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.248691974 (602h5m47.119775009s) [noise=true , address=145.79.227.11:4731 , piece_id=PFDYMYBZVHSHRVX63RJXKDCKGWNSXKDOFMORFPXO564HQ2SU4JPQ, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.252485076 (602h5m47.123568111s) [noise=true , address=145.79.227.27:4691 , piece_id=IGC6IL4673U7NUHOH26UKY6TZUTOQZLBDSKL4JXV3MNTLCXCOK7Q, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.261360666 (602h5m47.132443702s) [noise=true , address=145.79.227.16:4910 , piece_id=XSJ4YM7I3BIIG3O3X3LK7HSVQKL5AA3Q4ZLWMJYQDM223AFQUFWA, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.265225967 (602h5m47.136309012s) [noise=true , address=145.79.227.23:4546 , piece_id=D2P5BLURS4DA47VFEDTIJ3EVNVXKC5YMDBDOHVBWMAICAOVDXMKQ, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.272045612 (602h5m47.143128648s) [noise=true , address=145.79.227.50:4373 , piece_id=G37TCLHWAN4YWBPADLY3SBOTCCYAICFDQGK7WIYTGWZIKRSAJM3A, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.279425644 (602h5m47.150508679s) [noise=true , address=145.79.227.9:4366 , piece_id=65KRZWTXUK5CT6Q5334FXBIKDVYZ6GNF4AWIUKFFSS5MFNPHOWFQ, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.284168746 (602h5m47.155251781s) [noise=true , address=145.79.227.21:4473 , piece_id=2FH5XZNFRXSWKQOLJ2ZAL26Y3KMWZZBQL2RTEBQSARKJFSEXQXFA, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.284835917 (602h5m47.155918962s) [noise=true , address=145.79.227.31:4799 , piece_id=KUELWFEJRZ5AKH4ANMBA4VWSABRKKHEJ5QF62LMPDX5PIO44Z5DA, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.288884154 (602h5m47.159967189s) [noise=true , address=145.79.227.22:4510 , piece_id=XWR5JQBOOU5FF226G6JY266ZE5OENBAJJ2P353UOI3QSZGRBQUIQ, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.326381835 (602h5m47.19746487s) [noise=true , address=145.79.227.35:4482 , piece_id=WWZLMRQNPXAW53HCZZP5I2NAQM66OXRE2IATP366UJFZHCJMI2VQ, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.583738587 (602h5m47.454821632s) [noise=true , address=45.91.116.9:4262 , piece_id=GXQMXHK6DSHZ6NBITKSPWLCCCRF5SUYI5VB7IO6FIMAD2FKYNVWA, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.625677305 (602h5m47.49676035s) [noise=true , address=45.91.116.8:4045 , piece_id=Q3RGXD2UR4ERW4KELMX4N7HSD5AP4IVLIVWS7UYL72WM3AIS2UIA, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.625694757 (602h5m47.496777792s) [seg_pos={0 0}, upload=527721]: Segment reached optimal threshold of 13 pieces.
Dec 30 17:34:47.625981503 (602h5m47.497064548s) [noise=true , address=45.91.116.14:4189 , piece_id=EN6UQV4BZ4M635TZ6HSSN45UW65LLBKMPU6EMZXGNQLHR3MARUGQ, seg_pos={0 0}, upload=527721]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF
Dec 30 17:34:47.625991513 (602h5m47.497074548s) [noise=true , address=45.91.116.14:4189 , piece_id=EN6UQV4BZ4M635TZ6HSSN45UW65LLBKMPU6EMZXGNQLHR3MARUGQ, seg_pos={0 0}, upload=527721]: Long tail cancellation due to optimal threshold
Dec 30 17:34:47.626166286 (602h5m47.497249331s) [noise=true , address=45.91.116.12:4225 , piece_id=SUUGK5PEBWJUTZOWYIYMSJRFBWGT3D7VWNYG76T6HUQF6F2YB2IQ, seg_pos={0 0}, upload=527721]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF
Dec 30 17:34:47.626174451 (602h5m47.497257486s) [noise=true , address=45.91.116.12:4225 , piece_id=SUUGK5PEBWJUTZOWYIYMSJRFBWGT3D7VWNYG76T6HUQF6F2YB2IQ, seg_pos={0 0}, upload=527721]: Long tail cancellation due to optimal threshold
Dec 30 17:34:47.626257829 (602h5m47.497340864s) [noise=true , address=45.91.116.11:4117 , piece_id=CWPARDQZXXV46MFGPDYLAFQTPPYXD25IEQ4FIZQLEPJQK6HVKB3A, seg_pos={0 0}, upload=527721]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Dec 30 17:34:47.626267428 (602h5m47.497350463s) [noise=true , address=45.91.116.11:4117 , piece_id=CWPARDQZXXV46MFGPDYLAFQTPPYXD25IEQ4FIZQLEPJQK6HVKB3A, seg_pos={0 0}, upload=527721]: Long tail cancellation due to optimal threshold
Dec 30 17:34:47.626304649 (602h5m47.497387694s) [seg_pos={0 0}, upload=527721]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 17:34:47.626426300 (602h5m47.497509335s) [upload=527721]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 17:34:47.626433895 (602h5m47.49751693s) [upload=527721]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 17:34:47.916655325 (602h5m47.78773838s) [upload=527721]: Done uploading segments...
Dec 30 17:34:47.916680373 (602h5m47.787763408s) [upload=527721]: Upload finished. err: <nil>
Duration: 5.822331524s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
12 out of 60 is failed (threshold=6) [..............X..X....X.XX..X..X......X.........X..XXX......] duration:
3.552133347s trace:
https://jaeger.dev.storj.io/trace/56692ebd5ab65c7c
FAIL (2025-12-30 17:37:40 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.552176669s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................] duration:
48.177256505s trace:
https://jaeger.dev.storj.io/trace/5ea8d7d4f61e13a2
FAIL (2025-12-30 17:37:36 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK, Piece ID: HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ) context deadline exceeded; piecestore: (Node ID: 12eLKUg2vTnZMrLaraSPEoGXe3jDVBJYyURdcn4f2gCN5pvvCQ8, Piece ID: KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ) context deadline exceeded; piecestore: (Node ID: 12W4n6uf6azQk1QgeR8Xw7UkpnZm19zdyFKwojCsqZt2GFHrpce, Piece ID: DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ) context deadline exceeded
======LOGS======
17.114181822 (601h42m16.985264867s) [noise=true , address=145.79.227.44:4208 , piece_id=3OXDCHZSOIFS5VQCI4SXSAQFICDCYHRE5NGFBERR33KOTGSZWLDQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.114641387 (601h42m16.985724432s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.115652532 (601h42m16.986735578s) [noise=true , address=77.90.31.17:4790 , piece_id=5RSRESKGDA2YF4NBRF4ZFABBKWJK4NO6VLEXW2Z6EY3OPXH2WPNA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.115937134 (601h42m16.987020179s) [noise=true , address=64.49.13.155:4923 , piece_id=MUJD7XLM2YRLRCHYVSDPBE6VBYLHCOA7HJQP22ZSET4N574XQMJA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.116814986 (601h42m16.987898031s) [noise=true , address=45.91.116.11:4139 , piece_id=72DPEWZ7ZPZZ53UGYV2VCUKLCK7HCM22CDBRHWVLJGDGO5B5JADQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.117501182 (601h42m16.988584228s) [noise=true , address=77.90.31.14:4665 , piece_id=DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.118263334 (601h42m16.989346379s) [noise=true , address=145.79.227.40:4084 , piece_id=ZADZDRKBIILLAI4IT335CYETOSFOFUUPQIH6HZJCTHKRTZSPDJUQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.118555850 (601h42m16.989638886s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.118765930 (601h42m16.989849036s) [noise=true , address=77.90.31.21:4905 , piece_id=HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119029533 (601h42m16.990112568s) [noise=true , address=77.90.31.10:4533 , piece_id=QFBBYNQHQL2G7PEREK43FZBRUMKMVNBC2XKTE2NWTVTKN2452LPA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119417672 (601h42m16.990500707s) [noise=true , address=45.91.116.12:4258 , piece_id=KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119441588 (601h42m16.990524643s) [noise=true , address=145.79.227.40:4053 , piece_id=JCSIUMWKEAQNQXTNSJTN6VSB3V2SIDRQK67QMK46B4GOUWEMQE2A, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119763470 (601h42m16.990846505s) [noise=true , address=77.90.31.18:4821 , piece_id=MXZFKXM3JV2HQURDLIU7BKWLAKDZ27O4ER3SGC7QBZV5I3J2EHKA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.120028685 (601h42m16.99111172s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.120086986 (601h42m16.991170031s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.120586507 (601h42m16.991669562s) [noise=true , address=45.91.116.13:4168 , piece_id=AKUE5H3A2UECQOZY3GV3MWZKIHSNKCDOG32DKCNHIRTJROJ74M3A, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.121258768 (601h42m16.992341813s) [noise=true , address=145.79.227.9:4379 , piece_id=QTFXFSGGKYBQSPWJYBBJMQUL3D5EEO3IX4C53ECD7FXMUOGUBC7Q, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:18.568739165 (601h42m18.43982221s) [upload=527370]: Next returned nil segment
Dec 30 17:11:18.568751328 (601h42m18.439834363s) [upload=527370]: Waiting for error group managing segments...
Dec 30 17:11:18.978424178 (601h42m18.849507213s) [noise=true , address=64.49.13.155:4923 , piece_id=MUJD7XLM2YRLRCHYVSDPBE6VBYLHCOA7HJQP22ZSET4N574XQMJA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:18.986258125 (601h42m18.85734116s) [noise=true , address=145.79.227.9:4379 , piece_id=QTFXFSGGKYBQSPWJYBBJMQUL3D5EEO3IX4C53ECD7FXMUOGUBC7Q, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.274519421 (601h42m19.145602456s) [noise=true , address=77.90.31.14:4665 , piece_id=DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.315775438 (601h42m19.186858473s) [noise=true , address=45.91.116.13:4168 , piece_id=AKUE5H3A2UECQOZY3GV3MWZKIHSNKCDOG32DKCNHIRTJROJ74M3A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.335765371 (601h42m19.206848406s) [noise=true , address=77.90.31.21:4905 , piece_id=HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.354812129 (601h42m19.225895164s) [noise=true , address=77.90.31.10:4533 , piece_id=QFBBYNQHQL2G7PEREK43FZBRUMKMVNBC2XKTE2NWTVTKN2452LPA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.392024917 (601h42m19.263107952s) [noise=true , address=45.91.116.11:4139 , piece_id=72DPEWZ7ZPZZ53UGYV2VCUKLCK7HCM22CDBRHWVLJGDGO5B5JADQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.396084327 (601h42m19.267167372s) [noise=true , address=77.90.31.18:4821 , piece_id=MXZFKXM3JV2HQURDLIU7BKWLAKDZ27O4ER3SGC7QBZV5I3J2EHKA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.862099099 (601h42m19.733182134s) [noise=true , address=77.90.31.17:4790 , piece_id=5RSRESKGDA2YF4NBRF4ZFABBKWJK4NO6VLEXW2Z6EY3OPXH2WPNA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.905412913 (601h42m19.776495948s) [noise=true , address=145.79.227.44:4208 , piece_id=3OXDCHZSOIFS5VQCI4SXSAQFICDCYHRE5NGFBERR33KOTGSZWLDQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.910309688 (601h42m19.781392743s) [noise=true , address=145.79.227.30:4794 , piece_id=EJH6Q3T2BMPVLG6F7CW7N4GYOA3EZFPADL6VPT7WKKG65B2WZVXQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.934233361 (601h42m19.805316397s) [noise=true , address=145.79.227.40:4084 , piece_id=ZADZDRKBIILLAI4IT335CYETOSFOFUUPQIH6HZJCTHKRTZSPDJUQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:20.074011527 (601h42m19.945094572s) [noise=true , address=45.91.116.12:4258 , piece_id=KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:20.106010337 (601h42m19.977093372s) [noise=true , address=145.79.227.40:4053 , piece_id=JCSIUMWKEAQNQXTNSJTN6VSB3V2SIDRQK67QMK46B4GOUWEMQE2A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:20.106080911 (601h42m19.977163946s) [seg_pos={0 0}, upload=527370]: Segment reached optimal threshold of 14 pieces.
Dec 30 17:11:20.106274520 (601h42m19.977357565s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106281644 (601h42m19.977364679s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106351607 (601h42m19.977434642s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106367767 (601h42m19.977450802s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106424967 (601h42m19.977508002s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106432451 (601h42m19.977515486s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106540666 (601h42m19.977623702s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106546207 (601h42m19.977629242s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106563380 (601h42m19.977646415s) [seg_pos={0 0}, upload=527370]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 17:11:20.106625529 (601h42m19.977708564s) [upload=527370]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 17:11:20.106629837 (601h42m19.977712882s) [upload=527370]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 17:11:20.406465452 (601h42m20.277548507s) [upload=527370]: Done uploading segments...
Dec 30 17:11:20.406504206 (601h42m20.277587251s) [upload=527370]: Upload finished. err: <nil>
Duration: 48.177287385s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.............X...X..........................................] duration:
5.943837513s trace:
https://jaeger.dev.storj.io/trace/6ea66b1ed6f4c6e5
FAIL (2025-12-30 17:37:32 UTC)
orbiter warning: context deadline exceeded
Duration: 5.943957342s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.................X...X..X............................XX.....] duration:
7.387559866s trace:
https://jaeger.dev.storj.io/trace/4173ea2fcd946b89
FAIL (2025-12-30 17:37:31 UTC)
orbiter warning: context deadline exceeded
Duration: 7.38759391s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
27 out of 60 is failed (threshold=3) [.....X.X..XX.X.X.X....X.XXXXX.XX.X.XXX......XX.XXX...XX..X..] duration:
5.751982826s trace:
https://jaeger.dev.storj.io/trace/3a1f7d23a1ef3354
FAIL (2025-12-30 17:36:55 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 30 17:34:45.008933692 (602h5m44.880016738s) [upload=527721]: Starting upload
Dec 30 17:34:45.009013564 (602h5m44.880096609s) [upload=527721]: Done starting upload
Dec 30 17:34:45.009131229 (602h5m44.880214274s) [upload=527721]: Uploading segments...
Dec 30 17:34:45.009254303 (602h5m44.880337348s) [upload=527721]: Got next segment. Inline: false
Dec 30 17:34:45.009307504 (602h5m44.88039056s) [upload=527721]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 17:34:45.009314247 (602h5m44.880397292s) [upload=527721]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 17:34:45.260909008 (602h5m45.131992073s) [seg_pos={0 0}, upload=527721]: Begin upload segment...
Dec 30 17:34:45.260991535 (602h5m45.13207458s) [seg_pos={0 0}, upload=527721]: Done begin upload segment.
Dec 30 17:34:45.261025630 (602h5m45.132108675s) [noise=true , address=45.91.116.9:4262 , piece_id=GXQMXHK6DSHZ6NBITKSPWLCCCRF5SUYI5VB7IO6FIMAD2FKYNVWA, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.261066187 (602h5m45.132149232s) [noise=true , address=145.79.227.50:4373 , piece_id=G37TCLHWAN4YWBPADLY3SBOTCCYAICFDQGK7WIYTGWZIKRSAJM3A, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.261501726 (602h5m45.132584761s) [noise=true , address=145.79.227.45:4230 , piece_id=X2ROAPLIHYN63DNINPSWV2YLZILWFMJV3P3ASEI5PDIFRXOJJJEQ, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.261952074 (602h5m45.133035119s) [noise=true , address=45.91.116.8:4045 , piece_id=Q3RGXD2UR4ERW4KELMX4N7HSD5AP4IVLIVWS7UYL72WM3AIS2UIA, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.262315346 (602h5m45.133398381s) [noise=true , address=145.79.227.27:4691 , piece_id=IGC6IL4673U7NUHOH26UKY6TZUTOQZLBDSKL4JXV3MNTLCXCOK7Q, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.262584518 (602h5m45.133667563s) [noise=true , address=145.79.227.9:4366 , piece_id=65KRZWTXUK5CT6Q5334FXBIKDVYZ6GNF4AWIUKFFSS5MFNPHOWFQ, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.262907303 (602h5m45.133990348s) [noise=true , address=145.79.227.31:4799 , piece_id=KUELWFEJRZ5AKH4ANMBA4VWSABRKKHEJ5QF62LMPDX5PIO44Z5DA, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.263203096 (602h5m45.134286141s) [noise=true , address=145.79.227.23:4546 , piece_id=D2P5BLURS4DA47VFEDTIJ3EVNVXKC5YMDBDOHVBWMAICAOVDXMKQ, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.263626272 (602h5m45.134709317s) [noise=true , address=145.79.227.11:4731 , piece_id=PFDYMYBZVHSHRVX63RJXKDCKGWNSXKDOFMORFPXO564HQ2SU4JPQ, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.265075592 (602h5m45.136158637s) [noise=true , address=45.91.116.12:4225 , piece_id=SUUGK5PEBWJUTZOWYIYMSJRFBWGT3D7VWNYG76T6HUQF6F2YB2IQ, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.266166759 (602h5m45.137249804s) [noise=true , address=145.79.227.22:4510 , piece_id=XWR5JQBOOU5FF226G6JY266ZE5OENBAJJ2P353UOI3QSZGRBQUIQ, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.267379789 (602h5m45.138462824s) [noise=true , address=145.79.227.21:4473 , piece_id=2FH5XZNFRXSWKQOLJ2ZAL26Y3KMWZZBQL2RTEBQSARKJFSEXQXFA, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.268484482 (602h5m45.139567527s) [noise=true , address=145.79.227.16:4910 , piece_id=XSJ4YM7I3BIIG3O3X3LK7HSVQKL5AA3Q4ZLWMJYQDM223AFQUFWA, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.269628030 (602h5m45.140711075s) [noise=true , address=45.91.116.11:4117 , piece_id=CWPARDQZXXV46MFGPDYLAFQTPPYXD25IEQ4FIZQLEPJQK6HVKB3A, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.272821340 (602h5m45.143904386s) [noise=true , address=45.91.116.14:4189 , piece_id=EN6UQV4BZ4M635TZ6HSSN45UW65LLBKMPU6EMZXGNQLHR3MARUGQ, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.274412831 (602h5m45.145495876s) [noise=true , address=145.79.227.35:4482 , piece_id=WWZLMRQNPXAW53HCZZP5I2NAQM66OXRE2IATP366UJFZHCJMI2VQ, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:46.959387699 (602h5m46.830470735s) [upload=527721]: Next returned nil segment
Dec 30 17:34:46.959401716 (602h5m46.830484751s) [upload=527721]: Waiting for error group managing segments...
Dec 30 17:34:47.233771963 (602h5m47.104854998s) [noise=true , address=145.79.227.45:4230 , piece_id=X2ROAPLIHYN63DNINPSWV2YLZILWFMJV3P3ASEI5PDIFRXOJJJEQ, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.248691974 (602h5m47.119775009s) [noise=true , address=145.79.227.11:4731 , piece_id=PFDYMYBZVHSHRVX63RJXKDCKGWNSXKDOFMORFPXO564HQ2SU4JPQ, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.252485076 (602h5m47.123568111s) [noise=true , address=145.79.227.27:4691 , piece_id=IGC6IL4673U7NUHOH26UKY6TZUTOQZLBDSKL4JXV3MNTLCXCOK7Q, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.261360666 (602h5m47.132443702s) [noise=true , address=145.79.227.16:4910 , piece_id=XSJ4YM7I3BIIG3O3X3LK7HSVQKL5AA3Q4ZLWMJYQDM223AFQUFWA, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.265225967 (602h5m47.136309012s) [noise=true , address=145.79.227.23:4546 , piece_id=D2P5BLURS4DA47VFEDTIJ3EVNVXKC5YMDBDOHVBWMAICAOVDXMKQ, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.272045612 (602h5m47.143128648s) [noise=true , address=145.79.227.50:4373 , piece_id=G37TCLHWAN4YWBPADLY3SBOTCCYAICFDQGK7WIYTGWZIKRSAJM3A, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.279425644 (602h5m47.150508679s) [noise=true , address=145.79.227.9:4366 , piece_id=65KRZWTXUK5CT6Q5334FXBIKDVYZ6GNF4AWIUKFFSS5MFNPHOWFQ, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.284168746 (602h5m47.155251781s) [noise=true , address=145.79.227.21:4473 , piece_id=2FH5XZNFRXSWKQOLJ2ZAL26Y3KMWZZBQL2RTEBQSARKJFSEXQXFA, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.284835917 (602h5m47.155918962s) [noise=true , address=145.79.227.31:4799 , piece_id=KUELWFEJRZ5AKH4ANMBA4VWSABRKKHEJ5QF62LMPDX5PIO44Z5DA, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.288884154 (602h5m47.159967189s) [noise=true , address=145.79.227.22:4510 , piece_id=XWR5JQBOOU5FF226G6JY266ZE5OENBAJJ2P353UOI3QSZGRBQUIQ, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.326381835 (602h5m47.19746487s) [noise=true , address=145.79.227.35:4482 , piece_id=WWZLMRQNPXAW53HCZZP5I2NAQM66OXRE2IATP366UJFZHCJMI2VQ, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.583738587 (602h5m47.454821632s) [noise=true , address=45.91.116.9:4262 , piece_id=GXQMXHK6DSHZ6NBITKSPWLCCCRF5SUYI5VB7IO6FIMAD2FKYNVWA, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.625677305 (602h5m47.49676035s) [noise=true , address=45.91.116.8:4045 , piece_id=Q3RGXD2UR4ERW4KELMX4N7HSD5AP4IVLIVWS7UYL72WM3AIS2UIA, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.625694757 (602h5m47.496777792s) [seg_pos={0 0}, upload=527721]: Segment reached optimal threshold of 13 pieces.
Dec 30 17:34:47.625981503 (602h5m47.497064548s) [noise=true , address=45.91.116.14:4189 , piece_id=EN6UQV4BZ4M635TZ6HSSN45UW65LLBKMPU6EMZXGNQLHR3MARUGQ, seg_pos={0 0}, upload=527721]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF
Dec 30 17:34:47.625991513 (602h5m47.497074548s) [noise=true , address=45.91.116.14:4189 , piece_id=EN6UQV4BZ4M635TZ6HSSN45UW65LLBKMPU6EMZXGNQLHR3MARUGQ, seg_pos={0 0}, upload=527721]: Long tail cancellation due to optimal threshold
Dec 30 17:34:47.626166286 (602h5m47.497249331s) [noise=true , address=45.91.116.12:4225 , piece_id=SUUGK5PEBWJUTZOWYIYMSJRFBWGT3D7VWNYG76T6HUQF6F2YB2IQ, seg_pos={0 0}, upload=527721]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF
Dec 30 17:34:47.626174451 (602h5m47.497257486s) [noise=true , address=45.91.116.12:4225 , piece_id=SUUGK5PEBWJUTZOWYIYMSJRFBWGT3D7VWNYG76T6HUQF6F2YB2IQ, seg_pos={0 0}, upload=527721]: Long tail cancellation due to optimal threshold
Dec 30 17:34:47.626257829 (602h5m47.497340864s) [noise=true , address=45.91.116.11:4117 , piece_id=CWPARDQZXXV46MFGPDYLAFQTPPYXD25IEQ4FIZQLEPJQK6HVKB3A, seg_pos={0 0}, upload=527721]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Dec 30 17:34:47.626267428 (602h5m47.497350463s) [noise=true , address=45.91.116.11:4117 , piece_id=CWPARDQZXXV46MFGPDYLAFQTPPYXD25IEQ4FIZQLEPJQK6HVKB3A, seg_pos={0 0}, upload=527721]: Long tail cancellation due to optimal threshold
Dec 30 17:34:47.626304649 (602h5m47.497387694s) [seg_pos={0 0}, upload=527721]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 17:34:47.626426300 (602h5m47.497509335s) [upload=527721]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 17:34:47.626433895 (602h5m47.49751693s) [upload=527721]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 17:34:47.916655325 (602h5m47.78773838s) [upload=527721]: Done uploading segments...
Dec 30 17:34:47.916680373 (602h5m47.787763408s) [upload=527721]: Upload finished. err: <nil>
Duration: 5.752007303s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
12 out of 60 is failed (threshold=6) [...............X..X....X.XX..X..X......X.........X..XXX.....] duration:
3.491527828s trace:
https://jaeger.dev.storj.io/trace/56692ebd5ab65c7c
FAIL (2025-12-30 17:36:49 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.491560138s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [..................X...X..X............................XX....] duration:
6.006758603s trace:
https://jaeger.dev.storj.io/trace/4173ea2fcd946b89
FAIL (2025-12-30 17:36:39 UTC)
orbiter warning: context deadline exceeded
Duration: 6.006785705s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..............X...X.........................................] duration:
6.339136287s trace:
https://jaeger.dev.storj.io/trace/6ea66b1ed6f4c6e5
FAIL (2025-12-30 17:36:31 UTC)
orbiter warning: context deadline exceeded
Duration: 6.33916362s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................] duration:
37.48892344s trace:
https://jaeger.dev.storj.io/trace/5ea8d7d4f61e13a2
FAIL (2025-12-30 17:36:16 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK, Piece ID: HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ) context deadline exceeded; piecestore: (Node ID: 12eLKUg2vTnZMrLaraSPEoGXe3jDVBJYyURdcn4f2gCN5pvvCQ8, Piece ID: KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ) context deadline exceeded; piecestore: (Node ID: 12W4n6uf6azQk1QgeR8Xw7UkpnZm19zdyFKwojCsqZt2GFHrpce, Piece ID: DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ) context deadline exceeded
======LOGS======
17.114181822 (601h42m16.985264867s) [noise=true , address=145.79.227.44:4208 , piece_id=3OXDCHZSOIFS5VQCI4SXSAQFICDCYHRE5NGFBERR33KOTGSZWLDQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.114641387 (601h42m16.985724432s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.115652532 (601h42m16.986735578s) [noise=true , address=77.90.31.17:4790 , piece_id=5RSRESKGDA2YF4NBRF4ZFABBKWJK4NO6VLEXW2Z6EY3OPXH2WPNA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.115937134 (601h42m16.987020179s) [noise=true , address=64.49.13.155:4923 , piece_id=MUJD7XLM2YRLRCHYVSDPBE6VBYLHCOA7HJQP22ZSET4N574XQMJA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.116814986 (601h42m16.987898031s) [noise=true , address=45.91.116.11:4139 , piece_id=72DPEWZ7ZPZZ53UGYV2VCUKLCK7HCM22CDBRHWVLJGDGO5B5JADQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.117501182 (601h42m16.988584228s) [noise=true , address=77.90.31.14:4665 , piece_id=DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.118263334 (601h42m16.989346379s) [noise=true , address=145.79.227.40:4084 , piece_id=ZADZDRKBIILLAI4IT335CYETOSFOFUUPQIH6HZJCTHKRTZSPDJUQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.118555850 (601h42m16.989638886s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.118765930 (601h42m16.989849036s) [noise=true , address=77.90.31.21:4905 , piece_id=HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119029533 (601h42m16.990112568s) [noise=true , address=77.90.31.10:4533 , piece_id=QFBBYNQHQL2G7PEREK43FZBRUMKMVNBC2XKTE2NWTVTKN2452LPA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119417672 (601h42m16.990500707s) [noise=true , address=45.91.116.12:4258 , piece_id=KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119441588 (601h42m16.990524643s) [noise=true , address=145.79.227.40:4053 , piece_id=JCSIUMWKEAQNQXTNSJTN6VSB3V2SIDRQK67QMK46B4GOUWEMQE2A, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119763470 (601h42m16.990846505s) [noise=true , address=77.90.31.18:4821 , piece_id=MXZFKXM3JV2HQURDLIU7BKWLAKDZ27O4ER3SGC7QBZV5I3J2EHKA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.120028685 (601h42m16.99111172s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.120086986 (601h42m16.991170031s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.120586507 (601h42m16.991669562s) [noise=true , address=45.91.116.13:4168 , piece_id=AKUE5H3A2UECQOZY3GV3MWZKIHSNKCDOG32DKCNHIRTJROJ74M3A, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.121258768 (601h42m16.992341813s) [noise=true , address=145.79.227.9:4379 , piece_id=QTFXFSGGKYBQSPWJYBBJMQUL3D5EEO3IX4C53ECD7FXMUOGUBC7Q, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:18.568739165 (601h42m18.43982221s) [upload=527370]: Next returned nil segment
Dec 30 17:11:18.568751328 (601h42m18.439834363s) [upload=527370]: Waiting for error group managing segments...
Dec 30 17:11:18.978424178 (601h42m18.849507213s) [noise=true , address=64.49.13.155:4923 , piece_id=MUJD7XLM2YRLRCHYVSDPBE6VBYLHCOA7HJQP22ZSET4N574XQMJA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:18.986258125 (601h42m18.85734116s) [noise=true , address=145.79.227.9:4379 , piece_id=QTFXFSGGKYBQSPWJYBBJMQUL3D5EEO3IX4C53ECD7FXMUOGUBC7Q, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.274519421 (601h42m19.145602456s) [noise=true , address=77.90.31.14:4665 , piece_id=DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.315775438 (601h42m19.186858473s) [noise=true , address=45.91.116.13:4168 , piece_id=AKUE5H3A2UECQOZY3GV3MWZKIHSNKCDOG32DKCNHIRTJROJ74M3A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.335765371 (601h42m19.206848406s) [noise=true , address=77.90.31.21:4905 , piece_id=HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.354812129 (601h42m19.225895164s) [noise=true , address=77.90.31.10:4533 , piece_id=QFBBYNQHQL2G7PEREK43FZBRUMKMVNBC2XKTE2NWTVTKN2452LPA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.392024917 (601h42m19.263107952s) [noise=true , address=45.91.116.11:4139 , piece_id=72DPEWZ7ZPZZ53UGYV2VCUKLCK7HCM22CDBRHWVLJGDGO5B5JADQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.396084327 (601h42m19.267167372s) [noise=true , address=77.90.31.18:4821 , piece_id=MXZFKXM3JV2HQURDLIU7BKWLAKDZ27O4ER3SGC7QBZV5I3J2EHKA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.862099099 (601h42m19.733182134s) [noise=true , address=77.90.31.17:4790 , piece_id=5RSRESKGDA2YF4NBRF4ZFABBKWJK4NO6VLEXW2Z6EY3OPXH2WPNA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.905412913 (601h42m19.776495948s) [noise=true , address=145.79.227.44:4208 , piece_id=3OXDCHZSOIFS5VQCI4SXSAQFICDCYHRE5NGFBERR33KOTGSZWLDQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.910309688 (601h42m19.781392743s) [noise=true , address=145.79.227.30:4794 , piece_id=EJH6Q3T2BMPVLG6F7CW7N4GYOA3EZFPADL6VPT7WKKG65B2WZVXQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.934233361 (601h42m19.805316397s) [noise=true , address=145.79.227.40:4084 , piece_id=ZADZDRKBIILLAI4IT335CYETOSFOFUUPQIH6HZJCTHKRTZSPDJUQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:20.074011527 (601h42m19.945094572s) [noise=true , address=45.91.116.12:4258 , piece_id=KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:20.106010337 (601h42m19.977093372s) [noise=true , address=145.79.227.40:4053 , piece_id=JCSIUMWKEAQNQXTNSJTN6VSB3V2SIDRQK67QMK46B4GOUWEMQE2A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:20.106080911 (601h42m19.977163946s) [seg_pos={0 0}, upload=527370]: Segment reached optimal threshold of 14 pieces.
Dec 30 17:11:20.106274520 (601h42m19.977357565s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106281644 (601h42m19.977364679s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106351607 (601h42m19.977434642s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106367767 (601h42m19.977450802s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106424967 (601h42m19.977508002s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106432451 (601h42m19.977515486s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106540666 (601h42m19.977623702s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106546207 (601h42m19.977629242s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106563380 (601h42m19.977646415s) [seg_pos={0 0}, upload=527370]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 17:11:20.106625529 (601h42m19.977708564s) [upload=527370]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 17:11:20.106629837 (601h42m19.977712882s) [upload=527370]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 17:11:20.406465452 (601h42m20.277548507s) [upload=527370]: Done uploading segments...
Dec 30 17:11:20.406504206 (601h42m20.277587251s) [upload=527370]: Upload finished. err: <nil>
Duration: 37.488967734s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
12 out of 60 is failed (threshold=6) [................X..X....X.XX..X..X......X.........X..XXX....] duration:
3.682830941s trace:
https://jaeger.dev.storj.io/trace/56692ebd5ab65c7c
FAIL (2025-12-30 17:36:01 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.682860607s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
27 out of 60 is failed (threshold=3) [......X.X..XX.X.X.X....X.XXXXX.XX.X.XXX......XX.XXX...XX..X.] duration:
6.049611494s trace:
https://jaeger.dev.storj.io/trace/3a1f7d23a1ef3354
FAIL (2025-12-30 17:35:58 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 30 17:34:45.008933692 (602h5m44.880016738s) [upload=527721]: Starting upload
Dec 30 17:34:45.009013564 (602h5m44.880096609s) [upload=527721]: Done starting upload
Dec 30 17:34:45.009131229 (602h5m44.880214274s) [upload=527721]: Uploading segments...
Dec 30 17:34:45.009254303 (602h5m44.880337348s) [upload=527721]: Got next segment. Inline: false
Dec 30 17:34:45.009307504 (602h5m44.88039056s) [upload=527721]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 17:34:45.009314247 (602h5m44.880397292s) [upload=527721]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 17:34:45.260909008 (602h5m45.131992073s) [seg_pos={0 0}, upload=527721]: Begin upload segment...
Dec 30 17:34:45.260991535 (602h5m45.13207458s) [seg_pos={0 0}, upload=527721]: Done begin upload segment.
Dec 30 17:34:45.261025630 (602h5m45.132108675s) [noise=true , address=45.91.116.9:4262 , piece_id=GXQMXHK6DSHZ6NBITKSPWLCCCRF5SUYI5VB7IO6FIMAD2FKYNVWA, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.261066187 (602h5m45.132149232s) [noise=true , address=145.79.227.50:4373 , piece_id=G37TCLHWAN4YWBPADLY3SBOTCCYAICFDQGK7WIYTGWZIKRSAJM3A, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.261501726 (602h5m45.132584761s) [noise=true , address=145.79.227.45:4230 , piece_id=X2ROAPLIHYN63DNINPSWV2YLZILWFMJV3P3ASEI5PDIFRXOJJJEQ, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.261952074 (602h5m45.133035119s) [noise=true , address=45.91.116.8:4045 , piece_id=Q3RGXD2UR4ERW4KELMX4N7HSD5AP4IVLIVWS7UYL72WM3AIS2UIA, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.262315346 (602h5m45.133398381s) [noise=true , address=145.79.227.27:4691 , piece_id=IGC6IL4673U7NUHOH26UKY6TZUTOQZLBDSKL4JXV3MNTLCXCOK7Q, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.262584518 (602h5m45.133667563s) [noise=true , address=145.79.227.9:4366 , piece_id=65KRZWTXUK5CT6Q5334FXBIKDVYZ6GNF4AWIUKFFSS5MFNPHOWFQ, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.262907303 (602h5m45.133990348s) [noise=true , address=145.79.227.31:4799 , piece_id=KUELWFEJRZ5AKH4ANMBA4VWSABRKKHEJ5QF62LMPDX5PIO44Z5DA, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.263203096 (602h5m45.134286141s) [noise=true , address=145.79.227.23:4546 , piece_id=D2P5BLURS4DA47VFEDTIJ3EVNVXKC5YMDBDOHVBWMAICAOVDXMKQ, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.263626272 (602h5m45.134709317s) [noise=true , address=145.79.227.11:4731 , piece_id=PFDYMYBZVHSHRVX63RJXKDCKGWNSXKDOFMORFPXO564HQ2SU4JPQ, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.265075592 (602h5m45.136158637s) [noise=true , address=45.91.116.12:4225 , piece_id=SUUGK5PEBWJUTZOWYIYMSJRFBWGT3D7VWNYG76T6HUQF6F2YB2IQ, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.266166759 (602h5m45.137249804s) [noise=true , address=145.79.227.22:4510 , piece_id=XWR5JQBOOU5FF226G6JY266ZE5OENBAJJ2P353UOI3QSZGRBQUIQ, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.267379789 (602h5m45.138462824s) [noise=true , address=145.79.227.21:4473 , piece_id=2FH5XZNFRXSWKQOLJ2ZAL26Y3KMWZZBQL2RTEBQSARKJFSEXQXFA, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.268484482 (602h5m45.139567527s) [noise=true , address=145.79.227.16:4910 , piece_id=XSJ4YM7I3BIIG3O3X3LK7HSVQKL5AA3Q4ZLWMJYQDM223AFQUFWA, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.269628030 (602h5m45.140711075s) [noise=true , address=45.91.116.11:4117 , piece_id=CWPARDQZXXV46MFGPDYLAFQTPPYXD25IEQ4FIZQLEPJQK6HVKB3A, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.272821340 (602h5m45.143904386s) [noise=true , address=45.91.116.14:4189 , piece_id=EN6UQV4BZ4M635TZ6HSSN45UW65LLBKMPU6EMZXGNQLHR3MARUGQ, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.274412831 (602h5m45.145495876s) [noise=true , address=145.79.227.35:4482 , piece_id=WWZLMRQNPXAW53HCZZP5I2NAQM66OXRE2IATP366UJFZHCJMI2VQ, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:46.959387699 (602h5m46.830470735s) [upload=527721]: Next returned nil segment
Dec 30 17:34:46.959401716 (602h5m46.830484751s) [upload=527721]: Waiting for error group managing segments...
Dec 30 17:34:47.233771963 (602h5m47.104854998s) [noise=true , address=145.79.227.45:4230 , piece_id=X2ROAPLIHYN63DNINPSWV2YLZILWFMJV3P3ASEI5PDIFRXOJJJEQ, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.248691974 (602h5m47.119775009s) [noise=true , address=145.79.227.11:4731 , piece_id=PFDYMYBZVHSHRVX63RJXKDCKGWNSXKDOFMORFPXO564HQ2SU4JPQ, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.252485076 (602h5m47.123568111s) [noise=true , address=145.79.227.27:4691 , piece_id=IGC6IL4673U7NUHOH26UKY6TZUTOQZLBDSKL4JXV3MNTLCXCOK7Q, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.261360666 (602h5m47.132443702s) [noise=true , address=145.79.227.16:4910 , piece_id=XSJ4YM7I3BIIG3O3X3LK7HSVQKL5AA3Q4ZLWMJYQDM223AFQUFWA, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.265225967 (602h5m47.136309012s) [noise=true , address=145.79.227.23:4546 , piece_id=D2P5BLURS4DA47VFEDTIJ3EVNVXKC5YMDBDOHVBWMAICAOVDXMKQ, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.272045612 (602h5m47.143128648s) [noise=true , address=145.79.227.50:4373 , piece_id=G37TCLHWAN4YWBPADLY3SBOTCCYAICFDQGK7WIYTGWZIKRSAJM3A, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.279425644 (602h5m47.150508679s) [noise=true , address=145.79.227.9:4366 , piece_id=65KRZWTXUK5CT6Q5334FXBIKDVYZ6GNF4AWIUKFFSS5MFNPHOWFQ, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.284168746 (602h5m47.155251781s) [noise=true , address=145.79.227.21:4473 , piece_id=2FH5XZNFRXSWKQOLJ2ZAL26Y3KMWZZBQL2RTEBQSARKJFSEXQXFA, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.284835917 (602h5m47.155918962s) [noise=true , address=145.79.227.31:4799 , piece_id=KUELWFEJRZ5AKH4ANMBA4VWSABRKKHEJ5QF62LMPDX5PIO44Z5DA, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.288884154 (602h5m47.159967189s) [noise=true , address=145.79.227.22:4510 , piece_id=XWR5JQBOOU5FF226G6JY266ZE5OENBAJJ2P353UOI3QSZGRBQUIQ, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.326381835 (602h5m47.19746487s) [noise=true , address=145.79.227.35:4482 , piece_id=WWZLMRQNPXAW53HCZZP5I2NAQM66OXRE2IATP366UJFZHCJMI2VQ, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.583738587 (602h5m47.454821632s) [noise=true , address=45.91.116.9:4262 , piece_id=GXQMXHK6DSHZ6NBITKSPWLCCCRF5SUYI5VB7IO6FIMAD2FKYNVWA, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.625677305 (602h5m47.49676035s) [noise=true , address=45.91.116.8:4045 , piece_id=Q3RGXD2UR4ERW4KELMX4N7HSD5AP4IVLIVWS7UYL72WM3AIS2UIA, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.625694757 (602h5m47.496777792s) [seg_pos={0 0}, upload=527721]: Segment reached optimal threshold of 13 pieces.
Dec 30 17:34:47.625981503 (602h5m47.497064548s) [noise=true , address=45.91.116.14:4189 , piece_id=EN6UQV4BZ4M635TZ6HSSN45UW65LLBKMPU6EMZXGNQLHR3MARUGQ, seg_pos={0 0}, upload=527721]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF
Dec 30 17:34:47.625991513 (602h5m47.497074548s) [noise=true , address=45.91.116.14:4189 , piece_id=EN6UQV4BZ4M635TZ6HSSN45UW65LLBKMPU6EMZXGNQLHR3MARUGQ, seg_pos={0 0}, upload=527721]: Long tail cancellation due to optimal threshold
Dec 30 17:34:47.626166286 (602h5m47.497249331s) [noise=true , address=45.91.116.12:4225 , piece_id=SUUGK5PEBWJUTZOWYIYMSJRFBWGT3D7VWNYG76T6HUQF6F2YB2IQ, seg_pos={0 0}, upload=527721]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF
Dec 30 17:34:47.626174451 (602h5m47.497257486s) [noise=true , address=45.91.116.12:4225 , piece_id=SUUGK5PEBWJUTZOWYIYMSJRFBWGT3D7VWNYG76T6HUQF6F2YB2IQ, seg_pos={0 0}, upload=527721]: Long tail cancellation due to optimal threshold
Dec 30 17:34:47.626257829 (602h5m47.497340864s) [noise=true , address=45.91.116.11:4117 , piece_id=CWPARDQZXXV46MFGPDYLAFQTPPYXD25IEQ4FIZQLEPJQK6HVKB3A, seg_pos={0 0}, upload=527721]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Dec 30 17:34:47.626267428 (602h5m47.497350463s) [noise=true , address=45.91.116.11:4117 , piece_id=CWPARDQZXXV46MFGPDYLAFQTPPYXD25IEQ4FIZQLEPJQK6HVKB3A, seg_pos={0 0}, upload=527721]: Long tail cancellation due to optimal threshold
Dec 30 17:34:47.626304649 (602h5m47.497387694s) [seg_pos={0 0}, upload=527721]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 17:34:47.626426300 (602h5m47.497509335s) [upload=527721]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 17:34:47.626433895 (602h5m47.49751693s) [upload=527721]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 17:34:47.916655325 (602h5m47.78773838s) [upload=527721]: Done uploading segments...
Dec 30 17:34:47.916680373 (602h5m47.787763408s) [upload=527721]: Upload finished. err: <nil>
Duration: 6.049942855s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...............X...X........................................] duration:
6.382536398s trace:
https://jaeger.dev.storj.io/trace/6ea66b1ed6f4c6e5
FAIL (2025-12-30 17:35:56 UTC)
orbiter warning: context deadline exceeded
Duration: 6.382587786s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [...................X...X..X............................XX...] duration:
5.870596393s trace:
https://jaeger.dev.storj.io/trace/4173ea2fcd946b89
FAIL (2025-12-30 17:35:45 UTC)
orbiter warning: context deadline exceeded
Duration: 5.870622774s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [................X...X.......................................] duration:
6.002932412s trace:
https://jaeger.dev.storj.io/trace/6ea66b1ed6f4c6e5
FAIL (2025-12-30 17:35:30 UTC)
orbiter warning: context deadline exceeded
Duration: 6.002957509s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
13 out of 60 is failed (threshold=6) [X................X..X....X.XX..X..X......X.........X..XXX...] duration:
3.572590002s trace:
https://jaeger.dev.storj.io/trace/56692ebd5ab65c7c
FAIL (2025-12-30 17:35:29 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.572613977s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................] duration:
50.17615778s trace:
https://jaeger.dev.storj.io/trace/5ea8d7d4f61e13a2
FAIL (2025-12-30 17:35:20 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK, Piece ID: HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ) context deadline exceeded; piecestore: (Node ID: 12eLKUg2vTnZMrLaraSPEoGXe3jDVBJYyURdcn4f2gCN5pvvCQ8, Piece ID: KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ) context deadline exceeded; piecestore: (Node ID: 12W4n6uf6azQk1QgeR8Xw7UkpnZm19zdyFKwojCsqZt2GFHrpce, Piece ID: DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ) context deadline exceeded
======LOGS======
17.114181822 (601h42m16.985264867s) [noise=true , address=145.79.227.44:4208 , piece_id=3OXDCHZSOIFS5VQCI4SXSAQFICDCYHRE5NGFBERR33KOTGSZWLDQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.114641387 (601h42m16.985724432s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.115652532 (601h42m16.986735578s) [noise=true , address=77.90.31.17:4790 , piece_id=5RSRESKGDA2YF4NBRF4ZFABBKWJK4NO6VLEXW2Z6EY3OPXH2WPNA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.115937134 (601h42m16.987020179s) [noise=true , address=64.49.13.155:4923 , piece_id=MUJD7XLM2YRLRCHYVSDPBE6VBYLHCOA7HJQP22ZSET4N574XQMJA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.116814986 (601h42m16.987898031s) [noise=true , address=45.91.116.11:4139 , piece_id=72DPEWZ7ZPZZ53UGYV2VCUKLCK7HCM22CDBRHWVLJGDGO5B5JADQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.117501182 (601h42m16.988584228s) [noise=true , address=77.90.31.14:4665 , piece_id=DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.118263334 (601h42m16.989346379s) [noise=true , address=145.79.227.40:4084 , piece_id=ZADZDRKBIILLAI4IT335CYETOSFOFUUPQIH6HZJCTHKRTZSPDJUQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.118555850 (601h42m16.989638886s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.118765930 (601h42m16.989849036s) [noise=true , address=77.90.31.21:4905 , piece_id=HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119029533 (601h42m16.990112568s) [noise=true , address=77.90.31.10:4533 , piece_id=QFBBYNQHQL2G7PEREK43FZBRUMKMVNBC2XKTE2NWTVTKN2452LPA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119417672 (601h42m16.990500707s) [noise=true , address=45.91.116.12:4258 , piece_id=KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119441588 (601h42m16.990524643s) [noise=true , address=145.79.227.40:4053 , piece_id=JCSIUMWKEAQNQXTNSJTN6VSB3V2SIDRQK67QMK46B4GOUWEMQE2A, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119763470 (601h42m16.990846505s) [noise=true , address=77.90.31.18:4821 , piece_id=MXZFKXM3JV2HQURDLIU7BKWLAKDZ27O4ER3SGC7QBZV5I3J2EHKA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.120028685 (601h42m16.99111172s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.120086986 (601h42m16.991170031s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.120586507 (601h42m16.991669562s) [noise=true , address=45.91.116.13:4168 , piece_id=AKUE5H3A2UECQOZY3GV3MWZKIHSNKCDOG32DKCNHIRTJROJ74M3A, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.121258768 (601h42m16.992341813s) [noise=true , address=145.79.227.9:4379 , piece_id=QTFXFSGGKYBQSPWJYBBJMQUL3D5EEO3IX4C53ECD7FXMUOGUBC7Q, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:18.568739165 (601h42m18.43982221s) [upload=527370]: Next returned nil segment
Dec 30 17:11:18.568751328 (601h42m18.439834363s) [upload=527370]: Waiting for error group managing segments...
Dec 30 17:11:18.978424178 (601h42m18.849507213s) [noise=true , address=64.49.13.155:4923 , piece_id=MUJD7XLM2YRLRCHYVSDPBE6VBYLHCOA7HJQP22ZSET4N574XQMJA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:18.986258125 (601h42m18.85734116s) [noise=true , address=145.79.227.9:4379 , piece_id=QTFXFSGGKYBQSPWJYBBJMQUL3D5EEO3IX4C53ECD7FXMUOGUBC7Q, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.274519421 (601h42m19.145602456s) [noise=true , address=77.90.31.14:4665 , piece_id=DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.315775438 (601h42m19.186858473s) [noise=true , address=45.91.116.13:4168 , piece_id=AKUE5H3A2UECQOZY3GV3MWZKIHSNKCDOG32DKCNHIRTJROJ74M3A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.335765371 (601h42m19.206848406s) [noise=true , address=77.90.31.21:4905 , piece_id=HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.354812129 (601h42m19.225895164s) [noise=true , address=77.90.31.10:4533 , piece_id=QFBBYNQHQL2G7PEREK43FZBRUMKMVNBC2XKTE2NWTVTKN2452LPA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.392024917 (601h42m19.263107952s) [noise=true , address=45.91.116.11:4139 , piece_id=72DPEWZ7ZPZZ53UGYV2VCUKLCK7HCM22CDBRHWVLJGDGO5B5JADQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.396084327 (601h42m19.267167372s) [noise=true , address=77.90.31.18:4821 , piece_id=MXZFKXM3JV2HQURDLIU7BKWLAKDZ27O4ER3SGC7QBZV5I3J2EHKA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.862099099 (601h42m19.733182134s) [noise=true , address=77.90.31.17:4790 , piece_id=5RSRESKGDA2YF4NBRF4ZFABBKWJK4NO6VLEXW2Z6EY3OPXH2WPNA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.905412913 (601h42m19.776495948s) [noise=true , address=145.79.227.44:4208 , piece_id=3OXDCHZSOIFS5VQCI4SXSAQFICDCYHRE5NGFBERR33KOTGSZWLDQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.910309688 (601h42m19.781392743s) [noise=true , address=145.79.227.30:4794 , piece_id=EJH6Q3T2BMPVLG6F7CW7N4GYOA3EZFPADL6VPT7WKKG65B2WZVXQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.934233361 (601h42m19.805316397s) [noise=true , address=145.79.227.40:4084 , piece_id=ZADZDRKBIILLAI4IT335CYETOSFOFUUPQIH6HZJCTHKRTZSPDJUQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:20.074011527 (601h42m19.945094572s) [noise=true , address=45.91.116.12:4258 , piece_id=KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:20.106010337 (601h42m19.977093372s) [noise=true , address=145.79.227.40:4053 , piece_id=JCSIUMWKEAQNQXTNSJTN6VSB3V2SIDRQK67QMK46B4GOUWEMQE2A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:20.106080911 (601h42m19.977163946s) [seg_pos={0 0}, upload=527370]: Segment reached optimal threshold of 14 pieces.
Dec 30 17:11:20.106274520 (601h42m19.977357565s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106281644 (601h42m19.977364679s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106351607 (601h42m19.977434642s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106367767 (601h42m19.977450802s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106424967 (601h42m19.977508002s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106432451 (601h42m19.977515486s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106540666 (601h42m19.977623702s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106546207 (601h42m19.977629242s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106563380 (601h42m19.977646415s) [seg_pos={0 0}, upload=527370]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 17:11:20.106625529 (601h42m19.977708564s) [upload=527370]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 17:11:20.106629837 (601h42m19.977712882s) [upload=527370]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 17:11:20.406465452 (601h42m20.277548507s) [upload=527370]: Done uploading segments...
Dec 30 17:11:20.406504206 (601h42m20.277587251s) [upload=527370]: Upload finished. err: <nil>
Duration: 50.176191825s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
27 out of 60 is failed (threshold=3) [.......X.X..XX.X.X.X....X.XXXXX.XX.X.XXX......XX.XXX...XX..X] duration:
30.00488772s trace:
https://jaeger.dev.storj.io/trace/3a1f7d23a1ef3354
FAIL (2025-12-30 17:35:14 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 30 17:34:45.008933692 (602h5m44.880016738s) [upload=527721]: Starting upload
Dec 30 17:34:45.009013564 (602h5m44.880096609s) [upload=527721]: Done starting upload
Dec 30 17:34:45.009131229 (602h5m44.880214274s) [upload=527721]: Uploading segments...
Dec 30 17:34:45.009254303 (602h5m44.880337348s) [upload=527721]: Got next segment. Inline: false
Dec 30 17:34:45.009307504 (602h5m44.88039056s) [upload=527721]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 17:34:45.009314247 (602h5m44.880397292s) [upload=527721]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 17:34:45.260909008 (602h5m45.131992073s) [seg_pos={0 0}, upload=527721]: Begin upload segment...
Dec 30 17:34:45.260991535 (602h5m45.13207458s) [seg_pos={0 0}, upload=527721]: Done begin upload segment.
Dec 30 17:34:45.261025630 (602h5m45.132108675s) [noise=true , address=45.91.116.9:4262 , piece_id=GXQMXHK6DSHZ6NBITKSPWLCCCRF5SUYI5VB7IO6FIMAD2FKYNVWA, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.261066187 (602h5m45.132149232s) [noise=true , address=145.79.227.50:4373 , piece_id=G37TCLHWAN4YWBPADLY3SBOTCCYAICFDQGK7WIYTGWZIKRSAJM3A, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.261501726 (602h5m45.132584761s) [noise=true , address=145.79.227.45:4230 , piece_id=X2ROAPLIHYN63DNINPSWV2YLZILWFMJV3P3ASEI5PDIFRXOJJJEQ, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.261952074 (602h5m45.133035119s) [noise=true , address=45.91.116.8:4045 , piece_id=Q3RGXD2UR4ERW4KELMX4N7HSD5AP4IVLIVWS7UYL72WM3AIS2UIA, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.262315346 (602h5m45.133398381s) [noise=true , address=145.79.227.27:4691 , piece_id=IGC6IL4673U7NUHOH26UKY6TZUTOQZLBDSKL4JXV3MNTLCXCOK7Q, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.262584518 (602h5m45.133667563s) [noise=true , address=145.79.227.9:4366 , piece_id=65KRZWTXUK5CT6Q5334FXBIKDVYZ6GNF4AWIUKFFSS5MFNPHOWFQ, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.262907303 (602h5m45.133990348s) [noise=true , address=145.79.227.31:4799 , piece_id=KUELWFEJRZ5AKH4ANMBA4VWSABRKKHEJ5QF62LMPDX5PIO44Z5DA, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.263203096 (602h5m45.134286141s) [noise=true , address=145.79.227.23:4546 , piece_id=D2P5BLURS4DA47VFEDTIJ3EVNVXKC5YMDBDOHVBWMAICAOVDXMKQ, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.263626272 (602h5m45.134709317s) [noise=true , address=145.79.227.11:4731 , piece_id=PFDYMYBZVHSHRVX63RJXKDCKGWNSXKDOFMORFPXO564HQ2SU4JPQ, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.265075592 (602h5m45.136158637s) [noise=true , address=45.91.116.12:4225 , piece_id=SUUGK5PEBWJUTZOWYIYMSJRFBWGT3D7VWNYG76T6HUQF6F2YB2IQ, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.266166759 (602h5m45.137249804s) [noise=true , address=145.79.227.22:4510 , piece_id=XWR5JQBOOU5FF226G6JY266ZE5OENBAJJ2P353UOI3QSZGRBQUIQ, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.267379789 (602h5m45.138462824s) [noise=true , address=145.79.227.21:4473 , piece_id=2FH5XZNFRXSWKQOLJ2ZAL26Y3KMWZZBQL2RTEBQSARKJFSEXQXFA, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.268484482 (602h5m45.139567527s) [noise=true , address=145.79.227.16:4910 , piece_id=XSJ4YM7I3BIIG3O3X3LK7HSVQKL5AA3Q4ZLWMJYQDM223AFQUFWA, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.269628030 (602h5m45.140711075s) [noise=true , address=45.91.116.11:4117 , piece_id=CWPARDQZXXV46MFGPDYLAFQTPPYXD25IEQ4FIZQLEPJQK6HVKB3A, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.272821340 (602h5m45.143904386s) [noise=true , address=45.91.116.14:4189 , piece_id=EN6UQV4BZ4M635TZ6HSSN45UW65LLBKMPU6EMZXGNQLHR3MARUGQ, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:45.274412831 (602h5m45.145495876s) [noise=true , address=145.79.227.35:4482 , piece_id=WWZLMRQNPXAW53HCZZP5I2NAQM66OXRE2IATP366UJFZHCJMI2VQ, seg_pos={0 0}, upload=527721]: Uploading piece...
Dec 30 17:34:46.959387699 (602h5m46.830470735s) [upload=527721]: Next returned nil segment
Dec 30 17:34:46.959401716 (602h5m46.830484751s) [upload=527721]: Waiting for error group managing segments...
Dec 30 17:34:47.233771963 (602h5m47.104854998s) [noise=true , address=145.79.227.45:4230 , piece_id=X2ROAPLIHYN63DNINPSWV2YLZILWFMJV3P3ASEI5PDIFRXOJJJEQ, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.248691974 (602h5m47.119775009s) [noise=true , address=145.79.227.11:4731 , piece_id=PFDYMYBZVHSHRVX63RJXKDCKGWNSXKDOFMORFPXO564HQ2SU4JPQ, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.252485076 (602h5m47.123568111s) [noise=true , address=145.79.227.27:4691 , piece_id=IGC6IL4673U7NUHOH26UKY6TZUTOQZLBDSKL4JXV3MNTLCXCOK7Q, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.261360666 (602h5m47.132443702s) [noise=true , address=145.79.227.16:4910 , piece_id=XSJ4YM7I3BIIG3O3X3LK7HSVQKL5AA3Q4ZLWMJYQDM223AFQUFWA, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.265225967 (602h5m47.136309012s) [noise=true , address=145.79.227.23:4546 , piece_id=D2P5BLURS4DA47VFEDTIJ3EVNVXKC5YMDBDOHVBWMAICAOVDXMKQ, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.272045612 (602h5m47.143128648s) [noise=true , address=145.79.227.50:4373 , piece_id=G37TCLHWAN4YWBPADLY3SBOTCCYAICFDQGK7WIYTGWZIKRSAJM3A, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.279425644 (602h5m47.150508679s) [noise=true , address=145.79.227.9:4366 , piece_id=65KRZWTXUK5CT6Q5334FXBIKDVYZ6GNF4AWIUKFFSS5MFNPHOWFQ, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.284168746 (602h5m47.155251781s) [noise=true , address=145.79.227.21:4473 , piece_id=2FH5XZNFRXSWKQOLJ2ZAL26Y3KMWZZBQL2RTEBQSARKJFSEXQXFA, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.284835917 (602h5m47.155918962s) [noise=true , address=145.79.227.31:4799 , piece_id=KUELWFEJRZ5AKH4ANMBA4VWSABRKKHEJ5QF62LMPDX5PIO44Z5DA, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.288884154 (602h5m47.159967189s) [noise=true , address=145.79.227.22:4510 , piece_id=XWR5JQBOOU5FF226G6JY266ZE5OENBAJJ2P353UOI3QSZGRBQUIQ, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.326381835 (602h5m47.19746487s) [noise=true , address=145.79.227.35:4482 , piece_id=WWZLMRQNPXAW53HCZZP5I2NAQM66OXRE2IATP366UJFZHCJMI2VQ, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.583738587 (602h5m47.454821632s) [noise=true , address=45.91.116.9:4262 , piece_id=GXQMXHK6DSHZ6NBITKSPWLCCCRF5SUYI5VB7IO6FIMAD2FKYNVWA, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.625677305 (602h5m47.49676035s) [noise=true , address=45.91.116.8:4045 , piece_id=Q3RGXD2UR4ERW4KELMX4N7HSD5AP4IVLIVWS7UYL72WM3AIS2UIA, seg_pos={0 0}, upload=527721]: Done uploading piece. err: <nil>
Dec 30 17:34:47.625694757 (602h5m47.496777792s) [seg_pos={0 0}, upload=527721]: Segment reached optimal threshold of 13 pieces.
Dec 30 17:34:47.625981503 (602h5m47.497064548s) [noise=true , address=45.91.116.14:4189 , piece_id=EN6UQV4BZ4M635TZ6HSSN45UW65LLBKMPU6EMZXGNQLHR3MARUGQ, seg_pos={0 0}, upload=527721]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF
Dec 30 17:34:47.625991513 (602h5m47.497074548s) [noise=true , address=45.91.116.14:4189 , piece_id=EN6UQV4BZ4M635TZ6HSSN45UW65LLBKMPU6EMZXGNQLHR3MARUGQ, seg_pos={0 0}, upload=527721]: Long tail cancellation due to optimal threshold
Dec 30 17:34:47.626166286 (602h5m47.497249331s) [noise=true , address=45.91.116.12:4225 , piece_id=SUUGK5PEBWJUTZOWYIYMSJRFBWGT3D7VWNYG76T6HUQF6F2YB2IQ, seg_pos={0 0}, upload=527721]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF
Dec 30 17:34:47.626174451 (602h5m47.497257486s) [noise=true , address=45.91.116.12:4225 , piece_id=SUUGK5PEBWJUTZOWYIYMSJRFBWGT3D7VWNYG76T6HUQF6F2YB2IQ, seg_pos={0 0}, upload=527721]: Long tail cancellation due to optimal threshold
Dec 30 17:34:47.626257829 (602h5m47.497340864s) [noise=true , address=45.91.116.11:4117 , piece_id=CWPARDQZXXV46MFGPDYLAFQTPPYXD25IEQ4FIZQLEPJQK6HVKB3A, seg_pos={0 0}, upload=527721]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Dec 30 17:34:47.626267428 (602h5m47.497350463s) [noise=true , address=45.91.116.11:4117 , piece_id=CWPARDQZXXV46MFGPDYLAFQTPPYXD25IEQ4FIZQLEPJQK6HVKB3A, seg_pos={0 0}, upload=527721]: Long tail cancellation due to optimal threshold
Dec 30 17:34:47.626304649 (602h5m47.497387694s) [seg_pos={0 0}, upload=527721]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 17:34:47.626426300 (602h5m47.497509335s) [upload=527721]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 17:34:47.626433895 (602h5m47.49751693s) [upload=527721]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 17:34:47.916655325 (602h5m47.78773838s) [upload=527721]: Done uploading segments...
Dec 30 17:34:47.916680373 (602h5m47.787763408s) [upload=527721]: Upload finished. err: <nil>
Duration: 30.004912037s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
14 out of 60 is failed (threshold=6) [XX................X..X....X.XX..X..X......X.........X..XXX..] duration:
3.533722393s trace:
https://jaeger.dev.storj.io/trace/56692ebd5ab65c7c
FAIL (2025-12-30 17:35:09 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.533755937s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.................X...X......................................] duration:
6.140813593s trace:
https://jaeger.dev.storj.io/trace/6ea66b1ed6f4c6e5
FAIL (2025-12-30 17:35:02 UTC)
orbiter warning: context deadline exceeded
Duration: 6.140847476s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
14 out of 60 is failed (threshold=6) [.XX................X..X....X.XX..X..X......X.........X..XXX.] duration:
3.536182195s trace:
https://jaeger.dev.storj.io/trace/56692ebd5ab65c7c
FAIL (2025-12-30 17:34:49 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.536196993s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [....................X...X..X............................XX..] duration:
12.697098595s trace:
https://jaeger.dev.storj.io/trace/4173ea2fcd946b89
FAIL (2025-12-30 17:34:45 UTC)
orbiter warning: context deadline exceeded
Duration: 12.697125366s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..................X...X.....................................] duration:
6.122734876s trace:
https://jaeger.dev.storj.io/trace/6ea66b1ed6f4c6e5
FAIL (2025-12-30 17:34:02 UTC)
orbiter warning: context deadline exceeded
Duration: 6.12276311s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [X................................X..........................] duration:
17.676169464s trace:
https://jaeger.dev.storj.io/trace/5ea8d7d4f61e13a2
FAIL (2025-12-30 17:34:00 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK, Piece ID: HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ) context deadline exceeded; piecestore: (Node ID: 12eLKUg2vTnZMrLaraSPEoGXe3jDVBJYyURdcn4f2gCN5pvvCQ8, Piece ID: KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ) context deadline exceeded; piecestore: (Node ID: 12W4n6uf6azQk1QgeR8Xw7UkpnZm19zdyFKwojCsqZt2GFHrpce, Piece ID: DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ) context deadline exceeded
======LOGS======
17.114181822 (601h42m16.985264867s) [noise=true , address=145.79.227.44:4208 , piece_id=3OXDCHZSOIFS5VQCI4SXSAQFICDCYHRE5NGFBERR33KOTGSZWLDQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.114641387 (601h42m16.985724432s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.115652532 (601h42m16.986735578s) [noise=true , address=77.90.31.17:4790 , piece_id=5RSRESKGDA2YF4NBRF4ZFABBKWJK4NO6VLEXW2Z6EY3OPXH2WPNA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.115937134 (601h42m16.987020179s) [noise=true , address=64.49.13.155:4923 , piece_id=MUJD7XLM2YRLRCHYVSDPBE6VBYLHCOA7HJQP22ZSET4N574XQMJA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.116814986 (601h42m16.987898031s) [noise=true , address=45.91.116.11:4139 , piece_id=72DPEWZ7ZPZZ53UGYV2VCUKLCK7HCM22CDBRHWVLJGDGO5B5JADQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.117501182 (601h42m16.988584228s) [noise=true , address=77.90.31.14:4665 , piece_id=DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.118263334 (601h42m16.989346379s) [noise=true , address=145.79.227.40:4084 , piece_id=ZADZDRKBIILLAI4IT335CYETOSFOFUUPQIH6HZJCTHKRTZSPDJUQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.118555850 (601h42m16.989638886s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.118765930 (601h42m16.989849036s) [noise=true , address=77.90.31.21:4905 , piece_id=HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119029533 (601h42m16.990112568s) [noise=true , address=77.90.31.10:4533 , piece_id=QFBBYNQHQL2G7PEREK43FZBRUMKMVNBC2XKTE2NWTVTKN2452LPA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119417672 (601h42m16.990500707s) [noise=true , address=45.91.116.12:4258 , piece_id=KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119441588 (601h42m16.990524643s) [noise=true , address=145.79.227.40:4053 , piece_id=JCSIUMWKEAQNQXTNSJTN6VSB3V2SIDRQK67QMK46B4GOUWEMQE2A, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119763470 (601h42m16.990846505s) [noise=true , address=77.90.31.18:4821 , piece_id=MXZFKXM3JV2HQURDLIU7BKWLAKDZ27O4ER3SGC7QBZV5I3J2EHKA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.120028685 (601h42m16.99111172s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.120086986 (601h42m16.991170031s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.120586507 (601h42m16.991669562s) [noise=true , address=45.91.116.13:4168 , piece_id=AKUE5H3A2UECQOZY3GV3MWZKIHSNKCDOG32DKCNHIRTJROJ74M3A, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.121258768 (601h42m16.992341813s) [noise=true , address=145.79.227.9:4379 , piece_id=QTFXFSGGKYBQSPWJYBBJMQUL3D5EEO3IX4C53ECD7FXMUOGUBC7Q, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:18.568739165 (601h42m18.43982221s) [upload=527370]: Next returned nil segment
Dec 30 17:11:18.568751328 (601h42m18.439834363s) [upload=527370]: Waiting for error group managing segments...
Dec 30 17:11:18.978424178 (601h42m18.849507213s) [noise=true , address=64.49.13.155:4923 , piece_id=MUJD7XLM2YRLRCHYVSDPBE6VBYLHCOA7HJQP22ZSET4N574XQMJA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:18.986258125 (601h42m18.85734116s) [noise=true , address=145.79.227.9:4379 , piece_id=QTFXFSGGKYBQSPWJYBBJMQUL3D5EEO3IX4C53ECD7FXMUOGUBC7Q, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.274519421 (601h42m19.145602456s) [noise=true , address=77.90.31.14:4665 , piece_id=DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.315775438 (601h42m19.186858473s) [noise=true , address=45.91.116.13:4168 , piece_id=AKUE5H3A2UECQOZY3GV3MWZKIHSNKCDOG32DKCNHIRTJROJ74M3A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.335765371 (601h42m19.206848406s) [noise=true , address=77.90.31.21:4905 , piece_id=HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.354812129 (601h42m19.225895164s) [noise=true , address=77.90.31.10:4533 , piece_id=QFBBYNQHQL2G7PEREK43FZBRUMKMVNBC2XKTE2NWTVTKN2452LPA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.392024917 (601h42m19.263107952s) [noise=true , address=45.91.116.11:4139 , piece_id=72DPEWZ7ZPZZ53UGYV2VCUKLCK7HCM22CDBRHWVLJGDGO5B5JADQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.396084327 (601h42m19.267167372s) [noise=true , address=77.90.31.18:4821 , piece_id=MXZFKXM3JV2HQURDLIU7BKWLAKDZ27O4ER3SGC7QBZV5I3J2EHKA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.862099099 (601h42m19.733182134s) [noise=true , address=77.90.31.17:4790 , piece_id=5RSRESKGDA2YF4NBRF4ZFABBKWJK4NO6VLEXW2Z6EY3OPXH2WPNA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.905412913 (601h42m19.776495948s) [noise=true , address=145.79.227.44:4208 , piece_id=3OXDCHZSOIFS5VQCI4SXSAQFICDCYHRE5NGFBERR33KOTGSZWLDQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.910309688 (601h42m19.781392743s) [noise=true , address=145.79.227.30:4794 , piece_id=EJH6Q3T2BMPVLG6F7CW7N4GYOA3EZFPADL6VPT7WKKG65B2WZVXQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.934233361 (601h42m19.805316397s) [noise=true , address=145.79.227.40:4084 , piece_id=ZADZDRKBIILLAI4IT335CYETOSFOFUUPQIH6HZJCTHKRTZSPDJUQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:20.074011527 (601h42m19.945094572s) [noise=true , address=45.91.116.12:4258 , piece_id=KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:20.106010337 (601h42m19.977093372s) [noise=true , address=145.79.227.40:4053 , piece_id=JCSIUMWKEAQNQXTNSJTN6VSB3V2SIDRQK67QMK46B4GOUWEMQE2A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:20.106080911 (601h42m19.977163946s) [seg_pos={0 0}, upload=527370]: Segment reached optimal threshold of 14 pieces.
Dec 30 17:11:20.106274520 (601h42m19.977357565s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106281644 (601h42m19.977364679s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106351607 (601h42m19.977434642s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106367767 (601h42m19.977450802s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106424967 (601h42m19.977508002s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106432451 (601h42m19.977515486s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106540666 (601h42m19.977623702s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106546207 (601h42m19.977629242s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106563380 (601h42m19.977646415s) [seg_pos={0 0}, upload=527370]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 17:11:20.106625529 (601h42m19.977708564s) [upload=527370]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 17:11:20.106629837 (601h42m19.977712882s) [upload=527370]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 17:11:20.406465452 (601h42m20.277548507s) [upload=527370]: Done uploading segments...
Dec 30 17:11:20.406504206 (601h42m20.277587251s) [upload=527370]: Upload finished. err: <nil>
Duration: 17.676184723s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.....................X...X..X............................XX.] duration:
5.782295114s trace:
https://jaeger.dev.storj.io/trace/4173ea2fcd946b89
FAIL (2025-12-30 17:33:56 UTC)
orbiter warning: context deadline exceeded
Duration: 5.78230858s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
15 out of 60 is failed (threshold=6) [X.XX................X..X....X.XX..X..X......X.........X..XXX] duration:
5.001786118s trace:
https://jaeger.dev.storj.io/trace/56692ebd5ab65c7c
FAIL (2025-12-30 17:33:52 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001810544s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [......................X...X..X............................XX] duration:
30.001131356s trace:
https://jaeger.dev.storj.io/trace/4173ea2fcd946b89
FAIL (2025-12-30 17:33:50 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001150554s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
27 out of 60 is failed (threshold=3) [X.......X.X..XX.X.X.X....X.XXXXX.XX.X.XXX......XX.XXX...XX..] duration:
14.836123629s trace:
https://jaeger.dev.storj.io/trace/3ad98db75ec758c
FAIL (2025-12-30 17:33:49 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: E4ABPVQUS3IBGFYJ7AGIKVRRWAED7CUPDEDMQC7U4B6DQIEI4KFQ) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: A6USQXOQ2NK42PKDDNGQII6RGQKYOUJHLOZCXTTW76QU4MXZMKNA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: F4HR7OOKVPOMSFNXODGEP72ELGY7B3A4OOTOSG4RXUGW4YODHSGQ) context deadline exceeded
======LOGS======
Dec 30 17:32:09.469380532 (602h3m9.340463567s) [upload=527684]: Starting upload
Dec 30 17:32:09.469446628 (602h3m9.340529673s) [upload=527684]: Done starting upload
Dec 30 17:32:09.469518865 (602h3m9.34060191s) [upload=527684]: Uploading segments...
Dec 30 17:32:09.469662689 (602h3m9.340745724s) [upload=527684]: Got next segment. Inline: false
Dec 30 17:32:09.469708266 (602h3m9.340791301s) [upload=527684]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 17:32:09.469713025 (602h3m9.34079607s) [upload=527684]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 17:32:09.715580348 (602h3m9.586663383s) [seg_pos={0 0}, upload=527684]: Begin upload segment...
Dec 30 17:32:09.715624413 (602h3m9.586707448s) [seg_pos={0 0}, upload=527684]: Done begin upload segment.
Dec 30 17:32:09.715656234 (602h3m9.586739269s) [noise=true , address=145.79.227.50:4374 , piece_id=M6TXJJL6ZE4HSCWAHXCIXOSPFSETFNQRYNON53XFSHE4SKH6DMZA, seg_pos={0 0}, upload=527684]: Uploading piece...
Dec 30 17:32:09.715886832 (602h3m9.586969867s) [noise=true , address=145.79.227.10:4402 , piece_id=BE2HJ3AVCEDFGFHEF5WBLCBYEHJ7CK7YMMZXZRGLDTRRFSB7PXGA, seg_pos={0 0}, upload=527684]: Uploading piece...
Dec 30 17:32:09.716669303 (602h3m9.587752348s) [noise=true , address=145.79.227.23:4546 , piece_id=2Y3TNBEX7JESIEYPKYLHAFYVW3OQCUIL3D3TBCHNQCRBNUHC33CA, seg_pos={0 0}, upload=527684]: Uploading piece...
Dec 30 17:32:09.717144767 (602h3m9.588227812s) [noise=true , address=77.90.31.11:4554 , piece_id=E4ABPVQUS3IBGFYJ7AGIKVRRWAED7CUPDEDMQC7U4B6DQIEI4KFQ, seg_pos={0 0}, upload=527684]: Uploading piece...
Dec 30 17:32:09.717448696 (602h3m9.588531731s) [noise=true , address=77.90.31.18:4808 , piece_id=GNYL3JMCVYCHV4JTTHNIBD4UEVIFP33TBMU2QQXRKHX4GYGAPGVQ, seg_pos={0 0}, upload=527684]: Uploading piece...
Dec 30 17:32:09.717722137 (602h3m9.588805172s) [noise=true , address=145.79.227.40:4049 , piece_id=ROSSDV4FISTPMOBHFWR5VTGQDUPHBC6VOMJZGFV3BRYIWRIIN4ZQ, seg_pos={0 0}, upload=527684]: Uploading piece...
Dec 30 17:32:09.718087442 (602h3m9.589170487s) [noise=true , address=145.79.227.36:4518 , piece_id=SEOJNCUGHUOSQOJBUQCDNNU2KIPQCIAI437EJ5RITQ65N2ONETMA, seg_pos={0 0}, upload=527684]: Uploading piece...
Dec 30 17:32:09.718464721 (602h3m9.589547756s) [noise=true , address=45.91.116.13:4155 , piece_id=LKYUPQ22WEIKOUJYF74ATZLNJPZZA2TDNSZU27W5RRBSU6ANWWVQ, seg_pos={0 0}, upload=527684]: Uploading piece...
Dec 30 17:32:09.719427244 (602h3m9.590510279s) [noise=true , address=45.91.116.12:4227 , piece_id=A6USQXOQ2NK42PKDDNGQII6RGQKYOUJHLOZCXTTW76QU4MXZMKNA, seg_pos={0 0}, upload=527684]: Uploading piece...
Dec 30 17:32:09.720558628 (602h3m9.591641663s) [noise=true , address=45.91.116.10:4083 , piece_id=YS4JCXLCUEERACSQQVS2H2C7JGUJX47KLSH6XXKNALMULQNXMM4A, seg_pos={0 0}, upload=527684]: Uploading piece...
Dec 30 17:32:09.722313760 (602h3m9.593396805s) [noise=true , address=77.90.31.17:4773 , piece_id=OQS26DLMRQOS2FBUXEOZOZM3J4X4D7V3F4LZ7YAVX2CW6Q77JFCA, seg_pos={0 0}, upload=527684]: Uploading piece...
Dec 30 17:32:09.724098268 (602h3m9.595181303s) [noise=true , address=45.91.116.5:4298 , piece_id=JPJSOQYWGXAEXIIDJTPVLFLBJREEEGUFCQAME4UWN37LTJNZWTJA, seg_pos={0 0}, upload=527684]: Uploading piece...
Dec 30 17:32:09.726001882 (602h3m9.597084917s) [noise=true , address=45.91.116.8:4046 , piece_id=F4HR7OOKVPOMSFNXODGEP72ELGY7B3A4OOTOSG4RXUGW4YODHSGQ, seg_pos={0 0}, upload=527684]: Uploading piece...
Dec 30 17:32:09.726750118 (602h3m9.597833163s) [noise=true , address=145.79.227.12:4767 , piece_id=HGCB4ST7HL5RWNAXUNR53UNHRS3ZG4IXQ46EWBUK67JSLYB5KZ2A, seg_pos={0 0}, upload=527684]: Uploading piece...
Dec 30 17:32:09.728103425 (602h3m9.59918646s) [noise=true , address=145.79.227.31:4797 , piece_id=NEV6V2BR4XIZFQZBWLYOCMQZQQMFLXZAXWOI5AGPR7GHQSN2CUMA, seg_pos={0 0}, upload=527684]: Uploading piece...
Dec 30 17:32:09.728361636 (602h3m9.599444681s) [noise=true , address=145.79.227.41:4087 , piece_id=7N2F6NB35UPVELPGZMOJFROVRL5SBK752EKUXGKFX5HQD3KW2Z7A, seg_pos={0 0}, upload=527684]: Uploading piece...
Dec 30 17:32:11.425122064 (602h3m11.29620511s) [upload=527684]: Next returned nil segment
Dec 30 17:32:11.425147243 (602h3m11.296230278s) [upload=527684]: Waiting for error group managing segments...
Dec 30 17:32:11.705331201 (602h3m11.576414236s) [noise=true , address=145.79.227.10:4402 , piece_id=BE2HJ3AVCEDFGFHEF5WBLCBYEHJ7CK7YMMZXZRGLDTRRFSB7PXGA, seg_pos={0 0}, upload=527684]: Done uploading piece. err: <nil>
Dec 30 17:32:11.710225240 (602h3m11.581308285s) [noise=true , address=145.79.227.41:4087 , piece_id=7N2F6NB35UPVELPGZMOJFROVRL5SBK752EKUXGKFX5HQD3KW2Z7A, seg_pos={0 0}, upload=527684]: Done uploading piece. err: <nil>
Dec 30 17:32:11.715421534 (602h3m11.586504569s) [noise=true , address=145.79.227.50:4374 , piece_id=M6TXJJL6ZE4HSCWAHXCIXOSPFSETFNQRYNON53XFSHE4SKH6DMZA, seg_pos={0 0}, upload=527684]: Done uploading piece. err: <nil>
Dec 30 17:32:11.722313567 (602h3m11.593396612s) [noise=true , address=145.79.227.23:4546 , piece_id=2Y3TNBEX7JESIEYPKYLHAFYVW3OQCUIL3D3TBCHNQCRBNUHC33CA, seg_pos={0 0}, upload=527684]: Done uploading piece. err: <nil>
Dec 30 17:32:11.732638598 (602h3m11.603721633s) [noise=true , address=145.79.227.36:4518 , piece_id=SEOJNCUGHUOSQOJBUQCDNNU2KIPQCIAI437EJ5RITQ65N2ONETMA, seg_pos={0 0}, upload=527684]: Done uploading piece. err: <nil>
Dec 30 17:32:11.733046444 (602h3m11.604129479s) [noise=true , address=145.79.227.12:4767 , piece_id=HGCB4ST7HL5RWNAXUNR53UNHRS3ZG4IXQ46EWBUK67JSLYB5KZ2A, seg_pos={0 0}, upload=527684]: Done uploading piece. err: <nil>
Dec 30 17:32:11.798023770 (602h3m11.669106816s) [noise=true , address=145.79.227.31:4797 , piece_id=NEV6V2BR4XIZFQZBWLYOCMQZQQMFLXZAXWOI5AGPR7GHQSN2CUMA, seg_pos={0 0}, upload=527684]: Done uploading piece. err: <nil>
Dec 30 17:32:11.885526657 (602h3m11.756609692s) [noise=true , address=77.90.31.11:4554 , piece_id=E4ABPVQUS3IBGFYJ7AGIKVRRWAED7CUPDEDMQC7U4B6DQIEI4KFQ, seg_pos={0 0}, upload=527684]: Done uploading piece. err: <nil>
Dec 30 17:32:11.891306192 (602h3m11.762389237s) [noise=true , address=77.90.31.18:4808 , piece_id=GNYL3JMCVYCHV4JTTHNIBD4UEVIFP33TBMU2QQXRKHX4GYGAPGVQ, seg_pos={0 0}, upload=527684]: Done uploading piece. err: <nil>
Dec 30 17:32:11.920545367 (602h3m11.791628412s) [noise=true , address=77.90.31.17:4773 , piece_id=OQS26DLMRQOS2FBUXEOZOZM3J4X4D7V3F4LZ7YAVX2CW6Q77JFCA, seg_pos={0 0}, upload=527684]: Done uploading piece. err: <nil>
Dec 30 17:32:12.291696659 (602h3m12.162779724s) [noise=true , address=45.91.116.12:4227 , piece_id=A6USQXOQ2NK42PKDDNGQII6RGQKYOUJHLOZCXTTW76QU4MXZMKNA, seg_pos={0 0}, upload=527684]: Done uploading piece. err: <nil>
Dec 30 17:32:12.523217416 (602h3m12.394300461s) [noise=true , address=45.91.116.8:4046 , piece_id=F4HR7OOKVPOMSFNXODGEP72ELGY7B3A4OOTOSG4RXUGW4YODHSGQ, seg_pos={0 0}, upload=527684]: Done uploading piece. err: <nil>
Dec 30 17:32:12.582188346 (602h3m12.453271391s) [noise=true , address=45.91.116.13:4155 , piece_id=LKYUPQ22WEIKOUJYF74ATZLNJPZZA2TDNSZU27W5RRBSU6ANWWVQ, seg_pos={0 0}, upload=527684]: Done uploading piece. err: <nil>
Dec 30 17:32:12.582213383 (602h3m12.453296428s) [seg_pos={0 0}, upload=527684]: Segment reached optimal threshold of 13 pieces.
Dec 30 17:32:12.582465514 (602h3m12.453548549s) [noise=true , address=45.91.116.10:4083 , piece_id=YS4JCXLCUEERACSQQVS2H2C7JGUJX47KLSH6XXKNALMULQNXMM4A, seg_pos={0 0}, upload=527684]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF
Dec 30 17:32:12.582473779 (602h3m12.453556824s) [noise=true , address=45.91.116.10:4083 , piece_id=YS4JCXLCUEERACSQQVS2H2C7JGUJX47KLSH6XXKNALMULQNXMM4A, seg_pos={0 0}, upload=527684]: Long tail cancellation due to optimal threshold
Dec 30 17:32:12.582675413 (602h3m12.453758448s) [noise=true , address=45.91.116.5:4298 , piece_id=JPJSOQYWGXAEXIIDJTPVLFLBJREEEGUFCQAME4UWN37LTJNZWTJA, seg_pos={0 0}, upload=527684]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:54708->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:54708->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54708->45.91.116.5:4298: use of closed network connection
Dec 30 17:32:12.582704347 (602h3m12.453787383s) [noise=true , address=45.91.116.5:4298 , piece_id=JPJSOQYWGXAEXIIDJTPVLFLBJREEEGUFCQAME4UWN37LTJNZWTJA, seg_pos={0 0}, upload=527684]: Long tail cancellation due to optimal threshold
Dec 30 17:32:12.582736829 (602h3m12.453819874s) [noise=true , address=145.79.227.40:4049 , piece_id=ROSSDV4FISTPMOBHFWR5VTGQDUPHBC6VOMJZGFV3BRYIWRIIN4ZQ, seg_pos={0 0}, upload=527684]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF
Dec 30 17:32:12.582746769 (602h3m12.453829804s) [noise=true , address=145.79.227.40:4049 , piece_id=ROSSDV4FISTPMOBHFWR5VTGQDUPHBC6VOMJZGFV3BRYIWRIIN4ZQ, seg_pos={0 0}, upload=527684]: Long tail cancellation due to optimal threshold
Dec 30 17:32:12.582852410 (602h3m12.453935445s) [seg_pos={0 0}, upload=527684]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 17:32:12.582950497 (602h3m12.454033542s) [upload=527684]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 17:32:12.582956428 (602h3m12.454039473s) [upload=527684]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 17:32:12.881619231 (602h3m12.752702266s) [upload=527684]: Done uploading segments...
Dec 30 17:32:12.881639721 (602h3m12.752722756s) [upload=527684]: Upload finished. err: <nil>
Duration: 14.836154908s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.X................................X.........................] duration:
46.140004052s trace:
https://jaeger.dev.storj.io/trace/5ea8d7d4f61e13a2
FAIL (2025-12-30 17:33:42 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK, Piece ID: HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ) context deadline exceeded; piecestore: (Node ID: 12eLKUg2vTnZMrLaraSPEoGXe3jDVBJYyURdcn4f2gCN5pvvCQ8, Piece ID: KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ) context deadline exceeded; piecestore: (Node ID: 12W4n6uf6azQk1QgeR8Xw7UkpnZm19zdyFKwojCsqZt2GFHrpce, Piece ID: DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ) context deadline exceeded
======LOGS======
17.114181822 (601h42m16.985264867s) [noise=true , address=145.79.227.44:4208 , piece_id=3OXDCHZSOIFS5VQCI4SXSAQFICDCYHRE5NGFBERR33KOTGSZWLDQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.114641387 (601h42m16.985724432s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.115652532 (601h42m16.986735578s) [noise=true , address=77.90.31.17:4790 , piece_id=5RSRESKGDA2YF4NBRF4ZFABBKWJK4NO6VLEXW2Z6EY3OPXH2WPNA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.115937134 (601h42m16.987020179s) [noise=true , address=64.49.13.155:4923 , piece_id=MUJD7XLM2YRLRCHYVSDPBE6VBYLHCOA7HJQP22ZSET4N574XQMJA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.116814986 (601h42m16.987898031s) [noise=true , address=45.91.116.11:4139 , piece_id=72DPEWZ7ZPZZ53UGYV2VCUKLCK7HCM22CDBRHWVLJGDGO5B5JADQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.117501182 (601h42m16.988584228s) [noise=true , address=77.90.31.14:4665 , piece_id=DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.118263334 (601h42m16.989346379s) [noise=true , address=145.79.227.40:4084 , piece_id=ZADZDRKBIILLAI4IT335CYETOSFOFUUPQIH6HZJCTHKRTZSPDJUQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.118555850 (601h42m16.989638886s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.118765930 (601h42m16.989849036s) [noise=true , address=77.90.31.21:4905 , piece_id=HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119029533 (601h42m16.990112568s) [noise=true , address=77.90.31.10:4533 , piece_id=QFBBYNQHQL2G7PEREK43FZBRUMKMVNBC2XKTE2NWTVTKN2452LPA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119417672 (601h42m16.990500707s) [noise=true , address=45.91.116.12:4258 , piece_id=KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119441588 (601h42m16.990524643s) [noise=true , address=145.79.227.40:4053 , piece_id=JCSIUMWKEAQNQXTNSJTN6VSB3V2SIDRQK67QMK46B4GOUWEMQE2A, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.119763470 (601h42m16.990846505s) [noise=true , address=77.90.31.18:4821 , piece_id=MXZFKXM3JV2HQURDLIU7BKWLAKDZ27O4ER3SGC7QBZV5I3J2EHKA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.120028685 (601h42m16.99111172s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.120086986 (601h42m16.991170031s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.120586507 (601h42m16.991669562s) [noise=true , address=45.91.116.13:4168 , piece_id=AKUE5H3A2UECQOZY3GV3MWZKIHSNKCDOG32DKCNHIRTJROJ74M3A, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:17.121258768 (601h42m16.992341813s) [noise=true , address=145.79.227.9:4379 , piece_id=QTFXFSGGKYBQSPWJYBBJMQUL3D5EEO3IX4C53ECD7FXMUOGUBC7Q, seg_pos={0 0}, upload=527370]: Uploading piece...
Dec 30 17:11:18.568739165 (601h42m18.43982221s) [upload=527370]: Next returned nil segment
Dec 30 17:11:18.568751328 (601h42m18.439834363s) [upload=527370]: Waiting for error group managing segments...
Dec 30 17:11:18.978424178 (601h42m18.849507213s) [noise=true , address=64.49.13.155:4923 , piece_id=MUJD7XLM2YRLRCHYVSDPBE6VBYLHCOA7HJQP22ZSET4N574XQMJA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:18.986258125 (601h42m18.85734116s) [noise=true , address=145.79.227.9:4379 , piece_id=QTFXFSGGKYBQSPWJYBBJMQUL3D5EEO3IX4C53ECD7FXMUOGUBC7Q, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.274519421 (601h42m19.145602456s) [noise=true , address=77.90.31.14:4665 , piece_id=DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.315775438 (601h42m19.186858473s) [noise=true , address=45.91.116.13:4168 , piece_id=AKUE5H3A2UECQOZY3GV3MWZKIHSNKCDOG32DKCNHIRTJROJ74M3A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.335765371 (601h42m19.206848406s) [noise=true , address=77.90.31.21:4905 , piece_id=HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.354812129 (601h42m19.225895164s) [noise=true , address=77.90.31.10:4533 , piece_id=QFBBYNQHQL2G7PEREK43FZBRUMKMVNBC2XKTE2NWTVTKN2452LPA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.392024917 (601h42m19.263107952s) [noise=true , address=45.91.116.11:4139 , piece_id=72DPEWZ7ZPZZ53UGYV2VCUKLCK7HCM22CDBRHWVLJGDGO5B5JADQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.396084327 (601h42m19.267167372s) [noise=true , address=77.90.31.18:4821 , piece_id=MXZFKXM3JV2HQURDLIU7BKWLAKDZ27O4ER3SGC7QBZV5I3J2EHKA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.862099099 (601h42m19.733182134s) [noise=true , address=77.90.31.17:4790 , piece_id=5RSRESKGDA2YF4NBRF4ZFABBKWJK4NO6VLEXW2Z6EY3OPXH2WPNA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.905412913 (601h42m19.776495948s) [noise=true , address=145.79.227.44:4208 , piece_id=3OXDCHZSOIFS5VQCI4SXSAQFICDCYHRE5NGFBERR33KOTGSZWLDQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.910309688 (601h42m19.781392743s) [noise=true , address=145.79.227.30:4794 , piece_id=EJH6Q3T2BMPVLG6F7CW7N4GYOA3EZFPADL6VPT7WKKG65B2WZVXQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:19.934233361 (601h42m19.805316397s) [noise=true , address=145.79.227.40:4084 , piece_id=ZADZDRKBIILLAI4IT335CYETOSFOFUUPQIH6HZJCTHKRTZSPDJUQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:20.074011527 (601h42m19.945094572s) [noise=true , address=45.91.116.12:4258 , piece_id=KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:20.106010337 (601h42m19.977093372s) [noise=true , address=145.79.227.40:4053 , piece_id=JCSIUMWKEAQNQXTNSJTN6VSB3V2SIDRQK67QMK46B4GOUWEMQE2A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil>
Dec 30 17:11:20.106080911 (601h42m19.977163946s) [seg_pos={0 0}, upload=527370]: Segment reached optimal threshold of 14 pieces.
Dec 30 17:11:20.106274520 (601h42m19.977357565s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106281644 (601h42m19.977364679s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106351607 (601h42m19.977434642s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106367767 (601h42m19.977450802s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106424967 (601h42m19.977508002s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106432451 (601h42m19.977515486s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106540666 (601h42m19.977623702s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF
Dec 30 17:11:20.106546207 (601h42m19.977629242s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold
Dec 30 17:11:20.106563380 (601h42m19.977646415s) [seg_pos={0 0}, upload=527370]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 17:11:20.106625529 (601h42m19.977708564s) [upload=527370]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 17:11:20.106629837 (601h42m19.977712882s) [upload=527370]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 17:11:20.406465452 (601h42m20.277548507s) [upload=527370]: Done uploading segments...
Dec 30 17:11:20.406504206 (601h42m20.277587251s) [upload=527370]: Upload finished. err: <nil>
Duration: 46.140029902s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
27 out of 60 is failed (threshold=3) [.X.......X.X..XX.X.X.X....X.XXXXX.XX.X.XXX......XX.XXX...XX.] duration:
6.283314587s trace:
https://jaeger.dev.storj.io/trace/3ad98db75ec758c
FAIL (2025-12-30 17:33:24 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: E4ABPVQUS3IBGFYJ7AGIKVRRWAED7CUPDEDMQC7U4B6DQIEI4KFQ) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: A6USQXOQ2NK42PKDDNGQII6RGQKYOUJHLOZCXTTW76QU4MXZMKNA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: F4HR7OOKVPOMSFNXODGEP72ELGY7B3A4OOTOSG4RXUGW4YODHSGQ) context deadline exceeded
======LOGS======
Dec 30 17:32:09.469380532 (602h3m9.340463567s) [upload=527684]: Starting upload
Dec 30 17:32:09.469446628 (602h3m9.340529673s) [upload=527684]: Done starting upload
Dec 30 17:32:09.469518865 (602h3m9.34060191s) [upload=527684]: Uploading segments...
Dec 30 17:32:09.469662689 (602h3m9.340745724s) [upload=527684]: Got next segment. Inline: false
Dec 30 17:32:09.469708266 (602h3m9.340791301s) [upload=527684]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 17:32:09.469713025 (602h3m9.34079607s) [upload=527684]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 17:32:09.715580348 (602h3m9.586663383s) [seg_pos={0 0}, upload=527684]: Begin upload segment...
Dec 30 17:32:09.715624413 (602h3m9.586707448s) [seg_pos={0 0}, upload=527684]: Done begin upload segment.
Dec 30 17:32:09.715656234 (602h3m9.586739269s) [noise=true , address=145.79.227.50:4374 , piece_id=M6TXJJL6ZE4HSCWAHXCIXOSPFSETFNQRYNON53XFSHE4SKH6DMZA, seg_pos={0 0}, upload=527684]: Uploading piece...
Dec 30 17:32:09.715886832 (602h3m9.586969867s) [noise=true , address=145.79.227.10:4402 , piece_id=BE2HJ3AVCEDFGFHEF5WBLCBYEHJ7CK7YMMZXZRGLDTRRFSB7PXGA, seg_pos={0 0}, upload=527684]: Uploading piece...
Dec 30 17:32:09.716669303 (602h3m9.587752348s) [noise=true , address=145.79.227.23:4546 , piece_id=2Y3TNBEX7JESIEYPKYLHAFYVW3OQCUIL3D3TBCHNQCRBNUHC33CA, seg_pos={0 0}, upload=527684]: Uploading piece...
Dec 30 17:32:09.717144767 (602h3m9.588227812s) [noise=true , address=77.90.31.11:4554 , piece_id=E4ABPVQUS3IBGFYJ7AGIKVRRWAED7CUPDEDMQC7U4B6DQIEI4KFQ, seg_pos={0 0}, upload=527684]: Uploading piece...
Dec 30 17:32:09.717448696 (602h3m9.588531731s) [noise=true , address=77.90.31.18:4808 , piece_id=GNYL3JMCVYCHV4JTTHNIBD4UEVIFP33TBMU2QQXRKHX4GYGAPGVQ, seg_pos={0 0}, upload=527684]: Uploading piece...
Dec 30 17:32:09.717722137 (602h3m9.588805172s) [noise=true , address=145.79.227.40:4049 , piece_id=ROSSDV4FISTPMOBHFWR5VTGQDUPHBC6VOMJZGFV3BRYIWRIIN4ZQ, seg_pos={0 0}, upload=527684]: Uploading piece...
Dec 30 17:32:09.718087442 (602h3m9.589170487s) [noise=true , address=145.79.227.36:4518 , piece_id=SEOJNCUGHUOSQOJBUQCDNNU2KIPQCIAI437EJ5RITQ65N2ONETMA, seg_pos={0 0}, upload=527684]: Uploading piece...
Dec 30 17:32:09.718464721 (602h3m9.589547756s) [noise=true , address=45.91.116.13:4155 , piece_id=LKYUPQ22WEIKOUJYF74ATZLNJPZZA2TDNSZU27W5RRBSU6ANWWVQ, seg_pos={0 0}, upload=527684]: Uploading piece...
Dec 30 17:32:09.719427244 (602h3m9.590510279s) [noise=true , address=45.91.116.12:4227 , piece_id=A6USQXOQ2NK42PKDDNGQII6RGQKYOUJHLOZCXTTW76QU4MXZMKNA, seg_pos={0 0}, upload=527684]: Uploading piece...
Dec 30 17:32:09.720558628 (602h3m9.591641663s) [noise=true , address=45.91.116.10:4083 , piece_id=YS4JCXLCUEERACSQQVS2H2C7JGUJX47KLSH6XXKNALMULQNXMM4A, seg_pos={0 0}, upload=527684]: Uploading piece...
Dec 30 17:32:09.722313760 (602h3m9.593396805s) [noise=true , address=77.90.31.17:4773 , piece_id=OQS26DLMRQOS2FBUXEOZOZM3J4X4D7V3F4LZ7YAVX2CW6Q77JFCA, seg_pos={0 0}, upload=527684]: Uploading piece...
Dec 30 17:32:09.724098268 (602h3m9.595181303s) [noise=true , address=45.91.116.5:4298 , piece_id=JPJSOQYWGXAEXIIDJTPVLFLBJREEEGUFCQAME4UWN37LTJNZWTJA, seg_pos={0 0}, upload=527684]: Uploading piece...
Dec 30 17:32:09.726001882 (602h3m9.597084917s) [noise=true , address=45.91.116.8:4046 , piece_id=F4HR7OOKVPOMSFNXODGEP72ELGY7B3A4OOTOSG4RXUGW4YODHSGQ, seg_pos={0 0}, upload=527684]: Uploading piece...
Dec 30 17:32:09.726750118 (602h3m9.597833163s) [noise=true , address=145.79.227.12:4767 , piece_id=HGCB4ST7HL5RWNAXUNR53UNHRS3ZG4IXQ46EWBUK67JSLYB5KZ2A, seg_pos={0 0}, upload=527684]: Uploading piece...
Dec 30 17:32:09.728103425 (602h3m9.59918646s) [noise=true , address=145.79.227.31:4797 , piece_id=NEV6V2BR4XIZFQZBWLYOCMQZQQMFLXZAXWOI5AGPR7GHQSN2CUMA, seg_pos={0 0}, upload=527684]: Uploading piece...
Dec 30 17:32:09.728361636 (602h3m9.599444681s) [noise=true , address=145.79.227.41:4087 , piece_id=7N2F6NB35UPVELPGZMOJFROVRL5SBK752EKUXGKFX5HQD3KW2Z7A, seg_pos={0 0}, upload=527684]: Uploading piece...
Dec 30 17:32:11.425122064 (602h3m11.29620511s) [upload=527684]: Next returned nil segment
Dec 30 17:32:11.425147243 (602h3m11.296230278s) [upload=527684]: Waiting for error group managing segments...
Dec 30 17:32:11.705331201 (602h3m11.576414236s) [noise=true , address=145.79.227.10:4402 , piece_id=BE2HJ3AVCEDFGFHEF5WBLCBYEHJ7CK7YMMZXZRGLDTRRFSB7PXGA, seg_pos={0 0}, upload=527684]: Done uploading piece. err: <nil>
Dec 30 17:32:11.710225240 (602h3m11.581308285s) [noise=true , address=145.79.227.41:4087 , piece_id=7N2F6NB35UPVELPGZMOJFROVRL5SBK752EKUXGKFX5HQD3KW2Z7A, seg_pos={0 0}, upload=527684]: Done uploading piece. err: <nil>
Dec 30 17:32:11.715421534 (602h3m11.586504569s) [noise=true , address=145.79.227.50:4374 , piece_id=M6TXJJL6ZE4HSCWAHXCIXOSPFSETFNQRYNON53XFSHE4SKH6DMZA, seg_pos={0 0}, upload=527684]: Done uploading piece. err: <nil>
Dec 30 17:32:11.722313567 (602h3m11.593396612s) [noise=true , address=145.79.227.23:4546 , piece_id=2Y3TNBEX7JESIEYPKYLHAFYVW3OQCUIL3D3TBCHNQCRBNUHC33CA, seg_pos={0 0}, upload=527684]: Done uploading piece. err: <nil>
Dec 30 17:32:11.732638598 (602h3m11.603721633s) [noise=true , address=145.79.227.36:4518 , piece_id=SEOJNCUGHUOSQOJBUQCDNNU2KIPQCIAI437EJ5RITQ65N2ONETMA, seg_pos={0 0}, upload=527684]: Done uploading piece. err: <nil>
Dec 30 17:32:11.733046444 (602h3m11.604129479s) [noise=true , address=145.79.227.12:4767 , piece_id=HGCB4ST7HL5RWNAXUNR53UNHRS3ZG4IXQ46EWBUK67JSLYB5KZ2A, seg_pos={0 0}, upload=527684]: Done uploading piece. err: <nil>
Dec 30 17:32:11.798023770 (602h3m11.669106816s) [noise=true , address=145.79.227.31:4797 , piece_id=NEV6V2BR4XIZFQZBWLYOCMQZQQMFLXZAXWOI5AGPR7GHQSN2CUMA, seg_pos={0 0}, upload=527684]: Done uploading piece. err: <nil>
Dec 30 17:32:11.885526657 (602h3m11.756609692s) [noise=true , address=77.90.31.11:4554 , piece_id=E4ABPVQUS3IBGFYJ7AGIKVRRWAED7CUPDEDMQC7U4B6DQIEI4KFQ, seg_pos={0 0}, upload=527684]: Done uploading piece. err: <nil>
Dec 30 17:32:11.891306192 (602h3m11.762389237s) [noise=true , address=77.90.31.18:4808 , piece_id=GNYL3JMCVYCHV4JTTHNIBD4UEVIFP33TBMU2QQXRKHX4GYGAPGVQ, seg_pos={0 0}, upload=527684]: Done uploading piece. err: <nil>
Dec 30 17:32:11.920545367 (602h3m11.791628412s) [noise=true , address=77.90.31.17:4773 , piece_id=OQS26DLMRQOS2FBUXEOZOZM3J4X4D7V3F4LZ7YAVX2CW6Q77JFCA, seg_pos={0 0}, upload=527684]: Done uploading piece. err: <nil>
Dec 30 17:32:12.291696659 (602h3m12.162779724s) [noise=true , address=45.91.116.12:4227 , piece_id=A6USQXOQ2NK42PKDDNGQII6RGQKYOUJHLOZCXTTW76QU4MXZMKNA, seg_pos={0 0}, upload=527684]: Done uploading piece. err: <nil>
Dec 30 17:32:12.523217416 (602h3m12.394300461s) [noise=true , address=45.91.116.8:4046 , piece_id=F4HR7OOKVPOMSFNXODGEP72ELGY7B3A4OOTOSG4RXUGW4YODHSGQ, seg_pos={0 0}, upload=527684]: Done uploading piece. err: <nil>
Dec 30 17:32:12.582188346 (602h3m12.453271391s) [noise=true , address=45.91.116.13:4155 , piece_id=LKYUPQ22WEIKOUJYF74ATZLNJPZZA2TDNSZU27W5RRBSU6ANWWVQ, seg_pos={0 0}, upload=527684]: Done uploading piece. err: <nil>
Dec 30 17:32:12.582213383 (602h3m12.453296428s) [seg_pos={0 0}, upload=527684]: Segment reached optimal threshold of 13 pieces.
Dec 30 17:32:12.582465514 (602h3m12.453548549s) [noise=true , address=45.91.116.10:4083 , piece_id=YS4JCXLCUEERACSQQVS2H2C7JGUJX47KLSH6XXKNALMULQNXMM4A, seg_pos={0 0}, upload=527684]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF
Dec 30 17:32:12.582473779 (602h3m12.453556824s) [noise=true , address=45.91.116.10:4083 , piece_id=YS4JCXLCUEERACSQQVS2H2C7JGUJX47KLSH6XXKNALMULQNXMM4A, seg_pos={0 0}, upload=527684]: Long tail cancellation due to optimal threshold
Dec 30 17:32:12.582675413 (602h3m12.453758448s) [noise=true , address=45.91.116.5:4298 , piece_id=JPJSOQYWGXAEXIIDJTPVLFLBJREEEGUFCQAME4UWN37LTJNZWTJA, seg_pos={0 0}, upload=527684]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:54708->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:54708->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54708->45.91.116.5:4298: use of closed network connection
Dec 30 17:32:12.582704347 (602h3m12.453787383s) [noise=true , address=45.91.116.5:4298 , piece_id=JPJSOQYWGXAEXIIDJTPVLFLBJREEEGUFCQAME4UWN37LTJNZWTJA, seg_pos={0 0}, upload=527684]: Long tail cancellation due to optimal threshold
Dec 30 17:32:12.582736829 (602h3m12.453819874s) [noise=true , address=145.79.227.40:4049 , piece_id=ROSSDV4FISTPMOBHFWR5VTGQDUPHBC6VOMJZGFV3BRYIWRIIN4ZQ, seg_pos={0 0}, upload=527684]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF
Dec 30 17:32:12.582746769 (602h3m12.453829804s) [noise=true , address=145.79.227.40:4049 , piece_id=ROSSDV4FISTPMOBHFWR5VTGQDUPHBC6VOMJZGFV3BRYIWRIIN4ZQ, seg_pos={0 0}, upload=527684]: Long tail cancellation due to optimal threshold
Dec 30 17:32:12.582852410 (602h3m12.453935445s) [seg_pos={0 0}, upload=527684]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 17:32:12.582950497 (602h3m12.454033542s) [upload=527684]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 17:32:12.582956428 (602h3m12.454039473s) [upload=527684]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 17:32:12.881619231 (602h3m12.752702266s) [upload=527684]: Done uploading segments...
Dec 30 17:32:12.881639721 (602h3m12.752722756s) [upload=527684]: Upload finished. err: <nil>
Duration: 6.283340256s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.......................X...X..X............................X] duration:
30.001692957s trace:
https://jaeger.dev.storj.io/trace/65e1276902004103
FAIL (2025-12-30 17:33:20 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001714828s
edge
gateway
us-select-1
|