Orbiter History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.117749192s trace: https://jaeger.dev.storj.io/trace/6d70ed7853c345d3 FAIL (2025-12-17 08:09:49 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.11778051s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 4.852248671s trace: https://jaeger.dev.storj.io/trace/6d70ed7853c345d3 FAIL (2025-12-17 08:09:31 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.85226911s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.107581951s trace: https://jaeger.dev.storj.io/trace/6d70ed7853c345d3 FAIL (2025-12-17 08:08:21 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.107610655s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.122827873s trace: https://jaeger.dev.storj.io/trace/6d70ed7853c345d3 FAIL (2025-12-17 08:07:58 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.122931459s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.0241126s trace: https://jaeger.dev.storj.io/trace/6d70ed7853c345d3 FAIL (2025-12-17 08:07:08 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.024150822s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.115120285s trace: https://jaeger.dev.storj.io/trace/6d70ed7853c345d3 FAIL (2025-12-17 08:06:30 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.115156274s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 4.890200298s trace: https://jaeger.dev.storj.io/trace/6d70ed7853c345d3 FAIL (2025-12-17 08:05:22 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.890233772s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.227236535s trace: https://jaeger.dev.storj.io/trace/6d70ed7853c345d3 FAIL (2025-12-17 08:05:06 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.227265821s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.454639358s trace: https://jaeger.dev.storj.io/trace/6d70ed7853c345d3 FAIL (2025-12-17 08:04:03 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.454664865s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.341677592s trace: https://jaeger.dev.storj.io/trace/6d70ed7853c345d3 FAIL (2025-12-17 08:03:34 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.341710594s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.31694322s trace: https://jaeger.dev.storj.io/trace/6d70ed7853c345d3 FAIL (2025-12-17 08:02:24 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.316971194s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.069488334s trace: https://jaeger.dev.storj.io/trace/6d70ed7853c345d3 FAIL (2025-12-17 08:01:40 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.069517711s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 4.86830265s trace: https://jaeger.dev.storj.io/trace/6d70ed7853c345d3 FAIL (2025-12-17 08:01:15 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.868334069s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.097591298s trace: https://jaeger.dev.storj.io/trace/6d70ed7853c345d3 FAIL (2025-12-17 08:00:40 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.097624231s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.232852065s trace: https://jaeger.dev.storj.io/trace/6d70ed7853c345d3 FAIL (2025-12-17 08:00:23 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.232873825s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 4.935494917s trace: https://jaeger.dev.storj.io/trace/6d70ed7853c345d3 FAIL (2025-12-17 07:59:10 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.935521927s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 4.763895039s trace: https://jaeger.dev.storj.io/trace/6d70ed7853c345d3 FAIL (2025-12-17 07:58:44 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.763925566s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 4.709786144s trace: https://jaeger.dev.storj.io/trace/6d70ed7853c345d3 FAIL (2025-12-17 07:58:28 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.709814899s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.780936047s trace: https://jaeger.dev.storj.io/trace/47e835adff344c8 FAIL (2025-12-17 07:58:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: MDYFTTVCCAUKQDEIFOYSFBFJCCAM72YEY7MV762OVDCCXBJWQF7A) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: EHNNMCVG6QW5AI6IB3WFOWDDQHP6GMVQM6XFSIIDWMMS6DEDIW6A) context deadline exceeded ======LOGS====== ec 17 07:12:37.946668541 (279h43m37.817751586s) [upload=245226]: Starting upload Dec 17 07:12:37.946758152 (279h43m37.817841197s) [upload=245226]: Done starting upload Dec 17 07:12:37.947199001 (279h43m37.818282066s) [upload=245226]: Uploading segments... Dec 17 07:12:37.947253795 (279h43m37.81833685s) [upload=245226]: Got next segment. Inline: false Dec 17 07:12:37.947295084 (279h43m37.818378129s) [upload=245226]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 07:12:37.947301105 (279h43m37.81838414s) [upload=245226]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 07:12:38.191736294 (279h43m38.062819349s) [seg_pos={0 0}, upload=245226]: Begin upload segment... Dec 17 07:12:38.191795317 (279h43m38.062878362s) [seg_pos={0 0}, upload=245226]: Done begin upload segment. Dec 17 07:12:38.191819202 (279h43m38.062902238s) [noise=true , address=145.79.227.25:4618 , piece_id=C6JEP3RAPX22ECSDSRZL7CFRIDP2U2ZNRSWPVMJRF742CPGZBGPA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.191830183 (279h43m38.062913228s) [noise=true , address=77.90.31.17:4773 , piece_id=NVICD5WRAIYRFLMSOWH5LNTKF74QSYBAUT7XLVDAQ5O3MDUOGA5A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.192400609 (279h43m38.063483654s) [noise=true , address=145.79.227.52:4446 , piece_id=FYI226BYJJ3OA4NYEC6LELDWV5EPDQIE6OVAMJV2BTXNNXNWZPQA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193213818 (279h43m38.064296853s) [noise=true , address=45.91.116.8:4046 , piece_id=QUKL5Q3ONBRNEOM5TCN5RVCEX5BGTNTUMG5CCZSMCFBMBQIROXYA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193605724 (279h43m38.064688769s) [noise=true , address=145.79.227.16:4911 , piece_id=XZSCHKVU7TN7GDL33AZ6GXE5VLUCFZRP3G7ULXNN5GUQRTA6EZ7A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193689663 (279h43m38.064772709s) [noise=true , address=145.79.227.42:4121 , piece_id=RLGJ3MHH52EQJGUYKVBF5DLX5EUQ3MSYBVE7VROLOMAUEXOYT4QQ, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193977442 (279h43m38.065060487s) [noise=true , address=77.90.31.14:4662 , piece_id=MDYFTTVCCAUKQDEIFOYSFBFJCCAM72YEY7MV762OVDCCXBJWQF7A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194245692 (279h43m38.065328737s) [noise=true , address=45.91.116.13:4156 , piece_id=2L6LQKALJCYD5RQ7SGRRDZP6BROA72REMBX6MKZSCBOUZTJOVVBQ, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194366663 (279h43m38.065449698s) [noise=true , address=45.91.116.6:4974 , piece_id=EHNNMCVG6QW5AI6IB3WFOWDDQHP6GMVQM6XFSIIDWMMS6DEDIW6A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194740605 (279h43m38.06582364s) [noise=true , address=145.79.227.38:4979 , piece_id=MILXDFE5TH2N22E76WKJRID7UPASBSTKNUBNEQSN5BD32LPLRCUA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194863168 (279h43m38.065946213s) [noise=true , address=145.79.227.36:4519 , piece_id=HF4YFXYBONFXGYDQ6TJWJMNFHOP57PXKKMCGKOLRJJYEVL7EZS2A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195047018 (279h43m38.066130053s) [noise=true , address=145.79.227.48:4302 , piece_id=3VMJMBLA2BHO4VUCKSSPAMC26IFPO3IO5TKY7OLGYDB7UVIAPQ5Q, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195387467 (279h43m38.066470512s) [noise=true , address=145.79.227.50:4374 , piece_id=RQY5LFQW2XNR6QD5LQF6YZCRCJAY6IAAZPXGYJJGXI2PZFUPPG6A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195413807 (279h43m38.066496842s) [noise=true , address=145.79.227.41:4086 , piece_id=RRV7U2CFFJEZLCEBUGBJNQ6VF7HFKEEY6BWAJTUS645L6NKZFBBQ, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195752793 (279h43m38.066835828s) [noise=true , address=145.79.227.37:4565 , piece_id=LKHAWS2BR622N3KLSHASDEIKG5HETZ2GTPEFK4KGQYJKDPHD2XYA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195926243 (279h43m38.067009288s) [noise=true , address=145.79.227.11:4732 , piece_id=HOYU2ZK34RZWREKJQSZNSFZSNPX6U3U7CNDG2OT6FDFFCBLMZSNA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:39.894104052 (279h43m39.765187097s) [upload=245226]: Next returned nil segment Dec 17 07:12:39.894125211 (279h43m39.765208246s) [upload=245226]: Waiting for error group managing segments... Dec 17 07:12:40.182872669 (279h43m40.053955705s) [noise=true , address=145.79.227.38:4979 , piece_id=MILXDFE5TH2N22E76WKJRID7UPASBSTKNUBNEQSN5BD32LPLRCUA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.187264572 (279h43m40.058347617s) [noise=true , address=145.79.227.41:4086 , piece_id=RRV7U2CFFJEZLCEBUGBJNQ6VF7HFKEEY6BWAJTUS645L6NKZFBBQ, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.187588328 (279h43m40.058671373s) [noise=true , address=145.79.227.52:4446 , piece_id=FYI226BYJJ3OA4NYEC6LELDWV5EPDQIE6OVAMJV2BTXNNXNWZPQA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.201386834 (279h43m40.072469869s) [noise=true , address=145.79.227.16:4911 , piece_id=XZSCHKVU7TN7GDL33AZ6GXE5VLUCFZRP3G7ULXNN5GUQRTA6EZ7A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.207224941 (279h43m40.078307976s) [noise=true , address=145.79.227.37:4565 , piece_id=LKHAWS2BR622N3KLSHASDEIKG5HETZ2GTPEFK4KGQYJKDPHD2XYA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.209900164 (279h43m40.080983209s) [noise=true , address=145.79.227.11:4732 , piece_id=HOYU2ZK34RZWREKJQSZNSFZSNPX6U3U7CNDG2OT6FDFFCBLMZSNA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.209956011 (279h43m40.081039046s) [noise=true , address=145.79.227.25:4618 , piece_id=C6JEP3RAPX22ECSDSRZL7CFRIDP2U2ZNRSWPVMJRF742CPGZBGPA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.211720020 (279h43m40.082803065s) [noise=true , address=145.79.227.48:4302 , piece_id=3VMJMBLA2BHO4VUCKSSPAMC26IFPO3IO5TKY7OLGYDB7UVIAPQ5Q, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.214865549 (279h43m40.085948594s) [noise=true , address=145.79.227.36:4519 , piece_id=HF4YFXYBONFXGYDQ6TJWJMNFHOP57PXKKMCGKOLRJJYEVL7EZS2A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.246003612 (279h43m40.117086657s) [noise=true , address=145.79.227.42:4121 , piece_id=RLGJ3MHH52EQJGUYKVBF5DLX5EUQ3MSYBVE7VROLOMAUEXOYT4QQ, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.277886172 (279h43m40.148969217s) [noise=true , address=145.79.227.50:4374 , piece_id=RQY5LFQW2XNR6QD5LQF6YZCRCJAY6IAAZPXGYJJGXI2PZFUPPG6A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.436062099 (279h43m40.307145134s) [noise=true , address=45.91.116.6:4974 , piece_id=EHNNMCVG6QW5AI6IB3WFOWDDQHP6GMVQM6XFSIIDWMMS6DEDIW6A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.436076746 (279h43m40.307159781s) [noise=true , address=77.90.31.14:4662 , piece_id=MDYFTTVCCAUKQDEIFOYSFBFJCCAM72YEY7MV762OVDCCXBJWQF7A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.436088268 (279h43m40.307171303s) [seg_pos={0 0}, upload=245226]: Segment reached optimal threshold of 13 pieces. Dec 17 07:12:40.436303448 (279h43m40.307386493s) [noise=true , address=45.91.116.13:4156 , piece_id=2L6LQKALJCYD5RQ7SGRRDZP6BROA72REMBX6MKZSCBOUZTJOVVBQ, seg_pos={0 0}, upload=245226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 17 07:12:40.436312615 (279h43m40.30739565s) [noise=true , address=45.91.116.13:4156 , piece_id=2L6LQKALJCYD5RQ7SGRRDZP6BROA72REMBX6MKZSCBOUZTJOVVBQ, seg_pos={0 0}, upload=245226]: Long tail cancellation due to optimal threshold Dec 17 07:12:40.436357541 (279h43m40.307440586s) [noise=true , address=77.90.31.17:4773 , piece_id=NVICD5WRAIYRFLMSOWH5LNTKF74QSYBAUT7XLVDAQ5O3MDUOGA5A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:41348->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:41348->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41348->77.90.31.17:4773: use of closed network connection Dec 17 07:12:40.436368862 (279h43m40.307451907s) [noise=true , address=77.90.31.17:4773 , piece_id=NVICD5WRAIYRFLMSOWH5LNTKF74QSYBAUT7XLVDAQ5O3MDUOGA5A, seg_pos={0 0}, upload=245226]: Long tail cancellation due to optimal threshold Dec 17 07:12:40.436437392 (279h43m40.307520437s) [noise=true , address=45.91.116.8:4046 , piece_id=QUKL5Q3ONBRNEOM5TCN5RVCEX5BGTNTUMG5CCZSMCFBMBQIROXYA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 17 07:12:40.436443625 (279h43m40.30752666s) [noise=true , address=45.91.116.8:4046 , piece_id=QUKL5Q3ONBRNEOM5TCN5RVCEX5BGTNTUMG5CCZSMCFBMBQIROXYA, seg_pos={0 0}, upload=245226]: Long tail cancellation due to optimal threshold Dec 17 07:12:40.436459425 (279h43m40.30754247s) [seg_pos={0 0}, upload=245226]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 07:12:40.436538986 (279h43m40.307622021s) [upload=245226]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 07:12:40.436544858 (279h43m40.307627903s) [upload=245226]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 07:12:40.725332982 (279h43m40.596416017s) [upload=245226]: Done uploading segments... Dec 17 07:12:40.725352159 (279h43m40.596435194s) [upload=245226]: Upload finished. err: <nil> Duration: 5.781146647s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 4.914304928s trace: https://jaeger.dev.storj.io/trace/6d70ed7853c345d3 FAIL (2025-12-17 07:58:04 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.914339043s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 4.870683586s trace: https://jaeger.dev.storj.io/trace/6d70ed7853c345d3 FAIL (2025-12-17 07:57:47 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.870711018s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.044480051s trace: https://jaeger.dev.storj.io/trace/6d70ed7853c345d3 FAIL (2025-12-17 07:57:22 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.044508936s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.830671156s trace: https://jaeger.dev.storj.io/trace/47e835adff344c8 FAIL (2025-12-17 07:57:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: MDYFTTVCCAUKQDEIFOYSFBFJCCAM72YEY7MV762OVDCCXBJWQF7A) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: EHNNMCVG6QW5AI6IB3WFOWDDQHP6GMVQM6XFSIIDWMMS6DEDIW6A) context deadline exceeded ======LOGS====== ec 17 07:12:37.946668541 (279h43m37.817751586s) [upload=245226]: Starting upload Dec 17 07:12:37.946758152 (279h43m37.817841197s) [upload=245226]: Done starting upload Dec 17 07:12:37.947199001 (279h43m37.818282066s) [upload=245226]: Uploading segments... Dec 17 07:12:37.947253795 (279h43m37.81833685s) [upload=245226]: Got next segment. Inline: false Dec 17 07:12:37.947295084 (279h43m37.818378129s) [upload=245226]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 07:12:37.947301105 (279h43m37.81838414s) [upload=245226]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 07:12:38.191736294 (279h43m38.062819349s) [seg_pos={0 0}, upload=245226]: Begin upload segment... Dec 17 07:12:38.191795317 (279h43m38.062878362s) [seg_pos={0 0}, upload=245226]: Done begin upload segment. Dec 17 07:12:38.191819202 (279h43m38.062902238s) [noise=true , address=145.79.227.25:4618 , piece_id=C6JEP3RAPX22ECSDSRZL7CFRIDP2U2ZNRSWPVMJRF742CPGZBGPA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.191830183 (279h43m38.062913228s) [noise=true , address=77.90.31.17:4773 , piece_id=NVICD5WRAIYRFLMSOWH5LNTKF74QSYBAUT7XLVDAQ5O3MDUOGA5A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.192400609 (279h43m38.063483654s) [noise=true , address=145.79.227.52:4446 , piece_id=FYI226BYJJ3OA4NYEC6LELDWV5EPDQIE6OVAMJV2BTXNNXNWZPQA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193213818 (279h43m38.064296853s) [noise=true , address=45.91.116.8:4046 , piece_id=QUKL5Q3ONBRNEOM5TCN5RVCEX5BGTNTUMG5CCZSMCFBMBQIROXYA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193605724 (279h43m38.064688769s) [noise=true , address=145.79.227.16:4911 , piece_id=XZSCHKVU7TN7GDL33AZ6GXE5VLUCFZRP3G7ULXNN5GUQRTA6EZ7A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193689663 (279h43m38.064772709s) [noise=true , address=145.79.227.42:4121 , piece_id=RLGJ3MHH52EQJGUYKVBF5DLX5EUQ3MSYBVE7VROLOMAUEXOYT4QQ, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193977442 (279h43m38.065060487s) [noise=true , address=77.90.31.14:4662 , piece_id=MDYFTTVCCAUKQDEIFOYSFBFJCCAM72YEY7MV762OVDCCXBJWQF7A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194245692 (279h43m38.065328737s) [noise=true , address=45.91.116.13:4156 , piece_id=2L6LQKALJCYD5RQ7SGRRDZP6BROA72REMBX6MKZSCBOUZTJOVVBQ, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194366663 (279h43m38.065449698s) [noise=true , address=45.91.116.6:4974 , piece_id=EHNNMCVG6QW5AI6IB3WFOWDDQHP6GMVQM6XFSIIDWMMS6DEDIW6A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194740605 (279h43m38.06582364s) [noise=true , address=145.79.227.38:4979 , piece_id=MILXDFE5TH2N22E76WKJRID7UPASBSTKNUBNEQSN5BD32LPLRCUA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194863168 (279h43m38.065946213s) [noise=true , address=145.79.227.36:4519 , piece_id=HF4YFXYBONFXGYDQ6TJWJMNFHOP57PXKKMCGKOLRJJYEVL7EZS2A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195047018 (279h43m38.066130053s) [noise=true , address=145.79.227.48:4302 , piece_id=3VMJMBLA2BHO4VUCKSSPAMC26IFPO3IO5TKY7OLGYDB7UVIAPQ5Q, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195387467 (279h43m38.066470512s) [noise=true , address=145.79.227.50:4374 , piece_id=RQY5LFQW2XNR6QD5LQF6YZCRCJAY6IAAZPXGYJJGXI2PZFUPPG6A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195413807 (279h43m38.066496842s) [noise=true , address=145.79.227.41:4086 , piece_id=RRV7U2CFFJEZLCEBUGBJNQ6VF7HFKEEY6BWAJTUS645L6NKZFBBQ, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195752793 (279h43m38.066835828s) [noise=true , address=145.79.227.37:4565 , piece_id=LKHAWS2BR622N3KLSHASDEIKG5HETZ2GTPEFK4KGQYJKDPHD2XYA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195926243 (279h43m38.067009288s) [noise=true , address=145.79.227.11:4732 , piece_id=HOYU2ZK34RZWREKJQSZNSFZSNPX6U3U7CNDG2OT6FDFFCBLMZSNA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:39.894104052 (279h43m39.765187097s) [upload=245226]: Next returned nil segment Dec 17 07:12:39.894125211 (279h43m39.765208246s) [upload=245226]: Waiting for error group managing segments... Dec 17 07:12:40.182872669 (279h43m40.053955705s) [noise=true , address=145.79.227.38:4979 , piece_id=MILXDFE5TH2N22E76WKJRID7UPASBSTKNUBNEQSN5BD32LPLRCUA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.187264572 (279h43m40.058347617s) [noise=true , address=145.79.227.41:4086 , piece_id=RRV7U2CFFJEZLCEBUGBJNQ6VF7HFKEEY6BWAJTUS645L6NKZFBBQ, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.187588328 (279h43m40.058671373s) [noise=true , address=145.79.227.52:4446 , piece_id=FYI226BYJJ3OA4NYEC6LELDWV5EPDQIE6OVAMJV2BTXNNXNWZPQA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.201386834 (279h43m40.072469869s) [noise=true , address=145.79.227.16:4911 , piece_id=XZSCHKVU7TN7GDL33AZ6GXE5VLUCFZRP3G7ULXNN5GUQRTA6EZ7A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.207224941 (279h43m40.078307976s) [noise=true , address=145.79.227.37:4565 , piece_id=LKHAWS2BR622N3KLSHASDEIKG5HETZ2GTPEFK4KGQYJKDPHD2XYA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.209900164 (279h43m40.080983209s) [noise=true , address=145.79.227.11:4732 , piece_id=HOYU2ZK34RZWREKJQSZNSFZSNPX6U3U7CNDG2OT6FDFFCBLMZSNA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.209956011 (279h43m40.081039046s) [noise=true , address=145.79.227.25:4618 , piece_id=C6JEP3RAPX22ECSDSRZL7CFRIDP2U2ZNRSWPVMJRF742CPGZBGPA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.211720020 (279h43m40.082803065s) [noise=true , address=145.79.227.48:4302 , piece_id=3VMJMBLA2BHO4VUCKSSPAMC26IFPO3IO5TKY7OLGYDB7UVIAPQ5Q, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.214865549 (279h43m40.085948594s) [noise=true , address=145.79.227.36:4519 , piece_id=HF4YFXYBONFXGYDQ6TJWJMNFHOP57PXKKMCGKOLRJJYEVL7EZS2A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.246003612 (279h43m40.117086657s) [noise=true , address=145.79.227.42:4121 , piece_id=RLGJ3MHH52EQJGUYKVBF5DLX5EUQ3MSYBVE7VROLOMAUEXOYT4QQ, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.277886172 (279h43m40.148969217s) [noise=true , address=145.79.227.50:4374 , piece_id=RQY5LFQW2XNR6QD5LQF6YZCRCJAY6IAAZPXGYJJGXI2PZFUPPG6A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.436062099 (279h43m40.307145134s) [noise=true , address=45.91.116.6:4974 , piece_id=EHNNMCVG6QW5AI6IB3WFOWDDQHP6GMVQM6XFSIIDWMMS6DEDIW6A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.436076746 (279h43m40.307159781s) [noise=true , address=77.90.31.14:4662 , piece_id=MDYFTTVCCAUKQDEIFOYSFBFJCCAM72YEY7MV762OVDCCXBJWQF7A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.436088268 (279h43m40.307171303s) [seg_pos={0 0}, upload=245226]: Segment reached optimal threshold of 13 pieces. Dec 17 07:12:40.436303448 (279h43m40.307386493s) [noise=true , address=45.91.116.13:4156 , piece_id=2L6LQKALJCYD5RQ7SGRRDZP6BROA72REMBX6MKZSCBOUZTJOVVBQ, seg_pos={0 0}, upload=245226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 17 07:12:40.436312615 (279h43m40.30739565s) [noise=true , address=45.91.116.13:4156 , piece_id=2L6LQKALJCYD5RQ7SGRRDZP6BROA72REMBX6MKZSCBOUZTJOVVBQ, seg_pos={0 0}, upload=245226]: Long tail cancellation due to optimal threshold Dec 17 07:12:40.436357541 (279h43m40.307440586s) [noise=true , address=77.90.31.17:4773 , piece_id=NVICD5WRAIYRFLMSOWH5LNTKF74QSYBAUT7XLVDAQ5O3MDUOGA5A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:41348->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:41348->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41348->77.90.31.17:4773: use of closed network connection Dec 17 07:12:40.436368862 (279h43m40.307451907s) [noise=true , address=77.90.31.17:4773 , piece_id=NVICD5WRAIYRFLMSOWH5LNTKF74QSYBAUT7XLVDAQ5O3MDUOGA5A, seg_pos={0 0}, upload=245226]: Long tail cancellation due to optimal threshold Dec 17 07:12:40.436437392 (279h43m40.307520437s) [noise=true , address=45.91.116.8:4046 , piece_id=QUKL5Q3ONBRNEOM5TCN5RVCEX5BGTNTUMG5CCZSMCFBMBQIROXYA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 17 07:12:40.436443625 (279h43m40.30752666s) [noise=true , address=45.91.116.8:4046 , piece_id=QUKL5Q3ONBRNEOM5TCN5RVCEX5BGTNTUMG5CCZSMCFBMBQIROXYA, seg_pos={0 0}, upload=245226]: Long tail cancellation due to optimal threshold Dec 17 07:12:40.436459425 (279h43m40.30754247s) [seg_pos={0 0}, upload=245226]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 07:12:40.436538986 (279h43m40.307622021s) [upload=245226]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 07:12:40.436544858 (279h43m40.307627903s) [upload=245226]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 07:12:40.725332982 (279h43m40.596416017s) [upload=245226]: Done uploading segments... Dec 17 07:12:40.725352159 (279h43m40.596435194s) [upload=245226]: Upload finished. err: <nil> Duration: 5.830701284s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 4.952505793s trace: https://jaeger.dev.storj.io/trace/6d70ed7853c345d3 FAIL (2025-12-17 07:56:48 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.952540459s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 6.035991217s trace: https://jaeger.dev.storj.io/trace/47e835adff344c8 FAIL (2025-12-17 07:56:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: MDYFTTVCCAUKQDEIFOYSFBFJCCAM72YEY7MV762OVDCCXBJWQF7A) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: EHNNMCVG6QW5AI6IB3WFOWDDQHP6GMVQM6XFSIIDWMMS6DEDIW6A) context deadline exceeded ======LOGS====== ec 17 07:12:37.946668541 (279h43m37.817751586s) [upload=245226]: Starting upload Dec 17 07:12:37.946758152 (279h43m37.817841197s) [upload=245226]: Done starting upload Dec 17 07:12:37.947199001 (279h43m37.818282066s) [upload=245226]: Uploading segments... Dec 17 07:12:37.947253795 (279h43m37.81833685s) [upload=245226]: Got next segment. Inline: false Dec 17 07:12:37.947295084 (279h43m37.818378129s) [upload=245226]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 07:12:37.947301105 (279h43m37.81838414s) [upload=245226]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 07:12:38.191736294 (279h43m38.062819349s) [seg_pos={0 0}, upload=245226]: Begin upload segment... Dec 17 07:12:38.191795317 (279h43m38.062878362s) [seg_pos={0 0}, upload=245226]: Done begin upload segment. Dec 17 07:12:38.191819202 (279h43m38.062902238s) [noise=true , address=145.79.227.25:4618 , piece_id=C6JEP3RAPX22ECSDSRZL7CFRIDP2U2ZNRSWPVMJRF742CPGZBGPA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.191830183 (279h43m38.062913228s) [noise=true , address=77.90.31.17:4773 , piece_id=NVICD5WRAIYRFLMSOWH5LNTKF74QSYBAUT7XLVDAQ5O3MDUOGA5A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.192400609 (279h43m38.063483654s) [noise=true , address=145.79.227.52:4446 , piece_id=FYI226BYJJ3OA4NYEC6LELDWV5EPDQIE6OVAMJV2BTXNNXNWZPQA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193213818 (279h43m38.064296853s) [noise=true , address=45.91.116.8:4046 , piece_id=QUKL5Q3ONBRNEOM5TCN5RVCEX5BGTNTUMG5CCZSMCFBMBQIROXYA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193605724 (279h43m38.064688769s) [noise=true , address=145.79.227.16:4911 , piece_id=XZSCHKVU7TN7GDL33AZ6GXE5VLUCFZRP3G7ULXNN5GUQRTA6EZ7A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193689663 (279h43m38.064772709s) [noise=true , address=145.79.227.42:4121 , piece_id=RLGJ3MHH52EQJGUYKVBF5DLX5EUQ3MSYBVE7VROLOMAUEXOYT4QQ, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193977442 (279h43m38.065060487s) [noise=true , address=77.90.31.14:4662 , piece_id=MDYFTTVCCAUKQDEIFOYSFBFJCCAM72YEY7MV762OVDCCXBJWQF7A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194245692 (279h43m38.065328737s) [noise=true , address=45.91.116.13:4156 , piece_id=2L6LQKALJCYD5RQ7SGRRDZP6BROA72REMBX6MKZSCBOUZTJOVVBQ, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194366663 (279h43m38.065449698s) [noise=true , address=45.91.116.6:4974 , piece_id=EHNNMCVG6QW5AI6IB3WFOWDDQHP6GMVQM6XFSIIDWMMS6DEDIW6A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194740605 (279h43m38.06582364s) [noise=true , address=145.79.227.38:4979 , piece_id=MILXDFE5TH2N22E76WKJRID7UPASBSTKNUBNEQSN5BD32LPLRCUA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194863168 (279h43m38.065946213s) [noise=true , address=145.79.227.36:4519 , piece_id=HF4YFXYBONFXGYDQ6TJWJMNFHOP57PXKKMCGKOLRJJYEVL7EZS2A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195047018 (279h43m38.066130053s) [noise=true , address=145.79.227.48:4302 , piece_id=3VMJMBLA2BHO4VUCKSSPAMC26IFPO3IO5TKY7OLGYDB7UVIAPQ5Q, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195387467 (279h43m38.066470512s) [noise=true , address=145.79.227.50:4374 , piece_id=RQY5LFQW2XNR6QD5LQF6YZCRCJAY6IAAZPXGYJJGXI2PZFUPPG6A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195413807 (279h43m38.066496842s) [noise=true , address=145.79.227.41:4086 , piece_id=RRV7U2CFFJEZLCEBUGBJNQ6VF7HFKEEY6BWAJTUS645L6NKZFBBQ, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195752793 (279h43m38.066835828s) [noise=true , address=145.79.227.37:4565 , piece_id=LKHAWS2BR622N3KLSHASDEIKG5HETZ2GTPEFK4KGQYJKDPHD2XYA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195926243 (279h43m38.067009288s) [noise=true , address=145.79.227.11:4732 , piece_id=HOYU2ZK34RZWREKJQSZNSFZSNPX6U3U7CNDG2OT6FDFFCBLMZSNA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:39.894104052 (279h43m39.765187097s) [upload=245226]: Next returned nil segment Dec 17 07:12:39.894125211 (279h43m39.765208246s) [upload=245226]: Waiting for error group managing segments... Dec 17 07:12:40.182872669 (279h43m40.053955705s) [noise=true , address=145.79.227.38:4979 , piece_id=MILXDFE5TH2N22E76WKJRID7UPASBSTKNUBNEQSN5BD32LPLRCUA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.187264572 (279h43m40.058347617s) [noise=true , address=145.79.227.41:4086 , piece_id=RRV7U2CFFJEZLCEBUGBJNQ6VF7HFKEEY6BWAJTUS645L6NKZFBBQ, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.187588328 (279h43m40.058671373s) [noise=true , address=145.79.227.52:4446 , piece_id=FYI226BYJJ3OA4NYEC6LELDWV5EPDQIE6OVAMJV2BTXNNXNWZPQA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.201386834 (279h43m40.072469869s) [noise=true , address=145.79.227.16:4911 , piece_id=XZSCHKVU7TN7GDL33AZ6GXE5VLUCFZRP3G7ULXNN5GUQRTA6EZ7A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.207224941 (279h43m40.078307976s) [noise=true , address=145.79.227.37:4565 , piece_id=LKHAWS2BR622N3KLSHASDEIKG5HETZ2GTPEFK4KGQYJKDPHD2XYA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.209900164 (279h43m40.080983209s) [noise=true , address=145.79.227.11:4732 , piece_id=HOYU2ZK34RZWREKJQSZNSFZSNPX6U3U7CNDG2OT6FDFFCBLMZSNA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.209956011 (279h43m40.081039046s) [noise=true , address=145.79.227.25:4618 , piece_id=C6JEP3RAPX22ECSDSRZL7CFRIDP2U2ZNRSWPVMJRF742CPGZBGPA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.211720020 (279h43m40.082803065s) [noise=true , address=145.79.227.48:4302 , piece_id=3VMJMBLA2BHO4VUCKSSPAMC26IFPO3IO5TKY7OLGYDB7UVIAPQ5Q, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.214865549 (279h43m40.085948594s) [noise=true , address=145.79.227.36:4519 , piece_id=HF4YFXYBONFXGYDQ6TJWJMNFHOP57PXKKMCGKOLRJJYEVL7EZS2A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.246003612 (279h43m40.117086657s) [noise=true , address=145.79.227.42:4121 , piece_id=RLGJ3MHH52EQJGUYKVBF5DLX5EUQ3MSYBVE7VROLOMAUEXOYT4QQ, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.277886172 (279h43m40.148969217s) [noise=true , address=145.79.227.50:4374 , piece_id=RQY5LFQW2XNR6QD5LQF6YZCRCJAY6IAAZPXGYJJGXI2PZFUPPG6A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.436062099 (279h43m40.307145134s) [noise=true , address=45.91.116.6:4974 , piece_id=EHNNMCVG6QW5AI6IB3WFOWDDQHP6GMVQM6XFSIIDWMMS6DEDIW6A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.436076746 (279h43m40.307159781s) [noise=true , address=77.90.31.14:4662 , piece_id=MDYFTTVCCAUKQDEIFOYSFBFJCCAM72YEY7MV762OVDCCXBJWQF7A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.436088268 (279h43m40.307171303s) [seg_pos={0 0}, upload=245226]: Segment reached optimal threshold of 13 pieces. Dec 17 07:12:40.436303448 (279h43m40.307386493s) [noise=true , address=45.91.116.13:4156 , piece_id=2L6LQKALJCYD5RQ7SGRRDZP6BROA72REMBX6MKZSCBOUZTJOVVBQ, seg_pos={0 0}, upload=245226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 17 07:12:40.436312615 (279h43m40.30739565s) [noise=true , address=45.91.116.13:4156 , piece_id=2L6LQKALJCYD5RQ7SGRRDZP6BROA72REMBX6MKZSCBOUZTJOVVBQ, seg_pos={0 0}, upload=245226]: Long tail cancellation due to optimal threshold Dec 17 07:12:40.436357541 (279h43m40.307440586s) [noise=true , address=77.90.31.17:4773 , piece_id=NVICD5WRAIYRFLMSOWH5LNTKF74QSYBAUT7XLVDAQ5O3MDUOGA5A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:41348->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:41348->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41348->77.90.31.17:4773: use of closed network connection Dec 17 07:12:40.436368862 (279h43m40.307451907s) [noise=true , address=77.90.31.17:4773 , piece_id=NVICD5WRAIYRFLMSOWH5LNTKF74QSYBAUT7XLVDAQ5O3MDUOGA5A, seg_pos={0 0}, upload=245226]: Long tail cancellation due to optimal threshold Dec 17 07:12:40.436437392 (279h43m40.307520437s) [noise=true , address=45.91.116.8:4046 , piece_id=QUKL5Q3ONBRNEOM5TCN5RVCEX5BGTNTUMG5CCZSMCFBMBQIROXYA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 17 07:12:40.436443625 (279h43m40.30752666s) [noise=true , address=45.91.116.8:4046 , piece_id=QUKL5Q3ONBRNEOM5TCN5RVCEX5BGTNTUMG5CCZSMCFBMBQIROXYA, seg_pos={0 0}, upload=245226]: Long tail cancellation due to optimal threshold Dec 17 07:12:40.436459425 (279h43m40.30754247s) [seg_pos={0 0}, upload=245226]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 07:12:40.436538986 (279h43m40.307622021s) [upload=245226]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 07:12:40.436544858 (279h43m40.307627903s) [upload=245226]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 07:12:40.725332982 (279h43m40.596416017s) [upload=245226]: Done uploading segments... Dec 17 07:12:40.725352159 (279h43m40.596435194s) [upload=245226]: Upload finished. err: <nil> Duration: 6.036021654s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 4.846320617s trace: https://jaeger.dev.storj.io/trace/6d70ed7853c345d3 FAIL (2025-12-17 07:55:41 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.846352598s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.6691664s trace: https://jaeger.dev.storj.io/trace/47e835adff344c8 FAIL (2025-12-17 07:55:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: MDYFTTVCCAUKQDEIFOYSFBFJCCAM72YEY7MV762OVDCCXBJWQF7A) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: EHNNMCVG6QW5AI6IB3WFOWDDQHP6GMVQM6XFSIIDWMMS6DEDIW6A) context deadline exceeded ======LOGS====== ec 17 07:12:37.946668541 (279h43m37.817751586s) [upload=245226]: Starting upload Dec 17 07:12:37.946758152 (279h43m37.817841197s) [upload=245226]: Done starting upload Dec 17 07:12:37.947199001 (279h43m37.818282066s) [upload=245226]: Uploading segments... Dec 17 07:12:37.947253795 (279h43m37.81833685s) [upload=245226]: Got next segment. Inline: false Dec 17 07:12:37.947295084 (279h43m37.818378129s) [upload=245226]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 07:12:37.947301105 (279h43m37.81838414s) [upload=245226]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 07:12:38.191736294 (279h43m38.062819349s) [seg_pos={0 0}, upload=245226]: Begin upload segment... Dec 17 07:12:38.191795317 (279h43m38.062878362s) [seg_pos={0 0}, upload=245226]: Done begin upload segment. Dec 17 07:12:38.191819202 (279h43m38.062902238s) [noise=true , address=145.79.227.25:4618 , piece_id=C6JEP3RAPX22ECSDSRZL7CFRIDP2U2ZNRSWPVMJRF742CPGZBGPA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.191830183 (279h43m38.062913228s) [noise=true , address=77.90.31.17:4773 , piece_id=NVICD5WRAIYRFLMSOWH5LNTKF74QSYBAUT7XLVDAQ5O3MDUOGA5A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.192400609 (279h43m38.063483654s) [noise=true , address=145.79.227.52:4446 , piece_id=FYI226BYJJ3OA4NYEC6LELDWV5EPDQIE6OVAMJV2BTXNNXNWZPQA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193213818 (279h43m38.064296853s) [noise=true , address=45.91.116.8:4046 , piece_id=QUKL5Q3ONBRNEOM5TCN5RVCEX5BGTNTUMG5CCZSMCFBMBQIROXYA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193605724 (279h43m38.064688769s) [noise=true , address=145.79.227.16:4911 , piece_id=XZSCHKVU7TN7GDL33AZ6GXE5VLUCFZRP3G7ULXNN5GUQRTA6EZ7A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193689663 (279h43m38.064772709s) [noise=true , address=145.79.227.42:4121 , piece_id=RLGJ3MHH52EQJGUYKVBF5DLX5EUQ3MSYBVE7VROLOMAUEXOYT4QQ, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193977442 (279h43m38.065060487s) [noise=true , address=77.90.31.14:4662 , piece_id=MDYFTTVCCAUKQDEIFOYSFBFJCCAM72YEY7MV762OVDCCXBJWQF7A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194245692 (279h43m38.065328737s) [noise=true , address=45.91.116.13:4156 , piece_id=2L6LQKALJCYD5RQ7SGRRDZP6BROA72REMBX6MKZSCBOUZTJOVVBQ, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194366663 (279h43m38.065449698s) [noise=true , address=45.91.116.6:4974 , piece_id=EHNNMCVG6QW5AI6IB3WFOWDDQHP6GMVQM6XFSIIDWMMS6DEDIW6A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194740605 (279h43m38.06582364s) [noise=true , address=145.79.227.38:4979 , piece_id=MILXDFE5TH2N22E76WKJRID7UPASBSTKNUBNEQSN5BD32LPLRCUA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194863168 (279h43m38.065946213s) [noise=true , address=145.79.227.36:4519 , piece_id=HF4YFXYBONFXGYDQ6TJWJMNFHOP57PXKKMCGKOLRJJYEVL7EZS2A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195047018 (279h43m38.066130053s) [noise=true , address=145.79.227.48:4302 , piece_id=3VMJMBLA2BHO4VUCKSSPAMC26IFPO3IO5TKY7OLGYDB7UVIAPQ5Q, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195387467 (279h43m38.066470512s) [noise=true , address=145.79.227.50:4374 , piece_id=RQY5LFQW2XNR6QD5LQF6YZCRCJAY6IAAZPXGYJJGXI2PZFUPPG6A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195413807 (279h43m38.066496842s) [noise=true , address=145.79.227.41:4086 , piece_id=RRV7U2CFFJEZLCEBUGBJNQ6VF7HFKEEY6BWAJTUS645L6NKZFBBQ, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195752793 (279h43m38.066835828s) [noise=true , address=145.79.227.37:4565 , piece_id=LKHAWS2BR622N3KLSHASDEIKG5HETZ2GTPEFK4KGQYJKDPHD2XYA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195926243 (279h43m38.067009288s) [noise=true , address=145.79.227.11:4732 , piece_id=HOYU2ZK34RZWREKJQSZNSFZSNPX6U3U7CNDG2OT6FDFFCBLMZSNA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:39.894104052 (279h43m39.765187097s) [upload=245226]: Next returned nil segment Dec 17 07:12:39.894125211 (279h43m39.765208246s) [upload=245226]: Waiting for error group managing segments... Dec 17 07:12:40.182872669 (279h43m40.053955705s) [noise=true , address=145.79.227.38:4979 , piece_id=MILXDFE5TH2N22E76WKJRID7UPASBSTKNUBNEQSN5BD32LPLRCUA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.187264572 (279h43m40.058347617s) [noise=true , address=145.79.227.41:4086 , piece_id=RRV7U2CFFJEZLCEBUGBJNQ6VF7HFKEEY6BWAJTUS645L6NKZFBBQ, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.187588328 (279h43m40.058671373s) [noise=true , address=145.79.227.52:4446 , piece_id=FYI226BYJJ3OA4NYEC6LELDWV5EPDQIE6OVAMJV2BTXNNXNWZPQA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.201386834 (279h43m40.072469869s) [noise=true , address=145.79.227.16:4911 , piece_id=XZSCHKVU7TN7GDL33AZ6GXE5VLUCFZRP3G7ULXNN5GUQRTA6EZ7A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.207224941 (279h43m40.078307976s) [noise=true , address=145.79.227.37:4565 , piece_id=LKHAWS2BR622N3KLSHASDEIKG5HETZ2GTPEFK4KGQYJKDPHD2XYA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.209900164 (279h43m40.080983209s) [noise=true , address=145.79.227.11:4732 , piece_id=HOYU2ZK34RZWREKJQSZNSFZSNPX6U3U7CNDG2OT6FDFFCBLMZSNA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.209956011 (279h43m40.081039046s) [noise=true , address=145.79.227.25:4618 , piece_id=C6JEP3RAPX22ECSDSRZL7CFRIDP2U2ZNRSWPVMJRF742CPGZBGPA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.211720020 (279h43m40.082803065s) [noise=true , address=145.79.227.48:4302 , piece_id=3VMJMBLA2BHO4VUCKSSPAMC26IFPO3IO5TKY7OLGYDB7UVIAPQ5Q, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.214865549 (279h43m40.085948594s) [noise=true , address=145.79.227.36:4519 , piece_id=HF4YFXYBONFXGYDQ6TJWJMNFHOP57PXKKMCGKOLRJJYEVL7EZS2A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.246003612 (279h43m40.117086657s) [noise=true , address=145.79.227.42:4121 , piece_id=RLGJ3MHH52EQJGUYKVBF5DLX5EUQ3MSYBVE7VROLOMAUEXOYT4QQ, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.277886172 (279h43m40.148969217s) [noise=true , address=145.79.227.50:4374 , piece_id=RQY5LFQW2XNR6QD5LQF6YZCRCJAY6IAAZPXGYJJGXI2PZFUPPG6A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.436062099 (279h43m40.307145134s) [noise=true , address=45.91.116.6:4974 , piece_id=EHNNMCVG6QW5AI6IB3WFOWDDQHP6GMVQM6XFSIIDWMMS6DEDIW6A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.436076746 (279h43m40.307159781s) [noise=true , address=77.90.31.14:4662 , piece_id=MDYFTTVCCAUKQDEIFOYSFBFJCCAM72YEY7MV762OVDCCXBJWQF7A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.436088268 (279h43m40.307171303s) [seg_pos={0 0}, upload=245226]: Segment reached optimal threshold of 13 pieces. Dec 17 07:12:40.436303448 (279h43m40.307386493s) [noise=true , address=45.91.116.13:4156 , piece_id=2L6LQKALJCYD5RQ7SGRRDZP6BROA72REMBX6MKZSCBOUZTJOVVBQ, seg_pos={0 0}, upload=245226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 17 07:12:40.436312615 (279h43m40.30739565s) [noise=true , address=45.91.116.13:4156 , piece_id=2L6LQKALJCYD5RQ7SGRRDZP6BROA72REMBX6MKZSCBOUZTJOVVBQ, seg_pos={0 0}, upload=245226]: Long tail cancellation due to optimal threshold Dec 17 07:12:40.436357541 (279h43m40.307440586s) [noise=true , address=77.90.31.17:4773 , piece_id=NVICD5WRAIYRFLMSOWH5LNTKF74QSYBAUT7XLVDAQ5O3MDUOGA5A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:41348->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:41348->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41348->77.90.31.17:4773: use of closed network connection Dec 17 07:12:40.436368862 (279h43m40.307451907s) [noise=true , address=77.90.31.17:4773 , piece_id=NVICD5WRAIYRFLMSOWH5LNTKF74QSYBAUT7XLVDAQ5O3MDUOGA5A, seg_pos={0 0}, upload=245226]: Long tail cancellation due to optimal threshold Dec 17 07:12:40.436437392 (279h43m40.307520437s) [noise=true , address=45.91.116.8:4046 , piece_id=QUKL5Q3ONBRNEOM5TCN5RVCEX5BGTNTUMG5CCZSMCFBMBQIROXYA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 17 07:12:40.436443625 (279h43m40.30752666s) [noise=true , address=45.91.116.8:4046 , piece_id=QUKL5Q3ONBRNEOM5TCN5RVCEX5BGTNTUMG5CCZSMCFBMBQIROXYA, seg_pos={0 0}, upload=245226]: Long tail cancellation due to optimal threshold Dec 17 07:12:40.436459425 (279h43m40.30754247s) [seg_pos={0 0}, upload=245226]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 07:12:40.436538986 (279h43m40.307622021s) [upload=245226]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 07:12:40.436544858 (279h43m40.307627903s) [upload=245226]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 07:12:40.725332982 (279h43m40.596416017s) [upload=245226]: Done uploading segments... Dec 17 07:12:40.725352159 (279h43m40.596435194s) [upload=245226]: Upload finished. err: <nil> Duration: 5.669194564s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.094680591s trace: https://jaeger.dev.storj.io/trace/6d70ed7853c345d3 FAIL (2025-12-17 07:55:02 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.094714006s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.969138855s trace: https://jaeger.dev.storj.io/trace/47e835adff344c8 FAIL (2025-12-17 07:54:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: MDYFTTVCCAUKQDEIFOYSFBFJCCAM72YEY7MV762OVDCCXBJWQF7A) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: EHNNMCVG6QW5AI6IB3WFOWDDQHP6GMVQM6XFSIIDWMMS6DEDIW6A) context deadline exceeded ======LOGS====== ec 17 07:12:37.946668541 (279h43m37.817751586s) [upload=245226]: Starting upload Dec 17 07:12:37.946758152 (279h43m37.817841197s) [upload=245226]: Done starting upload Dec 17 07:12:37.947199001 (279h43m37.818282066s) [upload=245226]: Uploading segments... Dec 17 07:12:37.947253795 (279h43m37.81833685s) [upload=245226]: Got next segment. Inline: false Dec 17 07:12:37.947295084 (279h43m37.818378129s) [upload=245226]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 07:12:37.947301105 (279h43m37.81838414s) [upload=245226]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 07:12:38.191736294 (279h43m38.062819349s) [seg_pos={0 0}, upload=245226]: Begin upload segment... Dec 17 07:12:38.191795317 (279h43m38.062878362s) [seg_pos={0 0}, upload=245226]: Done begin upload segment. Dec 17 07:12:38.191819202 (279h43m38.062902238s) [noise=true , address=145.79.227.25:4618 , piece_id=C6JEP3RAPX22ECSDSRZL7CFRIDP2U2ZNRSWPVMJRF742CPGZBGPA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.191830183 (279h43m38.062913228s) [noise=true , address=77.90.31.17:4773 , piece_id=NVICD5WRAIYRFLMSOWH5LNTKF74QSYBAUT7XLVDAQ5O3MDUOGA5A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.192400609 (279h43m38.063483654s) [noise=true , address=145.79.227.52:4446 , piece_id=FYI226BYJJ3OA4NYEC6LELDWV5EPDQIE6OVAMJV2BTXNNXNWZPQA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193213818 (279h43m38.064296853s) [noise=true , address=45.91.116.8:4046 , piece_id=QUKL5Q3ONBRNEOM5TCN5RVCEX5BGTNTUMG5CCZSMCFBMBQIROXYA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193605724 (279h43m38.064688769s) [noise=true , address=145.79.227.16:4911 , piece_id=XZSCHKVU7TN7GDL33AZ6GXE5VLUCFZRP3G7ULXNN5GUQRTA6EZ7A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193689663 (279h43m38.064772709s) [noise=true , address=145.79.227.42:4121 , piece_id=RLGJ3MHH52EQJGUYKVBF5DLX5EUQ3MSYBVE7VROLOMAUEXOYT4QQ, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193977442 (279h43m38.065060487s) [noise=true , address=77.90.31.14:4662 , piece_id=MDYFTTVCCAUKQDEIFOYSFBFJCCAM72YEY7MV762OVDCCXBJWQF7A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194245692 (279h43m38.065328737s) [noise=true , address=45.91.116.13:4156 , piece_id=2L6LQKALJCYD5RQ7SGRRDZP6BROA72REMBX6MKZSCBOUZTJOVVBQ, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194366663 (279h43m38.065449698s) [noise=true , address=45.91.116.6:4974 , piece_id=EHNNMCVG6QW5AI6IB3WFOWDDQHP6GMVQM6XFSIIDWMMS6DEDIW6A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194740605 (279h43m38.06582364s) [noise=true , address=145.79.227.38:4979 , piece_id=MILXDFE5TH2N22E76WKJRID7UPASBSTKNUBNEQSN5BD32LPLRCUA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194863168 (279h43m38.065946213s) [noise=true , address=145.79.227.36:4519 , piece_id=HF4YFXYBONFXGYDQ6TJWJMNFHOP57PXKKMCGKOLRJJYEVL7EZS2A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195047018 (279h43m38.066130053s) [noise=true , address=145.79.227.48:4302 , piece_id=3VMJMBLA2BHO4VUCKSSPAMC26IFPO3IO5TKY7OLGYDB7UVIAPQ5Q, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195387467 (279h43m38.066470512s) [noise=true , address=145.79.227.50:4374 , piece_id=RQY5LFQW2XNR6QD5LQF6YZCRCJAY6IAAZPXGYJJGXI2PZFUPPG6A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195413807 (279h43m38.066496842s) [noise=true , address=145.79.227.41:4086 , piece_id=RRV7U2CFFJEZLCEBUGBJNQ6VF7HFKEEY6BWAJTUS645L6NKZFBBQ, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195752793 (279h43m38.066835828s) [noise=true , address=145.79.227.37:4565 , piece_id=LKHAWS2BR622N3KLSHASDEIKG5HETZ2GTPEFK4KGQYJKDPHD2XYA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195926243 (279h43m38.067009288s) [noise=true , address=145.79.227.11:4732 , piece_id=HOYU2ZK34RZWREKJQSZNSFZSNPX6U3U7CNDG2OT6FDFFCBLMZSNA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:39.894104052 (279h43m39.765187097s) [upload=245226]: Next returned nil segment Dec 17 07:12:39.894125211 (279h43m39.765208246s) [upload=245226]: Waiting for error group managing segments... Dec 17 07:12:40.182872669 (279h43m40.053955705s) [noise=true , address=145.79.227.38:4979 , piece_id=MILXDFE5TH2N22E76WKJRID7UPASBSTKNUBNEQSN5BD32LPLRCUA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.187264572 (279h43m40.058347617s) [noise=true , address=145.79.227.41:4086 , piece_id=RRV7U2CFFJEZLCEBUGBJNQ6VF7HFKEEY6BWAJTUS645L6NKZFBBQ, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.187588328 (279h43m40.058671373s) [noise=true , address=145.79.227.52:4446 , piece_id=FYI226BYJJ3OA4NYEC6LELDWV5EPDQIE6OVAMJV2BTXNNXNWZPQA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.201386834 (279h43m40.072469869s) [noise=true , address=145.79.227.16:4911 , piece_id=XZSCHKVU7TN7GDL33AZ6GXE5VLUCFZRP3G7ULXNN5GUQRTA6EZ7A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.207224941 (279h43m40.078307976s) [noise=true , address=145.79.227.37:4565 , piece_id=LKHAWS2BR622N3KLSHASDEIKG5HETZ2GTPEFK4KGQYJKDPHD2XYA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.209900164 (279h43m40.080983209s) [noise=true , address=145.79.227.11:4732 , piece_id=HOYU2ZK34RZWREKJQSZNSFZSNPX6U3U7CNDG2OT6FDFFCBLMZSNA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.209956011 (279h43m40.081039046s) [noise=true , address=145.79.227.25:4618 , piece_id=C6JEP3RAPX22ECSDSRZL7CFRIDP2U2ZNRSWPVMJRF742CPGZBGPA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.211720020 (279h43m40.082803065s) [noise=true , address=145.79.227.48:4302 , piece_id=3VMJMBLA2BHO4VUCKSSPAMC26IFPO3IO5TKY7OLGYDB7UVIAPQ5Q, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.214865549 (279h43m40.085948594s) [noise=true , address=145.79.227.36:4519 , piece_id=HF4YFXYBONFXGYDQ6TJWJMNFHOP57PXKKMCGKOLRJJYEVL7EZS2A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.246003612 (279h43m40.117086657s) [noise=true , address=145.79.227.42:4121 , piece_id=RLGJ3MHH52EQJGUYKVBF5DLX5EUQ3MSYBVE7VROLOMAUEXOYT4QQ, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.277886172 (279h43m40.148969217s) [noise=true , address=145.79.227.50:4374 , piece_id=RQY5LFQW2XNR6QD5LQF6YZCRCJAY6IAAZPXGYJJGXI2PZFUPPG6A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.436062099 (279h43m40.307145134s) [noise=true , address=45.91.116.6:4974 , piece_id=EHNNMCVG6QW5AI6IB3WFOWDDQHP6GMVQM6XFSIIDWMMS6DEDIW6A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.436076746 (279h43m40.307159781s) [noise=true , address=77.90.31.14:4662 , piece_id=MDYFTTVCCAUKQDEIFOYSFBFJCCAM72YEY7MV762OVDCCXBJWQF7A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.436088268 (279h43m40.307171303s) [seg_pos={0 0}, upload=245226]: Segment reached optimal threshold of 13 pieces. Dec 17 07:12:40.436303448 (279h43m40.307386493s) [noise=true , address=45.91.116.13:4156 , piece_id=2L6LQKALJCYD5RQ7SGRRDZP6BROA72REMBX6MKZSCBOUZTJOVVBQ, seg_pos={0 0}, upload=245226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 17 07:12:40.436312615 (279h43m40.30739565s) [noise=true , address=45.91.116.13:4156 , piece_id=2L6LQKALJCYD5RQ7SGRRDZP6BROA72REMBX6MKZSCBOUZTJOVVBQ, seg_pos={0 0}, upload=245226]: Long tail cancellation due to optimal threshold Dec 17 07:12:40.436357541 (279h43m40.307440586s) [noise=true , address=77.90.31.17:4773 , piece_id=NVICD5WRAIYRFLMSOWH5LNTKF74QSYBAUT7XLVDAQ5O3MDUOGA5A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:41348->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:41348->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41348->77.90.31.17:4773: use of closed network connection Dec 17 07:12:40.436368862 (279h43m40.307451907s) [noise=true , address=77.90.31.17:4773 , piece_id=NVICD5WRAIYRFLMSOWH5LNTKF74QSYBAUT7XLVDAQ5O3MDUOGA5A, seg_pos={0 0}, upload=245226]: Long tail cancellation due to optimal threshold Dec 17 07:12:40.436437392 (279h43m40.307520437s) [noise=true , address=45.91.116.8:4046 , piece_id=QUKL5Q3ONBRNEOM5TCN5RVCEX5BGTNTUMG5CCZSMCFBMBQIROXYA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 17 07:12:40.436443625 (279h43m40.30752666s) [noise=true , address=45.91.116.8:4046 , piece_id=QUKL5Q3ONBRNEOM5TCN5RVCEX5BGTNTUMG5CCZSMCFBMBQIROXYA, seg_pos={0 0}, upload=245226]: Long tail cancellation due to optimal threshold Dec 17 07:12:40.436459425 (279h43m40.30754247s) [seg_pos={0 0}, upload=245226]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 07:12:40.436538986 (279h43m40.307622021s) [upload=245226]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 07:12:40.436544858 (279h43m40.307627903s) [upload=245226]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 07:12:40.725332982 (279h43m40.596416017s) [upload=245226]: Done uploading segments... Dec 17 07:12:40.725352159 (279h43m40.596435194s) [upload=245226]: Upload finished. err: <nil> Duration: 5.96916849s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 4.807050267s trace: https://jaeger.dev.storj.io/trace/6d70ed7853c345d3 FAIL (2025-12-17 07:54:26 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.807080395s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 4.992540037s trace: https://jaeger.dev.storj.io/trace/6d70ed7853c345d3 FAIL (2025-12-17 07:54:08 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.992572528s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.77288437s trace: https://jaeger.dev.storj.io/trace/47e835adff344c8 FAIL (2025-12-17 07:53:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: MDYFTTVCCAUKQDEIFOYSFBFJCCAM72YEY7MV762OVDCCXBJWQF7A) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: EHNNMCVG6QW5AI6IB3WFOWDDQHP6GMVQM6XFSIIDWMMS6DEDIW6A) context deadline exceeded ======LOGS====== ec 17 07:12:37.946668541 (279h43m37.817751586s) [upload=245226]: Starting upload Dec 17 07:12:37.946758152 (279h43m37.817841197s) [upload=245226]: Done starting upload Dec 17 07:12:37.947199001 (279h43m37.818282066s) [upload=245226]: Uploading segments... Dec 17 07:12:37.947253795 (279h43m37.81833685s) [upload=245226]: Got next segment. Inline: false Dec 17 07:12:37.947295084 (279h43m37.818378129s) [upload=245226]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 07:12:37.947301105 (279h43m37.81838414s) [upload=245226]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 07:12:38.191736294 (279h43m38.062819349s) [seg_pos={0 0}, upload=245226]: Begin upload segment... Dec 17 07:12:38.191795317 (279h43m38.062878362s) [seg_pos={0 0}, upload=245226]: Done begin upload segment. Dec 17 07:12:38.191819202 (279h43m38.062902238s) [noise=true , address=145.79.227.25:4618 , piece_id=C6JEP3RAPX22ECSDSRZL7CFRIDP2U2ZNRSWPVMJRF742CPGZBGPA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.191830183 (279h43m38.062913228s) [noise=true , address=77.90.31.17:4773 , piece_id=NVICD5WRAIYRFLMSOWH5LNTKF74QSYBAUT7XLVDAQ5O3MDUOGA5A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.192400609 (279h43m38.063483654s) [noise=true , address=145.79.227.52:4446 , piece_id=FYI226BYJJ3OA4NYEC6LELDWV5EPDQIE6OVAMJV2BTXNNXNWZPQA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193213818 (279h43m38.064296853s) [noise=true , address=45.91.116.8:4046 , piece_id=QUKL5Q3ONBRNEOM5TCN5RVCEX5BGTNTUMG5CCZSMCFBMBQIROXYA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193605724 (279h43m38.064688769s) [noise=true , address=145.79.227.16:4911 , piece_id=XZSCHKVU7TN7GDL33AZ6GXE5VLUCFZRP3G7ULXNN5GUQRTA6EZ7A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193689663 (279h43m38.064772709s) [noise=true , address=145.79.227.42:4121 , piece_id=RLGJ3MHH52EQJGUYKVBF5DLX5EUQ3MSYBVE7VROLOMAUEXOYT4QQ, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193977442 (279h43m38.065060487s) [noise=true , address=77.90.31.14:4662 , piece_id=MDYFTTVCCAUKQDEIFOYSFBFJCCAM72YEY7MV762OVDCCXBJWQF7A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194245692 (279h43m38.065328737s) [noise=true , address=45.91.116.13:4156 , piece_id=2L6LQKALJCYD5RQ7SGRRDZP6BROA72REMBX6MKZSCBOUZTJOVVBQ, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194366663 (279h43m38.065449698s) [noise=true , address=45.91.116.6:4974 , piece_id=EHNNMCVG6QW5AI6IB3WFOWDDQHP6GMVQM6XFSIIDWMMS6DEDIW6A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194740605 (279h43m38.06582364s) [noise=true , address=145.79.227.38:4979 , piece_id=MILXDFE5TH2N22E76WKJRID7UPASBSTKNUBNEQSN5BD32LPLRCUA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194863168 (279h43m38.065946213s) [noise=true , address=145.79.227.36:4519 , piece_id=HF4YFXYBONFXGYDQ6TJWJMNFHOP57PXKKMCGKOLRJJYEVL7EZS2A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195047018 (279h43m38.066130053s) [noise=true , address=145.79.227.48:4302 , piece_id=3VMJMBLA2BHO4VUCKSSPAMC26IFPO3IO5TKY7OLGYDB7UVIAPQ5Q, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195387467 (279h43m38.066470512s) [noise=true , address=145.79.227.50:4374 , piece_id=RQY5LFQW2XNR6QD5LQF6YZCRCJAY6IAAZPXGYJJGXI2PZFUPPG6A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195413807 (279h43m38.066496842s) [noise=true , address=145.79.227.41:4086 , piece_id=RRV7U2CFFJEZLCEBUGBJNQ6VF7HFKEEY6BWAJTUS645L6NKZFBBQ, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195752793 (279h43m38.066835828s) [noise=true , address=145.79.227.37:4565 , piece_id=LKHAWS2BR622N3KLSHASDEIKG5HETZ2GTPEFK4KGQYJKDPHD2XYA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195926243 (279h43m38.067009288s) [noise=true , address=145.79.227.11:4732 , piece_id=HOYU2ZK34RZWREKJQSZNSFZSNPX6U3U7CNDG2OT6FDFFCBLMZSNA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:39.894104052 (279h43m39.765187097s) [upload=245226]: Next returned nil segment Dec 17 07:12:39.894125211 (279h43m39.765208246s) [upload=245226]: Waiting for error group managing segments... Dec 17 07:12:40.182872669 (279h43m40.053955705s) [noise=true , address=145.79.227.38:4979 , piece_id=MILXDFE5TH2N22E76WKJRID7UPASBSTKNUBNEQSN5BD32LPLRCUA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.187264572 (279h43m40.058347617s) [noise=true , address=145.79.227.41:4086 , piece_id=RRV7U2CFFJEZLCEBUGBJNQ6VF7HFKEEY6BWAJTUS645L6NKZFBBQ, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.187588328 (279h43m40.058671373s) [noise=true , address=145.79.227.52:4446 , piece_id=FYI226BYJJ3OA4NYEC6LELDWV5EPDQIE6OVAMJV2BTXNNXNWZPQA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.201386834 (279h43m40.072469869s) [noise=true , address=145.79.227.16:4911 , piece_id=XZSCHKVU7TN7GDL33AZ6GXE5VLUCFZRP3G7ULXNN5GUQRTA6EZ7A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.207224941 (279h43m40.078307976s) [noise=true , address=145.79.227.37:4565 , piece_id=LKHAWS2BR622N3KLSHASDEIKG5HETZ2GTPEFK4KGQYJKDPHD2XYA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.209900164 (279h43m40.080983209s) [noise=true , address=145.79.227.11:4732 , piece_id=HOYU2ZK34RZWREKJQSZNSFZSNPX6U3U7CNDG2OT6FDFFCBLMZSNA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.209956011 (279h43m40.081039046s) [noise=true , address=145.79.227.25:4618 , piece_id=C6JEP3RAPX22ECSDSRZL7CFRIDP2U2ZNRSWPVMJRF742CPGZBGPA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.211720020 (279h43m40.082803065s) [noise=true , address=145.79.227.48:4302 , piece_id=3VMJMBLA2BHO4VUCKSSPAMC26IFPO3IO5TKY7OLGYDB7UVIAPQ5Q, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.214865549 (279h43m40.085948594s) [noise=true , address=145.79.227.36:4519 , piece_id=HF4YFXYBONFXGYDQ6TJWJMNFHOP57PXKKMCGKOLRJJYEVL7EZS2A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.246003612 (279h43m40.117086657s) [noise=true , address=145.79.227.42:4121 , piece_id=RLGJ3MHH52EQJGUYKVBF5DLX5EUQ3MSYBVE7VROLOMAUEXOYT4QQ, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.277886172 (279h43m40.148969217s) [noise=true , address=145.79.227.50:4374 , piece_id=RQY5LFQW2XNR6QD5LQF6YZCRCJAY6IAAZPXGYJJGXI2PZFUPPG6A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.436062099 (279h43m40.307145134s) [noise=true , address=45.91.116.6:4974 , piece_id=EHNNMCVG6QW5AI6IB3WFOWDDQHP6GMVQM6XFSIIDWMMS6DEDIW6A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.436076746 (279h43m40.307159781s) [noise=true , address=77.90.31.14:4662 , piece_id=MDYFTTVCCAUKQDEIFOYSFBFJCCAM72YEY7MV762OVDCCXBJWQF7A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.436088268 (279h43m40.307171303s) [seg_pos={0 0}, upload=245226]: Segment reached optimal threshold of 13 pieces. Dec 17 07:12:40.436303448 (279h43m40.307386493s) [noise=true , address=45.91.116.13:4156 , piece_id=2L6LQKALJCYD5RQ7SGRRDZP6BROA72REMBX6MKZSCBOUZTJOVVBQ, seg_pos={0 0}, upload=245226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 17 07:12:40.436312615 (279h43m40.30739565s) [noise=true , address=45.91.116.13:4156 , piece_id=2L6LQKALJCYD5RQ7SGRRDZP6BROA72REMBX6MKZSCBOUZTJOVVBQ, seg_pos={0 0}, upload=245226]: Long tail cancellation due to optimal threshold Dec 17 07:12:40.436357541 (279h43m40.307440586s) [noise=true , address=77.90.31.17:4773 , piece_id=NVICD5WRAIYRFLMSOWH5LNTKF74QSYBAUT7XLVDAQ5O3MDUOGA5A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:41348->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:41348->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41348->77.90.31.17:4773: use of closed network connection Dec 17 07:12:40.436368862 (279h43m40.307451907s) [noise=true , address=77.90.31.17:4773 , piece_id=NVICD5WRAIYRFLMSOWH5LNTKF74QSYBAUT7XLVDAQ5O3MDUOGA5A, seg_pos={0 0}, upload=245226]: Long tail cancellation due to optimal threshold Dec 17 07:12:40.436437392 (279h43m40.307520437s) [noise=true , address=45.91.116.8:4046 , piece_id=QUKL5Q3ONBRNEOM5TCN5RVCEX5BGTNTUMG5CCZSMCFBMBQIROXYA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 17 07:12:40.436443625 (279h43m40.30752666s) [noise=true , address=45.91.116.8:4046 , piece_id=QUKL5Q3ONBRNEOM5TCN5RVCEX5BGTNTUMG5CCZSMCFBMBQIROXYA, seg_pos={0 0}, upload=245226]: Long tail cancellation due to optimal threshold Dec 17 07:12:40.436459425 (279h43m40.30754247s) [seg_pos={0 0}, upload=245226]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 07:12:40.436538986 (279h43m40.307622021s) [upload=245226]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 07:12:40.436544858 (279h43m40.307627903s) [upload=245226]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 07:12:40.725332982 (279h43m40.596416017s) [upload=245226]: Done uploading segments... Dec 17 07:12:40.725352159 (279h43m40.596435194s) [upload=245226]: Upload finished. err: <nil> Duration: 5.772904509s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.839707424s trace: https://jaeger.dev.storj.io/trace/47e835adff344c8 FAIL (2025-12-17 07:53:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: MDYFTTVCCAUKQDEIFOYSFBFJCCAM72YEY7MV762OVDCCXBJWQF7A) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: EHNNMCVG6QW5AI6IB3WFOWDDQHP6GMVQM6XFSIIDWMMS6DEDIW6A) context deadline exceeded ======LOGS====== ec 17 07:12:37.946668541 (279h43m37.817751586s) [upload=245226]: Starting upload Dec 17 07:12:37.946758152 (279h43m37.817841197s) [upload=245226]: Done starting upload Dec 17 07:12:37.947199001 (279h43m37.818282066s) [upload=245226]: Uploading segments... Dec 17 07:12:37.947253795 (279h43m37.81833685s) [upload=245226]: Got next segment. Inline: false Dec 17 07:12:37.947295084 (279h43m37.818378129s) [upload=245226]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 07:12:37.947301105 (279h43m37.81838414s) [upload=245226]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 07:12:38.191736294 (279h43m38.062819349s) [seg_pos={0 0}, upload=245226]: Begin upload segment... Dec 17 07:12:38.191795317 (279h43m38.062878362s) [seg_pos={0 0}, upload=245226]: Done begin upload segment. Dec 17 07:12:38.191819202 (279h43m38.062902238s) [noise=true , address=145.79.227.25:4618 , piece_id=C6JEP3RAPX22ECSDSRZL7CFRIDP2U2ZNRSWPVMJRF742CPGZBGPA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.191830183 (279h43m38.062913228s) [noise=true , address=77.90.31.17:4773 , piece_id=NVICD5WRAIYRFLMSOWH5LNTKF74QSYBAUT7XLVDAQ5O3MDUOGA5A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.192400609 (279h43m38.063483654s) [noise=true , address=145.79.227.52:4446 , piece_id=FYI226BYJJ3OA4NYEC6LELDWV5EPDQIE6OVAMJV2BTXNNXNWZPQA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193213818 (279h43m38.064296853s) [noise=true , address=45.91.116.8:4046 , piece_id=QUKL5Q3ONBRNEOM5TCN5RVCEX5BGTNTUMG5CCZSMCFBMBQIROXYA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193605724 (279h43m38.064688769s) [noise=true , address=145.79.227.16:4911 , piece_id=XZSCHKVU7TN7GDL33AZ6GXE5VLUCFZRP3G7ULXNN5GUQRTA6EZ7A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193689663 (279h43m38.064772709s) [noise=true , address=145.79.227.42:4121 , piece_id=RLGJ3MHH52EQJGUYKVBF5DLX5EUQ3MSYBVE7VROLOMAUEXOYT4QQ, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193977442 (279h43m38.065060487s) [noise=true , address=77.90.31.14:4662 , piece_id=MDYFTTVCCAUKQDEIFOYSFBFJCCAM72YEY7MV762OVDCCXBJWQF7A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194245692 (279h43m38.065328737s) [noise=true , address=45.91.116.13:4156 , piece_id=2L6LQKALJCYD5RQ7SGRRDZP6BROA72REMBX6MKZSCBOUZTJOVVBQ, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194366663 (279h43m38.065449698s) [noise=true , address=45.91.116.6:4974 , piece_id=EHNNMCVG6QW5AI6IB3WFOWDDQHP6GMVQM6XFSIIDWMMS6DEDIW6A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194740605 (279h43m38.06582364s) [noise=true , address=145.79.227.38:4979 , piece_id=MILXDFE5TH2N22E76WKJRID7UPASBSTKNUBNEQSN5BD32LPLRCUA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194863168 (279h43m38.065946213s) [noise=true , address=145.79.227.36:4519 , piece_id=HF4YFXYBONFXGYDQ6TJWJMNFHOP57PXKKMCGKOLRJJYEVL7EZS2A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195047018 (279h43m38.066130053s) [noise=true , address=145.79.227.48:4302 , piece_id=3VMJMBLA2BHO4VUCKSSPAMC26IFPO3IO5TKY7OLGYDB7UVIAPQ5Q, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195387467 (279h43m38.066470512s) [noise=true , address=145.79.227.50:4374 , piece_id=RQY5LFQW2XNR6QD5LQF6YZCRCJAY6IAAZPXGYJJGXI2PZFUPPG6A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195413807 (279h43m38.066496842s) [noise=true , address=145.79.227.41:4086 , piece_id=RRV7U2CFFJEZLCEBUGBJNQ6VF7HFKEEY6BWAJTUS645L6NKZFBBQ, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195752793 (279h43m38.066835828s) [noise=true , address=145.79.227.37:4565 , piece_id=LKHAWS2BR622N3KLSHASDEIKG5HETZ2GTPEFK4KGQYJKDPHD2XYA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195926243 (279h43m38.067009288s) [noise=true , address=145.79.227.11:4732 , piece_id=HOYU2ZK34RZWREKJQSZNSFZSNPX6U3U7CNDG2OT6FDFFCBLMZSNA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:39.894104052 (279h43m39.765187097s) [upload=245226]: Next returned nil segment Dec 17 07:12:39.894125211 (279h43m39.765208246s) [upload=245226]: Waiting for error group managing segments... Dec 17 07:12:40.182872669 (279h43m40.053955705s) [noise=true , address=145.79.227.38:4979 , piece_id=MILXDFE5TH2N22E76WKJRID7UPASBSTKNUBNEQSN5BD32LPLRCUA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.187264572 (279h43m40.058347617s) [noise=true , address=145.79.227.41:4086 , piece_id=RRV7U2CFFJEZLCEBUGBJNQ6VF7HFKEEY6BWAJTUS645L6NKZFBBQ, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.187588328 (279h43m40.058671373s) [noise=true , address=145.79.227.52:4446 , piece_id=FYI226BYJJ3OA4NYEC6LELDWV5EPDQIE6OVAMJV2BTXNNXNWZPQA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.201386834 (279h43m40.072469869s) [noise=true , address=145.79.227.16:4911 , piece_id=XZSCHKVU7TN7GDL33AZ6GXE5VLUCFZRP3G7ULXNN5GUQRTA6EZ7A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.207224941 (279h43m40.078307976s) [noise=true , address=145.79.227.37:4565 , piece_id=LKHAWS2BR622N3KLSHASDEIKG5HETZ2GTPEFK4KGQYJKDPHD2XYA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.209900164 (279h43m40.080983209s) [noise=true , address=145.79.227.11:4732 , piece_id=HOYU2ZK34RZWREKJQSZNSFZSNPX6U3U7CNDG2OT6FDFFCBLMZSNA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.209956011 (279h43m40.081039046s) [noise=true , address=145.79.227.25:4618 , piece_id=C6JEP3RAPX22ECSDSRZL7CFRIDP2U2ZNRSWPVMJRF742CPGZBGPA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.211720020 (279h43m40.082803065s) [noise=true , address=145.79.227.48:4302 , piece_id=3VMJMBLA2BHO4VUCKSSPAMC26IFPO3IO5TKY7OLGYDB7UVIAPQ5Q, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.214865549 (279h43m40.085948594s) [noise=true , address=145.79.227.36:4519 , piece_id=HF4YFXYBONFXGYDQ6TJWJMNFHOP57PXKKMCGKOLRJJYEVL7EZS2A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.246003612 (279h43m40.117086657s) [noise=true , address=145.79.227.42:4121 , piece_id=RLGJ3MHH52EQJGUYKVBF5DLX5EUQ3MSYBVE7VROLOMAUEXOYT4QQ, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.277886172 (279h43m40.148969217s) [noise=true , address=145.79.227.50:4374 , piece_id=RQY5LFQW2XNR6QD5LQF6YZCRCJAY6IAAZPXGYJJGXI2PZFUPPG6A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.436062099 (279h43m40.307145134s) [noise=true , address=45.91.116.6:4974 , piece_id=EHNNMCVG6QW5AI6IB3WFOWDDQHP6GMVQM6XFSIIDWMMS6DEDIW6A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.436076746 (279h43m40.307159781s) [noise=true , address=77.90.31.14:4662 , piece_id=MDYFTTVCCAUKQDEIFOYSFBFJCCAM72YEY7MV762OVDCCXBJWQF7A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.436088268 (279h43m40.307171303s) [seg_pos={0 0}, upload=245226]: Segment reached optimal threshold of 13 pieces. Dec 17 07:12:40.436303448 (279h43m40.307386493s) [noise=true , address=45.91.116.13:4156 , piece_id=2L6LQKALJCYD5RQ7SGRRDZP6BROA72REMBX6MKZSCBOUZTJOVVBQ, seg_pos={0 0}, upload=245226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 17 07:12:40.436312615 (279h43m40.30739565s) [noise=true , address=45.91.116.13:4156 , piece_id=2L6LQKALJCYD5RQ7SGRRDZP6BROA72REMBX6MKZSCBOUZTJOVVBQ, seg_pos={0 0}, upload=245226]: Long tail cancellation due to optimal threshold Dec 17 07:12:40.436357541 (279h43m40.307440586s) [noise=true , address=77.90.31.17:4773 , piece_id=NVICD5WRAIYRFLMSOWH5LNTKF74QSYBAUT7XLVDAQ5O3MDUOGA5A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:41348->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:41348->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41348->77.90.31.17:4773: use of closed network connection Dec 17 07:12:40.436368862 (279h43m40.307451907s) [noise=true , address=77.90.31.17:4773 , piece_id=NVICD5WRAIYRFLMSOWH5LNTKF74QSYBAUT7XLVDAQ5O3MDUOGA5A, seg_pos={0 0}, upload=245226]: Long tail cancellation due to optimal threshold Dec 17 07:12:40.436437392 (279h43m40.307520437s) [noise=true , address=45.91.116.8:4046 , piece_id=QUKL5Q3ONBRNEOM5TCN5RVCEX5BGTNTUMG5CCZSMCFBMBQIROXYA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 17 07:12:40.436443625 (279h43m40.30752666s) [noise=true , address=45.91.116.8:4046 , piece_id=QUKL5Q3ONBRNEOM5TCN5RVCEX5BGTNTUMG5CCZSMCFBMBQIROXYA, seg_pos={0 0}, upload=245226]: Long tail cancellation due to optimal threshold Dec 17 07:12:40.436459425 (279h43m40.30754247s) [seg_pos={0 0}, upload=245226]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 07:12:40.436538986 (279h43m40.307622021s) [upload=245226]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 07:12:40.436544858 (279h43m40.307627903s) [upload=245226]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 07:12:40.725332982 (279h43m40.596416017s) [upload=245226]: Done uploading segments... Dec 17 07:12:40.725352159 (279h43m40.596435194s) [upload=245226]: Upload finished. err: <nil> Duration: 5.839736239s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 4.782568182s trace: https://jaeger.dev.storj.io/trace/6d70ed7853c345d3 FAIL (2025-12-17 07:53:27 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.782595384s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.823471097s trace: https://jaeger.dev.storj.io/trace/47e835adff344c8 FAIL (2025-12-17 07:52:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: MDYFTTVCCAUKQDEIFOYSFBFJCCAM72YEY7MV762OVDCCXBJWQF7A) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: EHNNMCVG6QW5AI6IB3WFOWDDQHP6GMVQM6XFSIIDWMMS6DEDIW6A) context deadline exceeded ======LOGS====== ec 17 07:12:37.946668541 (279h43m37.817751586s) [upload=245226]: Starting upload Dec 17 07:12:37.946758152 (279h43m37.817841197s) [upload=245226]: Done starting upload Dec 17 07:12:37.947199001 (279h43m37.818282066s) [upload=245226]: Uploading segments... Dec 17 07:12:37.947253795 (279h43m37.81833685s) [upload=245226]: Got next segment. Inline: false Dec 17 07:12:37.947295084 (279h43m37.818378129s) [upload=245226]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 07:12:37.947301105 (279h43m37.81838414s) [upload=245226]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 07:12:38.191736294 (279h43m38.062819349s) [seg_pos={0 0}, upload=245226]: Begin upload segment... Dec 17 07:12:38.191795317 (279h43m38.062878362s) [seg_pos={0 0}, upload=245226]: Done begin upload segment. Dec 17 07:12:38.191819202 (279h43m38.062902238s) [noise=true , address=145.79.227.25:4618 , piece_id=C6JEP3RAPX22ECSDSRZL7CFRIDP2U2ZNRSWPVMJRF742CPGZBGPA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.191830183 (279h43m38.062913228s) [noise=true , address=77.90.31.17:4773 , piece_id=NVICD5WRAIYRFLMSOWH5LNTKF74QSYBAUT7XLVDAQ5O3MDUOGA5A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.192400609 (279h43m38.063483654s) [noise=true , address=145.79.227.52:4446 , piece_id=FYI226BYJJ3OA4NYEC6LELDWV5EPDQIE6OVAMJV2BTXNNXNWZPQA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193213818 (279h43m38.064296853s) [noise=true , address=45.91.116.8:4046 , piece_id=QUKL5Q3ONBRNEOM5TCN5RVCEX5BGTNTUMG5CCZSMCFBMBQIROXYA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193605724 (279h43m38.064688769s) [noise=true , address=145.79.227.16:4911 , piece_id=XZSCHKVU7TN7GDL33AZ6GXE5VLUCFZRP3G7ULXNN5GUQRTA6EZ7A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193689663 (279h43m38.064772709s) [noise=true , address=145.79.227.42:4121 , piece_id=RLGJ3MHH52EQJGUYKVBF5DLX5EUQ3MSYBVE7VROLOMAUEXOYT4QQ, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193977442 (279h43m38.065060487s) [noise=true , address=77.90.31.14:4662 , piece_id=MDYFTTVCCAUKQDEIFOYSFBFJCCAM72YEY7MV762OVDCCXBJWQF7A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194245692 (279h43m38.065328737s) [noise=true , address=45.91.116.13:4156 , piece_id=2L6LQKALJCYD5RQ7SGRRDZP6BROA72REMBX6MKZSCBOUZTJOVVBQ, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194366663 (279h43m38.065449698s) [noise=true , address=45.91.116.6:4974 , piece_id=EHNNMCVG6QW5AI6IB3WFOWDDQHP6GMVQM6XFSIIDWMMS6DEDIW6A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194740605 (279h43m38.06582364s) [noise=true , address=145.79.227.38:4979 , piece_id=MILXDFE5TH2N22E76WKJRID7UPASBSTKNUBNEQSN5BD32LPLRCUA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194863168 (279h43m38.065946213s) [noise=true , address=145.79.227.36:4519 , piece_id=HF4YFXYBONFXGYDQ6TJWJMNFHOP57PXKKMCGKOLRJJYEVL7EZS2A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195047018 (279h43m38.066130053s) [noise=true , address=145.79.227.48:4302 , piece_id=3VMJMBLA2BHO4VUCKSSPAMC26IFPO3IO5TKY7OLGYDB7UVIAPQ5Q, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195387467 (279h43m38.066470512s) [noise=true , address=145.79.227.50:4374 , piece_id=RQY5LFQW2XNR6QD5LQF6YZCRCJAY6IAAZPXGYJJGXI2PZFUPPG6A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195413807 (279h43m38.066496842s) [noise=true , address=145.79.227.41:4086 , piece_id=RRV7U2CFFJEZLCEBUGBJNQ6VF7HFKEEY6BWAJTUS645L6NKZFBBQ, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195752793 (279h43m38.066835828s) [noise=true , address=145.79.227.37:4565 , piece_id=LKHAWS2BR622N3KLSHASDEIKG5HETZ2GTPEFK4KGQYJKDPHD2XYA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195926243 (279h43m38.067009288s) [noise=true , address=145.79.227.11:4732 , piece_id=HOYU2ZK34RZWREKJQSZNSFZSNPX6U3U7CNDG2OT6FDFFCBLMZSNA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:39.894104052 (279h43m39.765187097s) [upload=245226]: Next returned nil segment Dec 17 07:12:39.894125211 (279h43m39.765208246s) [upload=245226]: Waiting for error group managing segments... Dec 17 07:12:40.182872669 (279h43m40.053955705s) [noise=true , address=145.79.227.38:4979 , piece_id=MILXDFE5TH2N22E76WKJRID7UPASBSTKNUBNEQSN5BD32LPLRCUA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.187264572 (279h43m40.058347617s) [noise=true , address=145.79.227.41:4086 , piece_id=RRV7U2CFFJEZLCEBUGBJNQ6VF7HFKEEY6BWAJTUS645L6NKZFBBQ, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.187588328 (279h43m40.058671373s) [noise=true , address=145.79.227.52:4446 , piece_id=FYI226BYJJ3OA4NYEC6LELDWV5EPDQIE6OVAMJV2BTXNNXNWZPQA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.201386834 (279h43m40.072469869s) [noise=true , address=145.79.227.16:4911 , piece_id=XZSCHKVU7TN7GDL33AZ6GXE5VLUCFZRP3G7ULXNN5GUQRTA6EZ7A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.207224941 (279h43m40.078307976s) [noise=true , address=145.79.227.37:4565 , piece_id=LKHAWS2BR622N3KLSHASDEIKG5HETZ2GTPEFK4KGQYJKDPHD2XYA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.209900164 (279h43m40.080983209s) [noise=true , address=145.79.227.11:4732 , piece_id=HOYU2ZK34RZWREKJQSZNSFZSNPX6U3U7CNDG2OT6FDFFCBLMZSNA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.209956011 (279h43m40.081039046s) [noise=true , address=145.79.227.25:4618 , piece_id=C6JEP3RAPX22ECSDSRZL7CFRIDP2U2ZNRSWPVMJRF742CPGZBGPA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.211720020 (279h43m40.082803065s) [noise=true , address=145.79.227.48:4302 , piece_id=3VMJMBLA2BHO4VUCKSSPAMC26IFPO3IO5TKY7OLGYDB7UVIAPQ5Q, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.214865549 (279h43m40.085948594s) [noise=true , address=145.79.227.36:4519 , piece_id=HF4YFXYBONFXGYDQ6TJWJMNFHOP57PXKKMCGKOLRJJYEVL7EZS2A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.246003612 (279h43m40.117086657s) [noise=true , address=145.79.227.42:4121 , piece_id=RLGJ3MHH52EQJGUYKVBF5DLX5EUQ3MSYBVE7VROLOMAUEXOYT4QQ, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.277886172 (279h43m40.148969217s) [noise=true , address=145.79.227.50:4374 , piece_id=RQY5LFQW2XNR6QD5LQF6YZCRCJAY6IAAZPXGYJJGXI2PZFUPPG6A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.436062099 (279h43m40.307145134s) [noise=true , address=45.91.116.6:4974 , piece_id=EHNNMCVG6QW5AI6IB3WFOWDDQHP6GMVQM6XFSIIDWMMS6DEDIW6A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.436076746 (279h43m40.307159781s) [noise=true , address=77.90.31.14:4662 , piece_id=MDYFTTVCCAUKQDEIFOYSFBFJCCAM72YEY7MV762OVDCCXBJWQF7A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.436088268 (279h43m40.307171303s) [seg_pos={0 0}, upload=245226]: Segment reached optimal threshold of 13 pieces. Dec 17 07:12:40.436303448 (279h43m40.307386493s) [noise=true , address=45.91.116.13:4156 , piece_id=2L6LQKALJCYD5RQ7SGRRDZP6BROA72REMBX6MKZSCBOUZTJOVVBQ, seg_pos={0 0}, upload=245226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 17 07:12:40.436312615 (279h43m40.30739565s) [noise=true , address=45.91.116.13:4156 , piece_id=2L6LQKALJCYD5RQ7SGRRDZP6BROA72REMBX6MKZSCBOUZTJOVVBQ, seg_pos={0 0}, upload=245226]: Long tail cancellation due to optimal threshold Dec 17 07:12:40.436357541 (279h43m40.307440586s) [noise=true , address=77.90.31.17:4773 , piece_id=NVICD5WRAIYRFLMSOWH5LNTKF74QSYBAUT7XLVDAQ5O3MDUOGA5A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:41348->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:41348->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41348->77.90.31.17:4773: use of closed network connection Dec 17 07:12:40.436368862 (279h43m40.307451907s) [noise=true , address=77.90.31.17:4773 , piece_id=NVICD5WRAIYRFLMSOWH5LNTKF74QSYBAUT7XLVDAQ5O3MDUOGA5A, seg_pos={0 0}, upload=245226]: Long tail cancellation due to optimal threshold Dec 17 07:12:40.436437392 (279h43m40.307520437s) [noise=true , address=45.91.116.8:4046 , piece_id=QUKL5Q3ONBRNEOM5TCN5RVCEX5BGTNTUMG5CCZSMCFBMBQIROXYA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 17 07:12:40.436443625 (279h43m40.30752666s) [noise=true , address=45.91.116.8:4046 , piece_id=QUKL5Q3ONBRNEOM5TCN5RVCEX5BGTNTUMG5CCZSMCFBMBQIROXYA, seg_pos={0 0}, upload=245226]: Long tail cancellation due to optimal threshold Dec 17 07:12:40.436459425 (279h43m40.30754247s) [seg_pos={0 0}, upload=245226]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 07:12:40.436538986 (279h43m40.307622021s) [upload=245226]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 07:12:40.436544858 (279h43m40.307627903s) [upload=245226]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 07:12:40.725332982 (279h43m40.596416017s) [upload=245226]: Done uploading segments... Dec 17 07:12:40.725352159 (279h43m40.596435194s) [upload=245226]: Upload finished. err: <nil> Duration: 5.823522685s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 4.928779684s trace: https://jaeger.dev.storj.io/trace/6d70ed7853c345d3 FAIL (2025-12-17 07:52:19 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.928809631s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.307266182s trace: https://jaeger.dev.storj.io/trace/47e835adff344c8 FAIL (2025-12-17 07:51:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: MDYFTTVCCAUKQDEIFOYSFBFJCCAM72YEY7MV762OVDCCXBJWQF7A) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: EHNNMCVG6QW5AI6IB3WFOWDDQHP6GMVQM6XFSIIDWMMS6DEDIW6A) context deadline exceeded ======LOGS====== ec 17 07:12:37.946668541 (279h43m37.817751586s) [upload=245226]: Starting upload Dec 17 07:12:37.946758152 (279h43m37.817841197s) [upload=245226]: Done starting upload Dec 17 07:12:37.947199001 (279h43m37.818282066s) [upload=245226]: Uploading segments... Dec 17 07:12:37.947253795 (279h43m37.81833685s) [upload=245226]: Got next segment. Inline: false Dec 17 07:12:37.947295084 (279h43m37.818378129s) [upload=245226]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 07:12:37.947301105 (279h43m37.81838414s) [upload=245226]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 07:12:38.191736294 (279h43m38.062819349s) [seg_pos={0 0}, upload=245226]: Begin upload segment... Dec 17 07:12:38.191795317 (279h43m38.062878362s) [seg_pos={0 0}, upload=245226]: Done begin upload segment. Dec 17 07:12:38.191819202 (279h43m38.062902238s) [noise=true , address=145.79.227.25:4618 , piece_id=C6JEP3RAPX22ECSDSRZL7CFRIDP2U2ZNRSWPVMJRF742CPGZBGPA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.191830183 (279h43m38.062913228s) [noise=true , address=77.90.31.17:4773 , piece_id=NVICD5WRAIYRFLMSOWH5LNTKF74QSYBAUT7XLVDAQ5O3MDUOGA5A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.192400609 (279h43m38.063483654s) [noise=true , address=145.79.227.52:4446 , piece_id=FYI226BYJJ3OA4NYEC6LELDWV5EPDQIE6OVAMJV2BTXNNXNWZPQA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193213818 (279h43m38.064296853s) [noise=true , address=45.91.116.8:4046 , piece_id=QUKL5Q3ONBRNEOM5TCN5RVCEX5BGTNTUMG5CCZSMCFBMBQIROXYA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193605724 (279h43m38.064688769s) [noise=true , address=145.79.227.16:4911 , piece_id=XZSCHKVU7TN7GDL33AZ6GXE5VLUCFZRP3G7ULXNN5GUQRTA6EZ7A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193689663 (279h43m38.064772709s) [noise=true , address=145.79.227.42:4121 , piece_id=RLGJ3MHH52EQJGUYKVBF5DLX5EUQ3MSYBVE7VROLOMAUEXOYT4QQ, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193977442 (279h43m38.065060487s) [noise=true , address=77.90.31.14:4662 , piece_id=MDYFTTVCCAUKQDEIFOYSFBFJCCAM72YEY7MV762OVDCCXBJWQF7A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194245692 (279h43m38.065328737s) [noise=true , address=45.91.116.13:4156 , piece_id=2L6LQKALJCYD5RQ7SGRRDZP6BROA72REMBX6MKZSCBOUZTJOVVBQ, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194366663 (279h43m38.065449698s) [noise=true , address=45.91.116.6:4974 , piece_id=EHNNMCVG6QW5AI6IB3WFOWDDQHP6GMVQM6XFSIIDWMMS6DEDIW6A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194740605 (279h43m38.06582364s) [noise=true , address=145.79.227.38:4979 , piece_id=MILXDFE5TH2N22E76WKJRID7UPASBSTKNUBNEQSN5BD32LPLRCUA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194863168 (279h43m38.065946213s) [noise=true , address=145.79.227.36:4519 , piece_id=HF4YFXYBONFXGYDQ6TJWJMNFHOP57PXKKMCGKOLRJJYEVL7EZS2A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195047018 (279h43m38.066130053s) [noise=true , address=145.79.227.48:4302 , piece_id=3VMJMBLA2BHO4VUCKSSPAMC26IFPO3IO5TKY7OLGYDB7UVIAPQ5Q, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195387467 (279h43m38.066470512s) [noise=true , address=145.79.227.50:4374 , piece_id=RQY5LFQW2XNR6QD5LQF6YZCRCJAY6IAAZPXGYJJGXI2PZFUPPG6A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195413807 (279h43m38.066496842s) [noise=true , address=145.79.227.41:4086 , piece_id=RRV7U2CFFJEZLCEBUGBJNQ6VF7HFKEEY6BWAJTUS645L6NKZFBBQ, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195752793 (279h43m38.066835828s) [noise=true , address=145.79.227.37:4565 , piece_id=LKHAWS2BR622N3KLSHASDEIKG5HETZ2GTPEFK4KGQYJKDPHD2XYA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195926243 (279h43m38.067009288s) [noise=true , address=145.79.227.11:4732 , piece_id=HOYU2ZK34RZWREKJQSZNSFZSNPX6U3U7CNDG2OT6FDFFCBLMZSNA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:39.894104052 (279h43m39.765187097s) [upload=245226]: Next returned nil segment Dec 17 07:12:39.894125211 (279h43m39.765208246s) [upload=245226]: Waiting for error group managing segments... Dec 17 07:12:40.182872669 (279h43m40.053955705s) [noise=true , address=145.79.227.38:4979 , piece_id=MILXDFE5TH2N22E76WKJRID7UPASBSTKNUBNEQSN5BD32LPLRCUA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.187264572 (279h43m40.058347617s) [noise=true , address=145.79.227.41:4086 , piece_id=RRV7U2CFFJEZLCEBUGBJNQ6VF7HFKEEY6BWAJTUS645L6NKZFBBQ, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.187588328 (279h43m40.058671373s) [noise=true , address=145.79.227.52:4446 , piece_id=FYI226BYJJ3OA4NYEC6LELDWV5EPDQIE6OVAMJV2BTXNNXNWZPQA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.201386834 (279h43m40.072469869s) [noise=true , address=145.79.227.16:4911 , piece_id=XZSCHKVU7TN7GDL33AZ6GXE5VLUCFZRP3G7ULXNN5GUQRTA6EZ7A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.207224941 (279h43m40.078307976s) [noise=true , address=145.79.227.37:4565 , piece_id=LKHAWS2BR622N3KLSHASDEIKG5HETZ2GTPEFK4KGQYJKDPHD2XYA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.209900164 (279h43m40.080983209s) [noise=true , address=145.79.227.11:4732 , piece_id=HOYU2ZK34RZWREKJQSZNSFZSNPX6U3U7CNDG2OT6FDFFCBLMZSNA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.209956011 (279h43m40.081039046s) [noise=true , address=145.79.227.25:4618 , piece_id=C6JEP3RAPX22ECSDSRZL7CFRIDP2U2ZNRSWPVMJRF742CPGZBGPA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.211720020 (279h43m40.082803065s) [noise=true , address=145.79.227.48:4302 , piece_id=3VMJMBLA2BHO4VUCKSSPAMC26IFPO3IO5TKY7OLGYDB7UVIAPQ5Q, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.214865549 (279h43m40.085948594s) [noise=true , address=145.79.227.36:4519 , piece_id=HF4YFXYBONFXGYDQ6TJWJMNFHOP57PXKKMCGKOLRJJYEVL7EZS2A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.246003612 (279h43m40.117086657s) [noise=true , address=145.79.227.42:4121 , piece_id=RLGJ3MHH52EQJGUYKVBF5DLX5EUQ3MSYBVE7VROLOMAUEXOYT4QQ, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.277886172 (279h43m40.148969217s) [noise=true , address=145.79.227.50:4374 , piece_id=RQY5LFQW2XNR6QD5LQF6YZCRCJAY6IAAZPXGYJJGXI2PZFUPPG6A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.436062099 (279h43m40.307145134s) [noise=true , address=45.91.116.6:4974 , piece_id=EHNNMCVG6QW5AI6IB3WFOWDDQHP6GMVQM6XFSIIDWMMS6DEDIW6A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.436076746 (279h43m40.307159781s) [noise=true , address=77.90.31.14:4662 , piece_id=MDYFTTVCCAUKQDEIFOYSFBFJCCAM72YEY7MV762OVDCCXBJWQF7A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.436088268 (279h43m40.307171303s) [seg_pos={0 0}, upload=245226]: Segment reached optimal threshold of 13 pieces. Dec 17 07:12:40.436303448 (279h43m40.307386493s) [noise=true , address=45.91.116.13:4156 , piece_id=2L6LQKALJCYD5RQ7SGRRDZP6BROA72REMBX6MKZSCBOUZTJOVVBQ, seg_pos={0 0}, upload=245226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 17 07:12:40.436312615 (279h43m40.30739565s) [noise=true , address=45.91.116.13:4156 , piece_id=2L6LQKALJCYD5RQ7SGRRDZP6BROA72REMBX6MKZSCBOUZTJOVVBQ, seg_pos={0 0}, upload=245226]: Long tail cancellation due to optimal threshold Dec 17 07:12:40.436357541 (279h43m40.307440586s) [noise=true , address=77.90.31.17:4773 , piece_id=NVICD5WRAIYRFLMSOWH5LNTKF74QSYBAUT7XLVDAQ5O3MDUOGA5A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:41348->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:41348->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41348->77.90.31.17:4773: use of closed network connection Dec 17 07:12:40.436368862 (279h43m40.307451907s) [noise=true , address=77.90.31.17:4773 , piece_id=NVICD5WRAIYRFLMSOWH5LNTKF74QSYBAUT7XLVDAQ5O3MDUOGA5A, seg_pos={0 0}, upload=245226]: Long tail cancellation due to optimal threshold Dec 17 07:12:40.436437392 (279h43m40.307520437s) [noise=true , address=45.91.116.8:4046 , piece_id=QUKL5Q3ONBRNEOM5TCN5RVCEX5BGTNTUMG5CCZSMCFBMBQIROXYA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 17 07:12:40.436443625 (279h43m40.30752666s) [noise=true , address=45.91.116.8:4046 , piece_id=QUKL5Q3ONBRNEOM5TCN5RVCEX5BGTNTUMG5CCZSMCFBMBQIROXYA, seg_pos={0 0}, upload=245226]: Long tail cancellation due to optimal threshold Dec 17 07:12:40.436459425 (279h43m40.30754247s) [seg_pos={0 0}, upload=245226]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 07:12:40.436538986 (279h43m40.307622021s) [upload=245226]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 07:12:40.436544858 (279h43m40.307627903s) [upload=245226]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 07:12:40.725332982 (279h43m40.596416017s) [upload=245226]: Done uploading segments... Dec 17 07:12:40.725352159 (279h43m40.596435194s) [upload=245226]: Upload finished. err: <nil> Duration: 6.307293324s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 4.879530455s trace: https://jaeger.dev.storj.io/trace/6d70ed7853c345d3 FAIL (2025-12-17 07:51:21 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.879559532s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.925875802s trace: https://jaeger.dev.storj.io/trace/47e835adff344c8 FAIL (2025-12-17 07:50:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: MDYFTTVCCAUKQDEIFOYSFBFJCCAM72YEY7MV762OVDCCXBJWQF7A) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: EHNNMCVG6QW5AI6IB3WFOWDDQHP6GMVQM6XFSIIDWMMS6DEDIW6A) context deadline exceeded ======LOGS====== ec 17 07:12:37.946668541 (279h43m37.817751586s) [upload=245226]: Starting upload Dec 17 07:12:37.946758152 (279h43m37.817841197s) [upload=245226]: Done starting upload Dec 17 07:12:37.947199001 (279h43m37.818282066s) [upload=245226]: Uploading segments... Dec 17 07:12:37.947253795 (279h43m37.81833685s) [upload=245226]: Got next segment. Inline: false Dec 17 07:12:37.947295084 (279h43m37.818378129s) [upload=245226]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 07:12:37.947301105 (279h43m37.81838414s) [upload=245226]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 07:12:38.191736294 (279h43m38.062819349s) [seg_pos={0 0}, upload=245226]: Begin upload segment... Dec 17 07:12:38.191795317 (279h43m38.062878362s) [seg_pos={0 0}, upload=245226]: Done begin upload segment. Dec 17 07:12:38.191819202 (279h43m38.062902238s) [noise=true , address=145.79.227.25:4618 , piece_id=C6JEP3RAPX22ECSDSRZL7CFRIDP2U2ZNRSWPVMJRF742CPGZBGPA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.191830183 (279h43m38.062913228s) [noise=true , address=77.90.31.17:4773 , piece_id=NVICD5WRAIYRFLMSOWH5LNTKF74QSYBAUT7XLVDAQ5O3MDUOGA5A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.192400609 (279h43m38.063483654s) [noise=true , address=145.79.227.52:4446 , piece_id=FYI226BYJJ3OA4NYEC6LELDWV5EPDQIE6OVAMJV2BTXNNXNWZPQA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193213818 (279h43m38.064296853s) [noise=true , address=45.91.116.8:4046 , piece_id=QUKL5Q3ONBRNEOM5TCN5RVCEX5BGTNTUMG5CCZSMCFBMBQIROXYA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193605724 (279h43m38.064688769s) [noise=true , address=145.79.227.16:4911 , piece_id=XZSCHKVU7TN7GDL33AZ6GXE5VLUCFZRP3G7ULXNN5GUQRTA6EZ7A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193689663 (279h43m38.064772709s) [noise=true , address=145.79.227.42:4121 , piece_id=RLGJ3MHH52EQJGUYKVBF5DLX5EUQ3MSYBVE7VROLOMAUEXOYT4QQ, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193977442 (279h43m38.065060487s) [noise=true , address=77.90.31.14:4662 , piece_id=MDYFTTVCCAUKQDEIFOYSFBFJCCAM72YEY7MV762OVDCCXBJWQF7A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194245692 (279h43m38.065328737s) [noise=true , address=45.91.116.13:4156 , piece_id=2L6LQKALJCYD5RQ7SGRRDZP6BROA72REMBX6MKZSCBOUZTJOVVBQ, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194366663 (279h43m38.065449698s) [noise=true , address=45.91.116.6:4974 , piece_id=EHNNMCVG6QW5AI6IB3WFOWDDQHP6GMVQM6XFSIIDWMMS6DEDIW6A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194740605 (279h43m38.06582364s) [noise=true , address=145.79.227.38:4979 , piece_id=MILXDFE5TH2N22E76WKJRID7UPASBSTKNUBNEQSN5BD32LPLRCUA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194863168 (279h43m38.065946213s) [noise=true , address=145.79.227.36:4519 , piece_id=HF4YFXYBONFXGYDQ6TJWJMNFHOP57PXKKMCGKOLRJJYEVL7EZS2A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195047018 (279h43m38.066130053s) [noise=true , address=145.79.227.48:4302 , piece_id=3VMJMBLA2BHO4VUCKSSPAMC26IFPO3IO5TKY7OLGYDB7UVIAPQ5Q, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195387467 (279h43m38.066470512s) [noise=true , address=145.79.227.50:4374 , piece_id=RQY5LFQW2XNR6QD5LQF6YZCRCJAY6IAAZPXGYJJGXI2PZFUPPG6A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195413807 (279h43m38.066496842s) [noise=true , address=145.79.227.41:4086 , piece_id=RRV7U2CFFJEZLCEBUGBJNQ6VF7HFKEEY6BWAJTUS645L6NKZFBBQ, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195752793 (279h43m38.066835828s) [noise=true , address=145.79.227.37:4565 , piece_id=LKHAWS2BR622N3KLSHASDEIKG5HETZ2GTPEFK4KGQYJKDPHD2XYA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195926243 (279h43m38.067009288s) [noise=true , address=145.79.227.11:4732 , piece_id=HOYU2ZK34RZWREKJQSZNSFZSNPX6U3U7CNDG2OT6FDFFCBLMZSNA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:39.894104052 (279h43m39.765187097s) [upload=245226]: Next returned nil segment Dec 17 07:12:39.894125211 (279h43m39.765208246s) [upload=245226]: Waiting for error group managing segments... Dec 17 07:12:40.182872669 (279h43m40.053955705s) [noise=true , address=145.79.227.38:4979 , piece_id=MILXDFE5TH2N22E76WKJRID7UPASBSTKNUBNEQSN5BD32LPLRCUA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.187264572 (279h43m40.058347617s) [noise=true , address=145.79.227.41:4086 , piece_id=RRV7U2CFFJEZLCEBUGBJNQ6VF7HFKEEY6BWAJTUS645L6NKZFBBQ, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.187588328 (279h43m40.058671373s) [noise=true , address=145.79.227.52:4446 , piece_id=FYI226BYJJ3OA4NYEC6LELDWV5EPDQIE6OVAMJV2BTXNNXNWZPQA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.201386834 (279h43m40.072469869s) [noise=true , address=145.79.227.16:4911 , piece_id=XZSCHKVU7TN7GDL33AZ6GXE5VLUCFZRP3G7ULXNN5GUQRTA6EZ7A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.207224941 (279h43m40.078307976s) [noise=true , address=145.79.227.37:4565 , piece_id=LKHAWS2BR622N3KLSHASDEIKG5HETZ2GTPEFK4KGQYJKDPHD2XYA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.209900164 (279h43m40.080983209s) [noise=true , address=145.79.227.11:4732 , piece_id=HOYU2ZK34RZWREKJQSZNSFZSNPX6U3U7CNDG2OT6FDFFCBLMZSNA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.209956011 (279h43m40.081039046s) [noise=true , address=145.79.227.25:4618 , piece_id=C6JEP3RAPX22ECSDSRZL7CFRIDP2U2ZNRSWPVMJRF742CPGZBGPA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.211720020 (279h43m40.082803065s) [noise=true , address=145.79.227.48:4302 , piece_id=3VMJMBLA2BHO4VUCKSSPAMC26IFPO3IO5TKY7OLGYDB7UVIAPQ5Q, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.214865549 (279h43m40.085948594s) [noise=true , address=145.79.227.36:4519 , piece_id=HF4YFXYBONFXGYDQ6TJWJMNFHOP57PXKKMCGKOLRJJYEVL7EZS2A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.246003612 (279h43m40.117086657s) [noise=true , address=145.79.227.42:4121 , piece_id=RLGJ3MHH52EQJGUYKVBF5DLX5EUQ3MSYBVE7VROLOMAUEXOYT4QQ, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.277886172 (279h43m40.148969217s) [noise=true , address=145.79.227.50:4374 , piece_id=RQY5LFQW2XNR6QD5LQF6YZCRCJAY6IAAZPXGYJJGXI2PZFUPPG6A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.436062099 (279h43m40.307145134s) [noise=true , address=45.91.116.6:4974 , piece_id=EHNNMCVG6QW5AI6IB3WFOWDDQHP6GMVQM6XFSIIDWMMS6DEDIW6A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.436076746 (279h43m40.307159781s) [noise=true , address=77.90.31.14:4662 , piece_id=MDYFTTVCCAUKQDEIFOYSFBFJCCAM72YEY7MV762OVDCCXBJWQF7A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.436088268 (279h43m40.307171303s) [seg_pos={0 0}, upload=245226]: Segment reached optimal threshold of 13 pieces. Dec 17 07:12:40.436303448 (279h43m40.307386493s) [noise=true , address=45.91.116.13:4156 , piece_id=2L6LQKALJCYD5RQ7SGRRDZP6BROA72REMBX6MKZSCBOUZTJOVVBQ, seg_pos={0 0}, upload=245226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 17 07:12:40.436312615 (279h43m40.30739565s) [noise=true , address=45.91.116.13:4156 , piece_id=2L6LQKALJCYD5RQ7SGRRDZP6BROA72REMBX6MKZSCBOUZTJOVVBQ, seg_pos={0 0}, upload=245226]: Long tail cancellation due to optimal threshold Dec 17 07:12:40.436357541 (279h43m40.307440586s) [noise=true , address=77.90.31.17:4773 , piece_id=NVICD5WRAIYRFLMSOWH5LNTKF74QSYBAUT7XLVDAQ5O3MDUOGA5A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:41348->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:41348->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41348->77.90.31.17:4773: use of closed network connection Dec 17 07:12:40.436368862 (279h43m40.307451907s) [noise=true , address=77.90.31.17:4773 , piece_id=NVICD5WRAIYRFLMSOWH5LNTKF74QSYBAUT7XLVDAQ5O3MDUOGA5A, seg_pos={0 0}, upload=245226]: Long tail cancellation due to optimal threshold Dec 17 07:12:40.436437392 (279h43m40.307520437s) [noise=true , address=45.91.116.8:4046 , piece_id=QUKL5Q3ONBRNEOM5TCN5RVCEX5BGTNTUMG5CCZSMCFBMBQIROXYA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 17 07:12:40.436443625 (279h43m40.30752666s) [noise=true , address=45.91.116.8:4046 , piece_id=QUKL5Q3ONBRNEOM5TCN5RVCEX5BGTNTUMG5CCZSMCFBMBQIROXYA, seg_pos={0 0}, upload=245226]: Long tail cancellation due to optimal threshold Dec 17 07:12:40.436459425 (279h43m40.30754247s) [seg_pos={0 0}, upload=245226]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 07:12:40.436538986 (279h43m40.307622021s) [upload=245226]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 07:12:40.436544858 (279h43m40.307627903s) [upload=245226]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 07:12:40.725332982 (279h43m40.596416017s) [upload=245226]: Done uploading segments... Dec 17 07:12:40.725352159 (279h43m40.596435194s) [upload=245226]: Upload finished. err: <nil> Duration: 5.925904246s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 4.926238156s trace: https://jaeger.dev.storj.io/trace/6d70ed7853c345d3 FAIL (2025-12-17 07:50:20 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.926266359s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.894037749s trace: https://jaeger.dev.storj.io/trace/47e835adff344c8 FAIL (2025-12-17 07:49:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: MDYFTTVCCAUKQDEIFOYSFBFJCCAM72YEY7MV762OVDCCXBJWQF7A) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: EHNNMCVG6QW5AI6IB3WFOWDDQHP6GMVQM6XFSIIDWMMS6DEDIW6A) context deadline exceeded ======LOGS====== ec 17 07:12:37.946668541 (279h43m37.817751586s) [upload=245226]: Starting upload Dec 17 07:12:37.946758152 (279h43m37.817841197s) [upload=245226]: Done starting upload Dec 17 07:12:37.947199001 (279h43m37.818282066s) [upload=245226]: Uploading segments... Dec 17 07:12:37.947253795 (279h43m37.81833685s) [upload=245226]: Got next segment. Inline: false Dec 17 07:12:37.947295084 (279h43m37.818378129s) [upload=245226]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 07:12:37.947301105 (279h43m37.81838414s) [upload=245226]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 07:12:38.191736294 (279h43m38.062819349s) [seg_pos={0 0}, upload=245226]: Begin upload segment... Dec 17 07:12:38.191795317 (279h43m38.062878362s) [seg_pos={0 0}, upload=245226]: Done begin upload segment. Dec 17 07:12:38.191819202 (279h43m38.062902238s) [noise=true , address=145.79.227.25:4618 , piece_id=C6JEP3RAPX22ECSDSRZL7CFRIDP2U2ZNRSWPVMJRF742CPGZBGPA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.191830183 (279h43m38.062913228s) [noise=true , address=77.90.31.17:4773 , piece_id=NVICD5WRAIYRFLMSOWH5LNTKF74QSYBAUT7XLVDAQ5O3MDUOGA5A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.192400609 (279h43m38.063483654s) [noise=true , address=145.79.227.52:4446 , piece_id=FYI226BYJJ3OA4NYEC6LELDWV5EPDQIE6OVAMJV2BTXNNXNWZPQA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193213818 (279h43m38.064296853s) [noise=true , address=45.91.116.8:4046 , piece_id=QUKL5Q3ONBRNEOM5TCN5RVCEX5BGTNTUMG5CCZSMCFBMBQIROXYA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193605724 (279h43m38.064688769s) [noise=true , address=145.79.227.16:4911 , piece_id=XZSCHKVU7TN7GDL33AZ6GXE5VLUCFZRP3G7ULXNN5GUQRTA6EZ7A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193689663 (279h43m38.064772709s) [noise=true , address=145.79.227.42:4121 , piece_id=RLGJ3MHH52EQJGUYKVBF5DLX5EUQ3MSYBVE7VROLOMAUEXOYT4QQ, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193977442 (279h43m38.065060487s) [noise=true , address=77.90.31.14:4662 , piece_id=MDYFTTVCCAUKQDEIFOYSFBFJCCAM72YEY7MV762OVDCCXBJWQF7A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194245692 (279h43m38.065328737s) [noise=true , address=45.91.116.13:4156 , piece_id=2L6LQKALJCYD5RQ7SGRRDZP6BROA72REMBX6MKZSCBOUZTJOVVBQ, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194366663 (279h43m38.065449698s) [noise=true , address=45.91.116.6:4974 , piece_id=EHNNMCVG6QW5AI6IB3WFOWDDQHP6GMVQM6XFSIIDWMMS6DEDIW6A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194740605 (279h43m38.06582364s) [noise=true , address=145.79.227.38:4979 , piece_id=MILXDFE5TH2N22E76WKJRID7UPASBSTKNUBNEQSN5BD32LPLRCUA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194863168 (279h43m38.065946213s) [noise=true , address=145.79.227.36:4519 , piece_id=HF4YFXYBONFXGYDQ6TJWJMNFHOP57PXKKMCGKOLRJJYEVL7EZS2A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195047018 (279h43m38.066130053s) [noise=true , address=145.79.227.48:4302 , piece_id=3VMJMBLA2BHO4VUCKSSPAMC26IFPO3IO5TKY7OLGYDB7UVIAPQ5Q, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195387467 (279h43m38.066470512s) [noise=true , address=145.79.227.50:4374 , piece_id=RQY5LFQW2XNR6QD5LQF6YZCRCJAY6IAAZPXGYJJGXI2PZFUPPG6A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195413807 (279h43m38.066496842s) [noise=true , address=145.79.227.41:4086 , piece_id=RRV7U2CFFJEZLCEBUGBJNQ6VF7HFKEEY6BWAJTUS645L6NKZFBBQ, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195752793 (279h43m38.066835828s) [noise=true , address=145.79.227.37:4565 , piece_id=LKHAWS2BR622N3KLSHASDEIKG5HETZ2GTPEFK4KGQYJKDPHD2XYA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195926243 (279h43m38.067009288s) [noise=true , address=145.79.227.11:4732 , piece_id=HOYU2ZK34RZWREKJQSZNSFZSNPX6U3U7CNDG2OT6FDFFCBLMZSNA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:39.894104052 (279h43m39.765187097s) [upload=245226]: Next returned nil segment Dec 17 07:12:39.894125211 (279h43m39.765208246s) [upload=245226]: Waiting for error group managing segments... Dec 17 07:12:40.182872669 (279h43m40.053955705s) [noise=true , address=145.79.227.38:4979 , piece_id=MILXDFE5TH2N22E76WKJRID7UPASBSTKNUBNEQSN5BD32LPLRCUA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.187264572 (279h43m40.058347617s) [noise=true , address=145.79.227.41:4086 , piece_id=RRV7U2CFFJEZLCEBUGBJNQ6VF7HFKEEY6BWAJTUS645L6NKZFBBQ, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.187588328 (279h43m40.058671373s) [noise=true , address=145.79.227.52:4446 , piece_id=FYI226BYJJ3OA4NYEC6LELDWV5EPDQIE6OVAMJV2BTXNNXNWZPQA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.201386834 (279h43m40.072469869s) [noise=true , address=145.79.227.16:4911 , piece_id=XZSCHKVU7TN7GDL33AZ6GXE5VLUCFZRP3G7ULXNN5GUQRTA6EZ7A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.207224941 (279h43m40.078307976s) [noise=true , address=145.79.227.37:4565 , piece_id=LKHAWS2BR622N3KLSHASDEIKG5HETZ2GTPEFK4KGQYJKDPHD2XYA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.209900164 (279h43m40.080983209s) [noise=true , address=145.79.227.11:4732 , piece_id=HOYU2ZK34RZWREKJQSZNSFZSNPX6U3U7CNDG2OT6FDFFCBLMZSNA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.209956011 (279h43m40.081039046s) [noise=true , address=145.79.227.25:4618 , piece_id=C6JEP3RAPX22ECSDSRZL7CFRIDP2U2ZNRSWPVMJRF742CPGZBGPA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.211720020 (279h43m40.082803065s) [noise=true , address=145.79.227.48:4302 , piece_id=3VMJMBLA2BHO4VUCKSSPAMC26IFPO3IO5TKY7OLGYDB7UVIAPQ5Q, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.214865549 (279h43m40.085948594s) [noise=true , address=145.79.227.36:4519 , piece_id=HF4YFXYBONFXGYDQ6TJWJMNFHOP57PXKKMCGKOLRJJYEVL7EZS2A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.246003612 (279h43m40.117086657s) [noise=true , address=145.79.227.42:4121 , piece_id=RLGJ3MHH52EQJGUYKVBF5DLX5EUQ3MSYBVE7VROLOMAUEXOYT4QQ, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.277886172 (279h43m40.148969217s) [noise=true , address=145.79.227.50:4374 , piece_id=RQY5LFQW2XNR6QD5LQF6YZCRCJAY6IAAZPXGYJJGXI2PZFUPPG6A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.436062099 (279h43m40.307145134s) [noise=true , address=45.91.116.6:4974 , piece_id=EHNNMCVG6QW5AI6IB3WFOWDDQHP6GMVQM6XFSIIDWMMS6DEDIW6A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.436076746 (279h43m40.307159781s) [noise=true , address=77.90.31.14:4662 , piece_id=MDYFTTVCCAUKQDEIFOYSFBFJCCAM72YEY7MV762OVDCCXBJWQF7A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.436088268 (279h43m40.307171303s) [seg_pos={0 0}, upload=245226]: Segment reached optimal threshold of 13 pieces. Dec 17 07:12:40.436303448 (279h43m40.307386493s) [noise=true , address=45.91.116.13:4156 , piece_id=2L6LQKALJCYD5RQ7SGRRDZP6BROA72REMBX6MKZSCBOUZTJOVVBQ, seg_pos={0 0}, upload=245226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 17 07:12:40.436312615 (279h43m40.30739565s) [noise=true , address=45.91.116.13:4156 , piece_id=2L6LQKALJCYD5RQ7SGRRDZP6BROA72REMBX6MKZSCBOUZTJOVVBQ, seg_pos={0 0}, upload=245226]: Long tail cancellation due to optimal threshold Dec 17 07:12:40.436357541 (279h43m40.307440586s) [noise=true , address=77.90.31.17:4773 , piece_id=NVICD5WRAIYRFLMSOWH5LNTKF74QSYBAUT7XLVDAQ5O3MDUOGA5A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:41348->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:41348->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41348->77.90.31.17:4773: use of closed network connection Dec 17 07:12:40.436368862 (279h43m40.307451907s) [noise=true , address=77.90.31.17:4773 , piece_id=NVICD5WRAIYRFLMSOWH5LNTKF74QSYBAUT7XLVDAQ5O3MDUOGA5A, seg_pos={0 0}, upload=245226]: Long tail cancellation due to optimal threshold Dec 17 07:12:40.436437392 (279h43m40.307520437s) [noise=true , address=45.91.116.8:4046 , piece_id=QUKL5Q3ONBRNEOM5TCN5RVCEX5BGTNTUMG5CCZSMCFBMBQIROXYA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 17 07:12:40.436443625 (279h43m40.30752666s) [noise=true , address=45.91.116.8:4046 , piece_id=QUKL5Q3ONBRNEOM5TCN5RVCEX5BGTNTUMG5CCZSMCFBMBQIROXYA, seg_pos={0 0}, upload=245226]: Long tail cancellation due to optimal threshold Dec 17 07:12:40.436459425 (279h43m40.30754247s) [seg_pos={0 0}, upload=245226]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 07:12:40.436538986 (279h43m40.307622021s) [upload=245226]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 07:12:40.436544858 (279h43m40.307627903s) [upload=245226]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 07:12:40.725332982 (279h43m40.596416017s) [upload=245226]: Done uploading segments... Dec 17 07:12:40.725352159 (279h43m40.596435194s) [upload=245226]: Upload finished. err: <nil> Duration: 5.894065262s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 4.965908959s trace: https://jaeger.dev.storj.io/trace/6d70ed7853c345d3 FAIL (2025-12-17 07:49:17 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.965944145s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 4.985776549s trace: https://jaeger.dev.storj.io/trace/6d70ed7853c345d3 FAIL (2025-12-17 07:48:47 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.985808601s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.825176533s trace: https://jaeger.dev.storj.io/trace/47e835adff344c8 FAIL (2025-12-17 07:48:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: MDYFTTVCCAUKQDEIFOYSFBFJCCAM72YEY7MV762OVDCCXBJWQF7A) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: EHNNMCVG6QW5AI6IB3WFOWDDQHP6GMVQM6XFSIIDWMMS6DEDIW6A) context deadline exceeded ======LOGS====== ec 17 07:12:37.946668541 (279h43m37.817751586s) [upload=245226]: Starting upload Dec 17 07:12:37.946758152 (279h43m37.817841197s) [upload=245226]: Done starting upload Dec 17 07:12:37.947199001 (279h43m37.818282066s) [upload=245226]: Uploading segments... Dec 17 07:12:37.947253795 (279h43m37.81833685s) [upload=245226]: Got next segment. Inline: false Dec 17 07:12:37.947295084 (279h43m37.818378129s) [upload=245226]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 07:12:37.947301105 (279h43m37.81838414s) [upload=245226]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 07:12:38.191736294 (279h43m38.062819349s) [seg_pos={0 0}, upload=245226]: Begin upload segment... Dec 17 07:12:38.191795317 (279h43m38.062878362s) [seg_pos={0 0}, upload=245226]: Done begin upload segment. Dec 17 07:12:38.191819202 (279h43m38.062902238s) [noise=true , address=145.79.227.25:4618 , piece_id=C6JEP3RAPX22ECSDSRZL7CFRIDP2U2ZNRSWPVMJRF742CPGZBGPA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.191830183 (279h43m38.062913228s) [noise=true , address=77.90.31.17:4773 , piece_id=NVICD5WRAIYRFLMSOWH5LNTKF74QSYBAUT7XLVDAQ5O3MDUOGA5A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.192400609 (279h43m38.063483654s) [noise=true , address=145.79.227.52:4446 , piece_id=FYI226BYJJ3OA4NYEC6LELDWV5EPDQIE6OVAMJV2BTXNNXNWZPQA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193213818 (279h43m38.064296853s) [noise=true , address=45.91.116.8:4046 , piece_id=QUKL5Q3ONBRNEOM5TCN5RVCEX5BGTNTUMG5CCZSMCFBMBQIROXYA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193605724 (279h43m38.064688769s) [noise=true , address=145.79.227.16:4911 , piece_id=XZSCHKVU7TN7GDL33AZ6GXE5VLUCFZRP3G7ULXNN5GUQRTA6EZ7A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193689663 (279h43m38.064772709s) [noise=true , address=145.79.227.42:4121 , piece_id=RLGJ3MHH52EQJGUYKVBF5DLX5EUQ3MSYBVE7VROLOMAUEXOYT4QQ, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193977442 (279h43m38.065060487s) [noise=true , address=77.90.31.14:4662 , piece_id=MDYFTTVCCAUKQDEIFOYSFBFJCCAM72YEY7MV762OVDCCXBJWQF7A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194245692 (279h43m38.065328737s) [noise=true , address=45.91.116.13:4156 , piece_id=2L6LQKALJCYD5RQ7SGRRDZP6BROA72REMBX6MKZSCBOUZTJOVVBQ, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194366663 (279h43m38.065449698s) [noise=true , address=45.91.116.6:4974 , piece_id=EHNNMCVG6QW5AI6IB3WFOWDDQHP6GMVQM6XFSIIDWMMS6DEDIW6A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194740605 (279h43m38.06582364s) [noise=true , address=145.79.227.38:4979 , piece_id=MILXDFE5TH2N22E76WKJRID7UPASBSTKNUBNEQSN5BD32LPLRCUA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194863168 (279h43m38.065946213s) [noise=true , address=145.79.227.36:4519 , piece_id=HF4YFXYBONFXGYDQ6TJWJMNFHOP57PXKKMCGKOLRJJYEVL7EZS2A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195047018 (279h43m38.066130053s) [noise=true , address=145.79.227.48:4302 , piece_id=3VMJMBLA2BHO4VUCKSSPAMC26IFPO3IO5TKY7OLGYDB7UVIAPQ5Q, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195387467 (279h43m38.066470512s) [noise=true , address=145.79.227.50:4374 , piece_id=RQY5LFQW2XNR6QD5LQF6YZCRCJAY6IAAZPXGYJJGXI2PZFUPPG6A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195413807 (279h43m38.066496842s) [noise=true , address=145.79.227.41:4086 , piece_id=RRV7U2CFFJEZLCEBUGBJNQ6VF7HFKEEY6BWAJTUS645L6NKZFBBQ, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195752793 (279h43m38.066835828s) [noise=true , address=145.79.227.37:4565 , piece_id=LKHAWS2BR622N3KLSHASDEIKG5HETZ2GTPEFK4KGQYJKDPHD2XYA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195926243 (279h43m38.067009288s) [noise=true , address=145.79.227.11:4732 , piece_id=HOYU2ZK34RZWREKJQSZNSFZSNPX6U3U7CNDG2OT6FDFFCBLMZSNA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:39.894104052 (279h43m39.765187097s) [upload=245226]: Next returned nil segment Dec 17 07:12:39.894125211 (279h43m39.765208246s) [upload=245226]: Waiting for error group managing segments... Dec 17 07:12:40.182872669 (279h43m40.053955705s) [noise=true , address=145.79.227.38:4979 , piece_id=MILXDFE5TH2N22E76WKJRID7UPASBSTKNUBNEQSN5BD32LPLRCUA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.187264572 (279h43m40.058347617s) [noise=true , address=145.79.227.41:4086 , piece_id=RRV7U2CFFJEZLCEBUGBJNQ6VF7HFKEEY6BWAJTUS645L6NKZFBBQ, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.187588328 (279h43m40.058671373s) [noise=true , address=145.79.227.52:4446 , piece_id=FYI226BYJJ3OA4NYEC6LELDWV5EPDQIE6OVAMJV2BTXNNXNWZPQA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.201386834 (279h43m40.072469869s) [noise=true , address=145.79.227.16:4911 , piece_id=XZSCHKVU7TN7GDL33AZ6GXE5VLUCFZRP3G7ULXNN5GUQRTA6EZ7A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.207224941 (279h43m40.078307976s) [noise=true , address=145.79.227.37:4565 , piece_id=LKHAWS2BR622N3KLSHASDEIKG5HETZ2GTPEFK4KGQYJKDPHD2XYA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.209900164 (279h43m40.080983209s) [noise=true , address=145.79.227.11:4732 , piece_id=HOYU2ZK34RZWREKJQSZNSFZSNPX6U3U7CNDG2OT6FDFFCBLMZSNA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.209956011 (279h43m40.081039046s) [noise=true , address=145.79.227.25:4618 , piece_id=C6JEP3RAPX22ECSDSRZL7CFRIDP2U2ZNRSWPVMJRF742CPGZBGPA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.211720020 (279h43m40.082803065s) [noise=true , address=145.79.227.48:4302 , piece_id=3VMJMBLA2BHO4VUCKSSPAMC26IFPO3IO5TKY7OLGYDB7UVIAPQ5Q, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.214865549 (279h43m40.085948594s) [noise=true , address=145.79.227.36:4519 , piece_id=HF4YFXYBONFXGYDQ6TJWJMNFHOP57PXKKMCGKOLRJJYEVL7EZS2A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.246003612 (279h43m40.117086657s) [noise=true , address=145.79.227.42:4121 , piece_id=RLGJ3MHH52EQJGUYKVBF5DLX5EUQ3MSYBVE7VROLOMAUEXOYT4QQ, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.277886172 (279h43m40.148969217s) [noise=true , address=145.79.227.50:4374 , piece_id=RQY5LFQW2XNR6QD5LQF6YZCRCJAY6IAAZPXGYJJGXI2PZFUPPG6A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.436062099 (279h43m40.307145134s) [noise=true , address=45.91.116.6:4974 , piece_id=EHNNMCVG6QW5AI6IB3WFOWDDQHP6GMVQM6XFSIIDWMMS6DEDIW6A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.436076746 (279h43m40.307159781s) [noise=true , address=77.90.31.14:4662 , piece_id=MDYFTTVCCAUKQDEIFOYSFBFJCCAM72YEY7MV762OVDCCXBJWQF7A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.436088268 (279h43m40.307171303s) [seg_pos={0 0}, upload=245226]: Segment reached optimal threshold of 13 pieces. Dec 17 07:12:40.436303448 (279h43m40.307386493s) [noise=true , address=45.91.116.13:4156 , piece_id=2L6LQKALJCYD5RQ7SGRRDZP6BROA72REMBX6MKZSCBOUZTJOVVBQ, seg_pos={0 0}, upload=245226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 17 07:12:40.436312615 (279h43m40.30739565s) [noise=true , address=45.91.116.13:4156 , piece_id=2L6LQKALJCYD5RQ7SGRRDZP6BROA72REMBX6MKZSCBOUZTJOVVBQ, seg_pos={0 0}, upload=245226]: Long tail cancellation due to optimal threshold Dec 17 07:12:40.436357541 (279h43m40.307440586s) [noise=true , address=77.90.31.17:4773 , piece_id=NVICD5WRAIYRFLMSOWH5LNTKF74QSYBAUT7XLVDAQ5O3MDUOGA5A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:41348->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:41348->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41348->77.90.31.17:4773: use of closed network connection Dec 17 07:12:40.436368862 (279h43m40.307451907s) [noise=true , address=77.90.31.17:4773 , piece_id=NVICD5WRAIYRFLMSOWH5LNTKF74QSYBAUT7XLVDAQ5O3MDUOGA5A, seg_pos={0 0}, upload=245226]: Long tail cancellation due to optimal threshold Dec 17 07:12:40.436437392 (279h43m40.307520437s) [noise=true , address=45.91.116.8:4046 , piece_id=QUKL5Q3ONBRNEOM5TCN5RVCEX5BGTNTUMG5CCZSMCFBMBQIROXYA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 17 07:12:40.436443625 (279h43m40.30752666s) [noise=true , address=45.91.116.8:4046 , piece_id=QUKL5Q3ONBRNEOM5TCN5RVCEX5BGTNTUMG5CCZSMCFBMBQIROXYA, seg_pos={0 0}, upload=245226]: Long tail cancellation due to optimal threshold Dec 17 07:12:40.436459425 (279h43m40.30754247s) [seg_pos={0 0}, upload=245226]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 07:12:40.436538986 (279h43m40.307622021s) [upload=245226]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 07:12:40.436544858 (279h43m40.307627903s) [upload=245226]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 07:12:40.725332982 (279h43m40.596416017s) [upload=245226]: Done uploading segments... Dec 17 07:12:40.725352159 (279h43m40.596435194s) [upload=245226]: Upload finished. err: <nil> Duration: 5.825203325s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 4.808230745s trace: https://jaeger.dev.storj.io/trace/6d70ed7853c345d3 FAIL (2025-12-17 07:48:05 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.808257837s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 6.017685848s trace: https://jaeger.dev.storj.io/trace/47e835adff344c8 FAIL (2025-12-17 07:47:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: MDYFTTVCCAUKQDEIFOYSFBFJCCAM72YEY7MV762OVDCCXBJWQF7A) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: EHNNMCVG6QW5AI6IB3WFOWDDQHP6GMVQM6XFSIIDWMMS6DEDIW6A) context deadline exceeded ======LOGS====== ec 17 07:12:37.946668541 (279h43m37.817751586s) [upload=245226]: Starting upload Dec 17 07:12:37.946758152 (279h43m37.817841197s) [upload=245226]: Done starting upload Dec 17 07:12:37.947199001 (279h43m37.818282066s) [upload=245226]: Uploading segments... Dec 17 07:12:37.947253795 (279h43m37.81833685s) [upload=245226]: Got next segment. Inline: false Dec 17 07:12:37.947295084 (279h43m37.818378129s) [upload=245226]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 07:12:37.947301105 (279h43m37.81838414s) [upload=245226]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 07:12:38.191736294 (279h43m38.062819349s) [seg_pos={0 0}, upload=245226]: Begin upload segment... Dec 17 07:12:38.191795317 (279h43m38.062878362s) [seg_pos={0 0}, upload=245226]: Done begin upload segment. Dec 17 07:12:38.191819202 (279h43m38.062902238s) [noise=true , address=145.79.227.25:4618 , piece_id=C6JEP3RAPX22ECSDSRZL7CFRIDP2U2ZNRSWPVMJRF742CPGZBGPA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.191830183 (279h43m38.062913228s) [noise=true , address=77.90.31.17:4773 , piece_id=NVICD5WRAIYRFLMSOWH5LNTKF74QSYBAUT7XLVDAQ5O3MDUOGA5A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.192400609 (279h43m38.063483654s) [noise=true , address=145.79.227.52:4446 , piece_id=FYI226BYJJ3OA4NYEC6LELDWV5EPDQIE6OVAMJV2BTXNNXNWZPQA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193213818 (279h43m38.064296853s) [noise=true , address=45.91.116.8:4046 , piece_id=QUKL5Q3ONBRNEOM5TCN5RVCEX5BGTNTUMG5CCZSMCFBMBQIROXYA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193605724 (279h43m38.064688769s) [noise=true , address=145.79.227.16:4911 , piece_id=XZSCHKVU7TN7GDL33AZ6GXE5VLUCFZRP3G7ULXNN5GUQRTA6EZ7A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193689663 (279h43m38.064772709s) [noise=true , address=145.79.227.42:4121 , piece_id=RLGJ3MHH52EQJGUYKVBF5DLX5EUQ3MSYBVE7VROLOMAUEXOYT4QQ, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193977442 (279h43m38.065060487s) [noise=true , address=77.90.31.14:4662 , piece_id=MDYFTTVCCAUKQDEIFOYSFBFJCCAM72YEY7MV762OVDCCXBJWQF7A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194245692 (279h43m38.065328737s) [noise=true , address=45.91.116.13:4156 , piece_id=2L6LQKALJCYD5RQ7SGRRDZP6BROA72REMBX6MKZSCBOUZTJOVVBQ, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194366663 (279h43m38.065449698s) [noise=true , address=45.91.116.6:4974 , piece_id=EHNNMCVG6QW5AI6IB3WFOWDDQHP6GMVQM6XFSIIDWMMS6DEDIW6A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194740605 (279h43m38.06582364s) [noise=true , address=145.79.227.38:4979 , piece_id=MILXDFE5TH2N22E76WKJRID7UPASBSTKNUBNEQSN5BD32LPLRCUA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194863168 (279h43m38.065946213s) [noise=true , address=145.79.227.36:4519 , piece_id=HF4YFXYBONFXGYDQ6TJWJMNFHOP57PXKKMCGKOLRJJYEVL7EZS2A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195047018 (279h43m38.066130053s) [noise=true , address=145.79.227.48:4302 , piece_id=3VMJMBLA2BHO4VUCKSSPAMC26IFPO3IO5TKY7OLGYDB7UVIAPQ5Q, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195387467 (279h43m38.066470512s) [noise=true , address=145.79.227.50:4374 , piece_id=RQY5LFQW2XNR6QD5LQF6YZCRCJAY6IAAZPXGYJJGXI2PZFUPPG6A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195413807 (279h43m38.066496842s) [noise=true , address=145.79.227.41:4086 , piece_id=RRV7U2CFFJEZLCEBUGBJNQ6VF7HFKEEY6BWAJTUS645L6NKZFBBQ, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195752793 (279h43m38.066835828s) [noise=true , address=145.79.227.37:4565 , piece_id=LKHAWS2BR622N3KLSHASDEIKG5HETZ2GTPEFK4KGQYJKDPHD2XYA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195926243 (279h43m38.067009288s) [noise=true , address=145.79.227.11:4732 , piece_id=HOYU2ZK34RZWREKJQSZNSFZSNPX6U3U7CNDG2OT6FDFFCBLMZSNA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:39.894104052 (279h43m39.765187097s) [upload=245226]: Next returned nil segment Dec 17 07:12:39.894125211 (279h43m39.765208246s) [upload=245226]: Waiting for error group managing segments... Dec 17 07:12:40.182872669 (279h43m40.053955705s) [noise=true , address=145.79.227.38:4979 , piece_id=MILXDFE5TH2N22E76WKJRID7UPASBSTKNUBNEQSN5BD32LPLRCUA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.187264572 (279h43m40.058347617s) [noise=true , address=145.79.227.41:4086 , piece_id=RRV7U2CFFJEZLCEBUGBJNQ6VF7HFKEEY6BWAJTUS645L6NKZFBBQ, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.187588328 (279h43m40.058671373s) [noise=true , address=145.79.227.52:4446 , piece_id=FYI226BYJJ3OA4NYEC6LELDWV5EPDQIE6OVAMJV2BTXNNXNWZPQA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.201386834 (279h43m40.072469869s) [noise=true , address=145.79.227.16:4911 , piece_id=XZSCHKVU7TN7GDL33AZ6GXE5VLUCFZRP3G7ULXNN5GUQRTA6EZ7A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.207224941 (279h43m40.078307976s) [noise=true , address=145.79.227.37:4565 , piece_id=LKHAWS2BR622N3KLSHASDEIKG5HETZ2GTPEFK4KGQYJKDPHD2XYA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.209900164 (279h43m40.080983209s) [noise=true , address=145.79.227.11:4732 , piece_id=HOYU2ZK34RZWREKJQSZNSFZSNPX6U3U7CNDG2OT6FDFFCBLMZSNA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.209956011 (279h43m40.081039046s) [noise=true , address=145.79.227.25:4618 , piece_id=C6JEP3RAPX22ECSDSRZL7CFRIDP2U2ZNRSWPVMJRF742CPGZBGPA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.211720020 (279h43m40.082803065s) [noise=true , address=145.79.227.48:4302 , piece_id=3VMJMBLA2BHO4VUCKSSPAMC26IFPO3IO5TKY7OLGYDB7UVIAPQ5Q, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.214865549 (279h43m40.085948594s) [noise=true , address=145.79.227.36:4519 , piece_id=HF4YFXYBONFXGYDQ6TJWJMNFHOP57PXKKMCGKOLRJJYEVL7EZS2A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.246003612 (279h43m40.117086657s) [noise=true , address=145.79.227.42:4121 , piece_id=RLGJ3MHH52EQJGUYKVBF5DLX5EUQ3MSYBVE7VROLOMAUEXOYT4QQ, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.277886172 (279h43m40.148969217s) [noise=true , address=145.79.227.50:4374 , piece_id=RQY5LFQW2XNR6QD5LQF6YZCRCJAY6IAAZPXGYJJGXI2PZFUPPG6A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.436062099 (279h43m40.307145134s) [noise=true , address=45.91.116.6:4974 , piece_id=EHNNMCVG6QW5AI6IB3WFOWDDQHP6GMVQM6XFSIIDWMMS6DEDIW6A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.436076746 (279h43m40.307159781s) [noise=true , address=77.90.31.14:4662 , piece_id=MDYFTTVCCAUKQDEIFOYSFBFJCCAM72YEY7MV762OVDCCXBJWQF7A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.436088268 (279h43m40.307171303s) [seg_pos={0 0}, upload=245226]: Segment reached optimal threshold of 13 pieces. Dec 17 07:12:40.436303448 (279h43m40.307386493s) [noise=true , address=45.91.116.13:4156 , piece_id=2L6LQKALJCYD5RQ7SGRRDZP6BROA72REMBX6MKZSCBOUZTJOVVBQ, seg_pos={0 0}, upload=245226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 17 07:12:40.436312615 (279h43m40.30739565s) [noise=true , address=45.91.116.13:4156 , piece_id=2L6LQKALJCYD5RQ7SGRRDZP6BROA72REMBX6MKZSCBOUZTJOVVBQ, seg_pos={0 0}, upload=245226]: Long tail cancellation due to optimal threshold Dec 17 07:12:40.436357541 (279h43m40.307440586s) [noise=true , address=77.90.31.17:4773 , piece_id=NVICD5WRAIYRFLMSOWH5LNTKF74QSYBAUT7XLVDAQ5O3MDUOGA5A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:41348->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:41348->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41348->77.90.31.17:4773: use of closed network connection Dec 17 07:12:40.436368862 (279h43m40.307451907s) [noise=true , address=77.90.31.17:4773 , piece_id=NVICD5WRAIYRFLMSOWH5LNTKF74QSYBAUT7XLVDAQ5O3MDUOGA5A, seg_pos={0 0}, upload=245226]: Long tail cancellation due to optimal threshold Dec 17 07:12:40.436437392 (279h43m40.307520437s) [noise=true , address=45.91.116.8:4046 , piece_id=QUKL5Q3ONBRNEOM5TCN5RVCEX5BGTNTUMG5CCZSMCFBMBQIROXYA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 17 07:12:40.436443625 (279h43m40.30752666s) [noise=true , address=45.91.116.8:4046 , piece_id=QUKL5Q3ONBRNEOM5TCN5RVCEX5BGTNTUMG5CCZSMCFBMBQIROXYA, seg_pos={0 0}, upload=245226]: Long tail cancellation due to optimal threshold Dec 17 07:12:40.436459425 (279h43m40.30754247s) [seg_pos={0 0}, upload=245226]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 07:12:40.436538986 (279h43m40.307622021s) [upload=245226]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 07:12:40.436544858 (279h43m40.307627903s) [upload=245226]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 07:12:40.725332982 (279h43m40.596416017s) [upload=245226]: Done uploading segments... Dec 17 07:12:40.725352159 (279h43m40.596435194s) [upload=245226]: Upload finished. err: <nil> Duration: 6.017714673s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 4.84131173s trace: https://jaeger.dev.storj.io/trace/6d70ed7853c345d3 FAIL (2025-12-17 07:47:14 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.841344543s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.815818223s trace: https://jaeger.dev.storj.io/trace/47e835adff344c8 FAIL (2025-12-17 07:46:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: MDYFTTVCCAUKQDEIFOYSFBFJCCAM72YEY7MV762OVDCCXBJWQF7A) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: EHNNMCVG6QW5AI6IB3WFOWDDQHP6GMVQM6XFSIIDWMMS6DEDIW6A) context deadline exceeded ======LOGS====== ec 17 07:12:37.946668541 (279h43m37.817751586s) [upload=245226]: Starting upload Dec 17 07:12:37.946758152 (279h43m37.817841197s) [upload=245226]: Done starting upload Dec 17 07:12:37.947199001 (279h43m37.818282066s) [upload=245226]: Uploading segments... Dec 17 07:12:37.947253795 (279h43m37.81833685s) [upload=245226]: Got next segment. Inline: false Dec 17 07:12:37.947295084 (279h43m37.818378129s) [upload=245226]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 07:12:37.947301105 (279h43m37.81838414s) [upload=245226]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 07:12:38.191736294 (279h43m38.062819349s) [seg_pos={0 0}, upload=245226]: Begin upload segment... Dec 17 07:12:38.191795317 (279h43m38.062878362s) [seg_pos={0 0}, upload=245226]: Done begin upload segment. Dec 17 07:12:38.191819202 (279h43m38.062902238s) [noise=true , address=145.79.227.25:4618 , piece_id=C6JEP3RAPX22ECSDSRZL7CFRIDP2U2ZNRSWPVMJRF742CPGZBGPA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.191830183 (279h43m38.062913228s) [noise=true , address=77.90.31.17:4773 , piece_id=NVICD5WRAIYRFLMSOWH5LNTKF74QSYBAUT7XLVDAQ5O3MDUOGA5A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.192400609 (279h43m38.063483654s) [noise=true , address=145.79.227.52:4446 , piece_id=FYI226BYJJ3OA4NYEC6LELDWV5EPDQIE6OVAMJV2BTXNNXNWZPQA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193213818 (279h43m38.064296853s) [noise=true , address=45.91.116.8:4046 , piece_id=QUKL5Q3ONBRNEOM5TCN5RVCEX5BGTNTUMG5CCZSMCFBMBQIROXYA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193605724 (279h43m38.064688769s) [noise=true , address=145.79.227.16:4911 , piece_id=XZSCHKVU7TN7GDL33AZ6GXE5VLUCFZRP3G7ULXNN5GUQRTA6EZ7A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193689663 (279h43m38.064772709s) [noise=true , address=145.79.227.42:4121 , piece_id=RLGJ3MHH52EQJGUYKVBF5DLX5EUQ3MSYBVE7VROLOMAUEXOYT4QQ, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.193977442 (279h43m38.065060487s) [noise=true , address=77.90.31.14:4662 , piece_id=MDYFTTVCCAUKQDEIFOYSFBFJCCAM72YEY7MV762OVDCCXBJWQF7A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194245692 (279h43m38.065328737s) [noise=true , address=45.91.116.13:4156 , piece_id=2L6LQKALJCYD5RQ7SGRRDZP6BROA72REMBX6MKZSCBOUZTJOVVBQ, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194366663 (279h43m38.065449698s) [noise=true , address=45.91.116.6:4974 , piece_id=EHNNMCVG6QW5AI6IB3WFOWDDQHP6GMVQM6XFSIIDWMMS6DEDIW6A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194740605 (279h43m38.06582364s) [noise=true , address=145.79.227.38:4979 , piece_id=MILXDFE5TH2N22E76WKJRID7UPASBSTKNUBNEQSN5BD32LPLRCUA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.194863168 (279h43m38.065946213s) [noise=true , address=145.79.227.36:4519 , piece_id=HF4YFXYBONFXGYDQ6TJWJMNFHOP57PXKKMCGKOLRJJYEVL7EZS2A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195047018 (279h43m38.066130053s) [noise=true , address=145.79.227.48:4302 , piece_id=3VMJMBLA2BHO4VUCKSSPAMC26IFPO3IO5TKY7OLGYDB7UVIAPQ5Q, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195387467 (279h43m38.066470512s) [noise=true , address=145.79.227.50:4374 , piece_id=RQY5LFQW2XNR6QD5LQF6YZCRCJAY6IAAZPXGYJJGXI2PZFUPPG6A, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195413807 (279h43m38.066496842s) [noise=true , address=145.79.227.41:4086 , piece_id=RRV7U2CFFJEZLCEBUGBJNQ6VF7HFKEEY6BWAJTUS645L6NKZFBBQ, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195752793 (279h43m38.066835828s) [noise=true , address=145.79.227.37:4565 , piece_id=LKHAWS2BR622N3KLSHASDEIKG5HETZ2GTPEFK4KGQYJKDPHD2XYA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:38.195926243 (279h43m38.067009288s) [noise=true , address=145.79.227.11:4732 , piece_id=HOYU2ZK34RZWREKJQSZNSFZSNPX6U3U7CNDG2OT6FDFFCBLMZSNA, seg_pos={0 0}, upload=245226]: Uploading piece... Dec 17 07:12:39.894104052 (279h43m39.765187097s) [upload=245226]: Next returned nil segment Dec 17 07:12:39.894125211 (279h43m39.765208246s) [upload=245226]: Waiting for error group managing segments... Dec 17 07:12:40.182872669 (279h43m40.053955705s) [noise=true , address=145.79.227.38:4979 , piece_id=MILXDFE5TH2N22E76WKJRID7UPASBSTKNUBNEQSN5BD32LPLRCUA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.187264572 (279h43m40.058347617s) [noise=true , address=145.79.227.41:4086 , piece_id=RRV7U2CFFJEZLCEBUGBJNQ6VF7HFKEEY6BWAJTUS645L6NKZFBBQ, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.187588328 (279h43m40.058671373s) [noise=true , address=145.79.227.52:4446 , piece_id=FYI226BYJJ3OA4NYEC6LELDWV5EPDQIE6OVAMJV2BTXNNXNWZPQA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.201386834 (279h43m40.072469869s) [noise=true , address=145.79.227.16:4911 , piece_id=XZSCHKVU7TN7GDL33AZ6GXE5VLUCFZRP3G7ULXNN5GUQRTA6EZ7A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.207224941 (279h43m40.078307976s) [noise=true , address=145.79.227.37:4565 , piece_id=LKHAWS2BR622N3KLSHASDEIKG5HETZ2GTPEFK4KGQYJKDPHD2XYA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.209900164 (279h43m40.080983209s) [noise=true , address=145.79.227.11:4732 , piece_id=HOYU2ZK34RZWREKJQSZNSFZSNPX6U3U7CNDG2OT6FDFFCBLMZSNA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.209956011 (279h43m40.081039046s) [noise=true , address=145.79.227.25:4618 , piece_id=C6JEP3RAPX22ECSDSRZL7CFRIDP2U2ZNRSWPVMJRF742CPGZBGPA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.211720020 (279h43m40.082803065s) [noise=true , address=145.79.227.48:4302 , piece_id=3VMJMBLA2BHO4VUCKSSPAMC26IFPO3IO5TKY7OLGYDB7UVIAPQ5Q, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.214865549 (279h43m40.085948594s) [noise=true , address=145.79.227.36:4519 , piece_id=HF4YFXYBONFXGYDQ6TJWJMNFHOP57PXKKMCGKOLRJJYEVL7EZS2A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.246003612 (279h43m40.117086657s) [noise=true , address=145.79.227.42:4121 , piece_id=RLGJ3MHH52EQJGUYKVBF5DLX5EUQ3MSYBVE7VROLOMAUEXOYT4QQ, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.277886172 (279h43m40.148969217s) [noise=true , address=145.79.227.50:4374 , piece_id=RQY5LFQW2XNR6QD5LQF6YZCRCJAY6IAAZPXGYJJGXI2PZFUPPG6A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.436062099 (279h43m40.307145134s) [noise=true , address=45.91.116.6:4974 , piece_id=EHNNMCVG6QW5AI6IB3WFOWDDQHP6GMVQM6XFSIIDWMMS6DEDIW6A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.436076746 (279h43m40.307159781s) [noise=true , address=77.90.31.14:4662 , piece_id=MDYFTTVCCAUKQDEIFOYSFBFJCCAM72YEY7MV762OVDCCXBJWQF7A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: <nil> Dec 17 07:12:40.436088268 (279h43m40.307171303s) [seg_pos={0 0}, upload=245226]: Segment reached optimal threshold of 13 pieces. Dec 17 07:12:40.436303448 (279h43m40.307386493s) [noise=true , address=45.91.116.13:4156 , piece_id=2L6LQKALJCYD5RQ7SGRRDZP6BROA72REMBX6MKZSCBOUZTJOVVBQ, seg_pos={0 0}, upload=245226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 17 07:12:40.436312615 (279h43m40.30739565s) [noise=true , address=45.91.116.13:4156 , piece_id=2L6LQKALJCYD5RQ7SGRRDZP6BROA72REMBX6MKZSCBOUZTJOVVBQ, seg_pos={0 0}, upload=245226]: Long tail cancellation due to optimal threshold Dec 17 07:12:40.436357541 (279h43m40.307440586s) [noise=true , address=77.90.31.17:4773 , piece_id=NVICD5WRAIYRFLMSOWH5LNTKF74QSYBAUT7XLVDAQ5O3MDUOGA5A, seg_pos={0 0}, upload=245226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:41348->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:41348->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41348->77.90.31.17:4773: use of closed network connection Dec 17 07:12:40.436368862 (279h43m40.307451907s) [noise=true , address=77.90.31.17:4773 , piece_id=NVICD5WRAIYRFLMSOWH5LNTKF74QSYBAUT7XLVDAQ5O3MDUOGA5A, seg_pos={0 0}, upload=245226]: Long tail cancellation due to optimal threshold Dec 17 07:12:40.436437392 (279h43m40.307520437s) [noise=true , address=45.91.116.8:4046 , piece_id=QUKL5Q3ONBRNEOM5TCN5RVCEX5BGTNTUMG5CCZSMCFBMBQIROXYA, seg_pos={0 0}, upload=245226]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 17 07:12:40.436443625 (279h43m40.30752666s) [noise=true , address=45.91.116.8:4046 , piece_id=QUKL5Q3ONBRNEOM5TCN5RVCEX5BGTNTUMG5CCZSMCFBMBQIROXYA, seg_pos={0 0}, upload=245226]: Long tail cancellation due to optimal threshold Dec 17 07:12:40.436459425 (279h43m40.30754247s) [seg_pos={0 0}, upload=245226]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 07:12:40.436538986 (279h43m40.307622021s) [upload=245226]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 07:12:40.436544858 (279h43m40.307627903s) [upload=245226]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 07:12:40.725332982 (279h43m40.596416017s) [upload=245226]: Done uploading segments... Dec 17 07:12:40.725352159 (279h43m40.596435194s) [upload=245226]: Upload finished. err: <nil> Duration: 5.815839074s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 4.980102644s trace: https://jaeger.dev.storj.io/trace/6d70ed7853c345d3 FAIL (2025-12-17 07:46:42 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.980243643s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.021853227s trace: https://jaeger.dev.storj.io/trace/6d70ed7853c345d3 FAIL (2025-12-17 07:46:07 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.021895297s core uplink eu1

Orbiter Check History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)