Orbiter History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.792665438s trace: https://jaeger.dev.storj.io/trace/7f1a3b811b64bfd3 FAIL (2025-11-25 13:07:19 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 5.792695284s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.804197609s trace: https://jaeger.dev.storj.io/trace/7f1a3b811b64bfd3 FAIL (2025-11-25 13:06:08 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 5.804229719s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.322950796s trace: https://jaeger.dev.storj.io/trace/7f1a3b811b64bfd3 FAIL (2025-11-25 13:05:38 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 5.322979919s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.782735719s trace: https://jaeger.dev.storj.io/trace/7f1a3b811b64bfd3 FAIL (2025-11-25 13:04:32 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 5.782770395s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 6.256589734s trace: https://jaeger.dev.storj.io/trace/7f1a3b811b64bfd3 FAIL (2025-11-25 13:03:30 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 6.256617057s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.715655992s trace: https://jaeger.dev.storj.io/trace/7f1a3b811b64bfd3 FAIL (2025-11-25 13:02:50 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 5.71568659s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.8900068s trace: https://jaeger.dev.storj.io/trace/7f1a3b811b64bfd3 FAIL (2025-11-25 13:02:14 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 5.890036507s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.798787424s trace: https://jaeger.dev.storj.io/trace/7f1a3b811b64bfd3 FAIL (2025-11-25 13:01:06 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 5.798816991s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.834392197s trace: https://jaeger.dev.storj.io/trace/7f1a3b811b64bfd3 FAIL (2025-11-25 13:00:47 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 5.834424298s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.787113288s trace: https://jaeger.dev.storj.io/trace/7f1a3b811b64bfd3 FAIL (2025-11-25 12:59:53 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 5.787149876s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.519451921s trace: https://jaeger.dev.storj.io/trace/7f1a3b811b64bfd3 FAIL (2025-11-25 12:59:36 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 5.519480093s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.855098165s trace: https://jaeger.dev.storj.io/trace/7f1a3b811b64bfd3 FAIL (2025-11-25 12:59:05 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 5.855128351s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.70071444s trace: https://jaeger.dev.storj.io/trace/7f1a3b811b64bfd3 FAIL (2025-11-25 12:58:00 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 5.700739327s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.802044431s trace: https://jaeger.dev.storj.io/trace/7f1a3b811b64bfd3 FAIL (2025-11-25 12:57:22 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 5.802077654s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.798600833s trace: https://jaeger.dev.storj.io/trace/7f1a3b811b64bfd3 FAIL (2025-11-25 12:56:33 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 5.798626021s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.777441131s trace: https://jaeger.dev.storj.io/trace/7f1a3b811b64bfd3 FAIL (2025-11-25 12:55:59 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 5.777492249s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.778555241s trace: https://jaeger.dev.storj.io/trace/7f1a3b811b64bfd3 FAIL (2025-11-25 12:55:11 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 5.778582683s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.802058316s trace: https://jaeger.dev.storj.io/trace/7f1a3b811b64bfd3 FAIL (2025-11-25 12:54:11 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 5.802086471s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.770150875s trace: https://jaeger.dev.storj.io/trace/7f1a3b811b64bfd3 FAIL (2025-11-25 12:53:18 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 5.770191593s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.789839637s trace: https://jaeger.dev.storj.io/trace/7f1a3b811b64bfd3 FAIL (2025-11-25 12:52:57 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 5.789871409s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.570625349s trace: https://jaeger.dev.storj.io/trace/7f1a3b811b64bfd3 FAIL (2025-11-25 12:52:05 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 5.570660046s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.811435484s trace: https://jaeger.dev.storj.io/trace/7f1a3b811b64bfd3 FAIL (2025-11-25 12:51:16 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 5.811465701s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 4.729019827s trace: https://jaeger.dev.storj.io/trace/7f1a3b811b64bfd3 FAIL (2025-11-25 12:50:43 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 4.729048083s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.670819271s trace: https://jaeger.dev.storj.io/trace/7f1a3b811b64bfd3 FAIL (2025-11-25 12:50:18 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 5.670855618s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.662872709s trace: https://jaeger.dev.storj.io/trace/7f1a3b811b64bfd3 FAIL (2025-11-25 12:49:48 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 5.662890693s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.001268418s trace: https://jaeger.dev.storj.io/trace/7f1a3b811b64bfd3 FAIL (2025-11-25 12:49:42 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 1m0.00136923s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.758455051s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 07:16:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.758485379s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.702908239s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 07:16:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.70293969s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.683050307s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 07:15:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.683077981s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 6.220347487s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 07:14:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 6.22036975s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.746825462s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 07:13:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.746854836s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.775479663s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 07:13:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.775510231s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.743192673s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 07:12:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.743256955s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.60496986s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 07:12:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.605001029s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.219103217s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 07:11:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 6.219122703s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.843317094s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 07:11:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.843353833s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.893145196s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 07:10:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.893172628s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.686212273s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 07:09:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.686248472s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.674885794s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 07:08:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.67491468s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.699379507s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 07:07:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.699411519s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.654185368s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 07:06:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.65421262s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.690880336s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 07:05:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.690902097s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.680339208s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 07:05:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.680402608s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.757393926s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 07:04:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.75742272s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.900458758s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 07:03:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.900610908s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.758546633s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 07:03:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.758651641s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.846745141s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 07:03:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.846773435s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.664996424s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 07:02:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.665028715s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.886089277s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 07:01:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.886119805s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.695858645s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 07:01:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.695877962s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)