Orbiter History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 1.385469916s trace: https://jaeger.dev.storj.io/trace/693c1929ad760ab0 FAIL (2025-11-16 23:21:20 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.385504554s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 1.391321582s trace: https://jaeger.dev.storj.io/trace/693c1929ad760ab0 FAIL (2025-11-16 23:20:42 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.391395362s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 1.439074834s trace: https://jaeger.dev.storj.io/trace/693c1929ad760ab0 FAIL (2025-11-16 23:20:27 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.439110402s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 1.440643553s trace: https://jaeger.dev.storj.io/trace/693c1929ad760ab0 FAIL (2025-11-16 23:19:28 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.440666606s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 1.380445484s trace: https://jaeger.dev.storj.io/trace/693c1929ad760ab0 FAIL (2025-11-16 23:18:55 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.380473446s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 1.378055702s trace: https://jaeger.dev.storj.io/trace/693c1929ad760ab0 FAIL (2025-11-16 23:18:08 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.378071873s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.00130259s trace: https://jaeger.dev.storj.io/trace/693c1929ad760ab0 FAIL (2025-11-16 23:18:06 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.001324912s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 3.427136582s trace: https://jaeger.dev.storj.io/trace/251e9c48735b3b0a FAIL (2025-11-16 22:45:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.427171277s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 3.406407381s trace: https://jaeger.dev.storj.io/trace/251e9c48735b3b0a FAIL (2025-11-16 22:44:20 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.406441558s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 3.428859072s trace: https://jaeger.dev.storj.io/trace/251e9c48735b3b0a FAIL (2025-11-16 22:43:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.428875023s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 3.488753111s trace: https://jaeger.dev.storj.io/trace/251e9c48735b3b0a FAIL (2025-11-16 22:42:01 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.488782568s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 3.40625448s trace: https://jaeger.dev.storj.io/trace/251e9c48735b3b0a FAIL (2025-11-16 22:41:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.406277604s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 3.548525573s trace: https://jaeger.dev.storj.io/trace/251e9c48735b3b0a FAIL (2025-11-16 22:40:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.548554177s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 3.428312837s trace: https://jaeger.dev.storj.io/trace/251e9c48735b3b0a FAIL (2025-11-16 22:40:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.428343364s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 3.371687557s trace: https://jaeger.dev.storj.io/trace/251e9c48735b3b0a FAIL (2025-11-16 22:39:40 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.371740918s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 3.530243952s trace: https://jaeger.dev.storj.io/trace/251e9c48735b3b0a FAIL (2025-11-16 22:38:56 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.530273659s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 3.430370422s trace: https://jaeger.dev.storj.io/trace/251e9c48735b3b0a FAIL (2025-11-16 22:37:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.430389178s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 3.395536499s trace: https://jaeger.dev.storj.io/trace/251e9c48735b3b0a FAIL (2025-11-16 22:37:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.395563401s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 3.508943603s trace: https://jaeger.dev.storj.io/trace/251e9c48735b3b0a FAIL (2025-11-16 22:36:06 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.508975313s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.010719253s trace: https://jaeger.dev.storj.io/trace/6c6e7bbb319587a2 FAIL (2025-11-16 22:36:01 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 16 21:49:44.857632528 (181h45m43.409102979s) [upload=160073]: Starting upload Nov 16 21:49:44.857711077 (181h45m43.409181538s) [upload=160073]: Done starting upload Nov 16 21:49:44.857867525 (181h45m43.409337986s) [upload=160073]: Uploading segments... Nov 16 21:49:44.858117080 (181h45m43.409587531s) [upload=160073]: Got next segment. Inline: false Nov 16 21:49:44.858184298 (181h45m43.409654749s) [upload=160073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 21:49:44.858191282 (181h45m43.409661733s) [upload=160073]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 21:49:46.079436121 (181h45m44.630906582s) [seg_pos={0 0}, upload=160073]: Begin upload segment... Nov 16 21:49:46.079498500 (181h45m44.630968961s) [seg_pos={0 0}, upload=160073]: Done begin upload segment. Nov 16 21:49:46.079551421 (181h45m44.631021882s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.079592329 (181h45m44.63106279s) [noise=true , address=145.79.227.25:4617 , piece_id=4H4BTA7PSPVFR7FJ7TCQCUOQPRVJCICXSBXUR6XJEZV5NSIBRZWQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.079721945 (181h45m44.631192406s) [noise=true , address=145.79.227.9:4366 , piece_id=SI7YJXWUTUXDVPNBYCLDPPVRE66RXHLPBLB43A45TND34UO4TF7Q, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080339301 (181h45m44.631809762s) [noise=true , address=145.79.227.10:4401 , piece_id=WIRM4I7DVYSHDQOEVCH23GRAC2G6HTQ2USSVL2HDTL6MO3VQQQJA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080638140 (181h45m44.632108601s) [noise=true , address=145.79.227.11:4730 , piece_id=CPUKLG5IGGB3RLTVTPSXHCW6T25KJ3P7XNAMKOCRMOJ7UC4EVKDA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080980683 (181h45m44.632451144s) [noise=true , address=145.79.227.30:4761 , piece_id=I256J6YH4LHPL4DU6H4P7NYFVJRI2XBV2AFJMASE2HPPDMO6QXLA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080993116 (181h45m44.632463577s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.081391766 (181h45m44.632862227s) [noise=true , address=145.79.227.15:4876 , piece_id=RSNHRRILCRJPBG6KJZTVFCQAROWHF5C6RBIB7XGSQH2SV3G6HUYQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.081400582 (181h45m44.632871043s) [noise=true , address=145.79.227.32:4834 , piece_id=4GON7GI63RXLBPJAJM32MO5BRMDTUFKJCRIJQDDWYDVOZA64LXGQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.082534582 (181h45m44.634005043s) [noise=true , address=145.79.227.15:4873 , piece_id=OHIKS4CCPNMIVJAVNDZCPDXRL4PZLBLNMX6T6FV67N7S3BL75DZQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.082659149 (181h45m44.63412961s) [noise=true , address=77.90.31.14:4663 , piece_id=W4MIXY47CPR4IYA6WCREO4VC5BVGAFBJB2BJV5ZNODIIHEXJT2VQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.083189538 (181h45m44.634659989s) [noise=true , address=145.79.227.31:4798 , piece_id=MBH7EMPFU4UKRIDVAHD6B7COGTRYBUZ3XS2L2UVCJ5ONXEJE4POQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.083640587 (181h45m44.635111048s) [noise=true , address=145.79.227.14:4837 , piece_id=YBBFYSC22KSQK5GENOT2NU624PDLUNX2TWQ24RNRW3WMGJT6P7RA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.084303179 (181h45m44.63577364s) [noise=true , address=145.79.227.17:4971 , piece_id=LBNF3Z5A3NZGUQKCWAIW4XPONBC2DD324VXP44FFCQ5FEXA5FZOA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.085550574 (181h45m44.637021025s) [noise=true , address=145.79.227.20:4439 , piece_id=PLNGHRV2N3GPQYYSGU7TKL2FMFCU4JQWXPM4K4HUHH3WKKLLQYVQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.091014480 (181h45m44.642484931s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:47.811332199 (181h45m46.36280266s) [upload=160073]: Next returned nil segment Nov 16 21:49:47.811354462 (181h45m46.362824913s) [upload=160073]: Waiting for error group managing segments... Nov 16 21:49:48.087848677 (181h45m46.639319128s) [noise=true , address=145.79.227.32:4834 , piece_id=4GON7GI63RXLBPJAJM32MO5BRMDTUFKJCRIJQDDWYDVOZA64LXGQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.105693638 (181h45m46.657164099s) [noise=true , address=145.79.227.25:4617 , piece_id=4H4BTA7PSPVFR7FJ7TCQCUOQPRVJCICXSBXUR6XJEZV5NSIBRZWQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.106011864 (181h45m46.657482335s) [noise=true , address=145.79.227.10:4401 , piece_id=WIRM4I7DVYSHDQOEVCH23GRAC2G6HTQ2USSVL2HDTL6MO3VQQQJA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.106913180 (181h45m46.658383641s) [noise=true , address=145.79.227.15:4876 , piece_id=RSNHRRILCRJPBG6KJZTVFCQAROWHF5C6RBIB7XGSQH2SV3G6HUYQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.107526719 (181h45m46.65899718s) [noise=true , address=145.79.227.15:4873 , piece_id=OHIKS4CCPNMIVJAVNDZCPDXRL4PZLBLNMX6T6FV67N7S3BL75DZQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.114554281 (181h45m46.666024732s) [noise=true , address=145.79.227.30:4761 , piece_id=I256J6YH4LHPL4DU6H4P7NYFVJRI2XBV2AFJMASE2HPPDMO6QXLA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.119027128 (181h45m46.670497589s) [noise=true , address=145.79.227.11:4730 , piece_id=CPUKLG5IGGB3RLTVTPSXHCW6T25KJ3P7XNAMKOCRMOJ7UC4EVKDA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.133964262 (181h45m46.685434713s) [noise=true , address=145.79.227.20:4439 , piece_id=PLNGHRV2N3GPQYYSGU7TKL2FMFCU4JQWXPM4K4HUHH3WKKLLQYVQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.146016122 (181h45m46.697486573s) [noise=true , address=145.79.227.17:4971 , piece_id=LBNF3Z5A3NZGUQKCWAIW4XPONBC2DD324VXP44FFCQ5FEXA5FZOA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.153123927 (181h45m46.704594388s) [noise=true , address=145.79.227.31:4798 , piece_id=MBH7EMPFU4UKRIDVAHD6B7COGTRYBUZ3XS2L2UVCJ5ONXEJE4POQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.174476267 (181h45m46.725946728s) [noise=true , address=145.79.227.9:4366 , piece_id=SI7YJXWUTUXDVPNBYCLDPPVRE66RXHLPBLB43A45TND34UO4TF7Q, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.197067667 (181h45m46.748538128s) [noise=true , address=145.79.227.14:4837 , piece_id=YBBFYSC22KSQK5GENOT2NU624PDLUNX2TWQ24RNRW3WMGJT6P7RA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.238467652 (181h45m46.789938113s) [noise=true , address=77.90.31.14:4663 , piece_id=W4MIXY47CPR4IYA6WCREO4VC5BVGAFBJB2BJV5ZNODIIHEXJT2VQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.238492430 (181h45m46.789962891s) [seg_pos={0 0}, upload=160073]: Segment reached optimal threshold of 13 pieces. Nov 16 21:49:48.238743558 (181h45m46.790214019s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.238753988 (181h45m46.790224449s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.238823801 (181h45m46.790294262s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.238844991 (181h45m46.790315442s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.239032308 (181h45m46.790502769s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.239041596 (181h45m46.790512047s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.239195508 (181h45m46.790665959s) [seg_pos={0 0}, upload=160073]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 21:49:48.239303494 (181h45m46.790773955s) [upload=160073]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 21:49:48.239313143 (181h45m46.790783594s) [upload=160073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 21:49:53.369411932 (181h45m51.920882393s) [upload=160073]: Done uploading segments... Nov 16 21:49:53.369436949 (181h45m51.92090741s) [upload=160073]: Upload finished. err: <nil> Duration: 6.0107497s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 3.562074374s trace: https://jaeger.dev.storj.io/trace/251e9c48735b3b0a FAIL (2025-11-16 22:35:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.562102507s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 3.485079303s trace: https://jaeger.dev.storj.io/trace/251e9c48735b3b0a FAIL (2025-11-16 22:35:31 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.48510986s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.869907223s trace: https://jaeger.dev.storj.io/trace/6c6e7bbb319587a2 FAIL (2025-11-16 22:34:49 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 16 21:49:44.857632528 (181h45m43.409102979s) [upload=160073]: Starting upload Nov 16 21:49:44.857711077 (181h45m43.409181538s) [upload=160073]: Done starting upload Nov 16 21:49:44.857867525 (181h45m43.409337986s) [upload=160073]: Uploading segments... Nov 16 21:49:44.858117080 (181h45m43.409587531s) [upload=160073]: Got next segment. Inline: false Nov 16 21:49:44.858184298 (181h45m43.409654749s) [upload=160073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 21:49:44.858191282 (181h45m43.409661733s) [upload=160073]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 21:49:46.079436121 (181h45m44.630906582s) [seg_pos={0 0}, upload=160073]: Begin upload segment... Nov 16 21:49:46.079498500 (181h45m44.630968961s) [seg_pos={0 0}, upload=160073]: Done begin upload segment. Nov 16 21:49:46.079551421 (181h45m44.631021882s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.079592329 (181h45m44.63106279s) [noise=true , address=145.79.227.25:4617 , piece_id=4H4BTA7PSPVFR7FJ7TCQCUOQPRVJCICXSBXUR6XJEZV5NSIBRZWQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.079721945 (181h45m44.631192406s) [noise=true , address=145.79.227.9:4366 , piece_id=SI7YJXWUTUXDVPNBYCLDPPVRE66RXHLPBLB43A45TND34UO4TF7Q, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080339301 (181h45m44.631809762s) [noise=true , address=145.79.227.10:4401 , piece_id=WIRM4I7DVYSHDQOEVCH23GRAC2G6HTQ2USSVL2HDTL6MO3VQQQJA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080638140 (181h45m44.632108601s) [noise=true , address=145.79.227.11:4730 , piece_id=CPUKLG5IGGB3RLTVTPSXHCW6T25KJ3P7XNAMKOCRMOJ7UC4EVKDA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080980683 (181h45m44.632451144s) [noise=true , address=145.79.227.30:4761 , piece_id=I256J6YH4LHPL4DU6H4P7NYFVJRI2XBV2AFJMASE2HPPDMO6QXLA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080993116 (181h45m44.632463577s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.081391766 (181h45m44.632862227s) [noise=true , address=145.79.227.15:4876 , piece_id=RSNHRRILCRJPBG6KJZTVFCQAROWHF5C6RBIB7XGSQH2SV3G6HUYQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.081400582 (181h45m44.632871043s) [noise=true , address=145.79.227.32:4834 , piece_id=4GON7GI63RXLBPJAJM32MO5BRMDTUFKJCRIJQDDWYDVOZA64LXGQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.082534582 (181h45m44.634005043s) [noise=true , address=145.79.227.15:4873 , piece_id=OHIKS4CCPNMIVJAVNDZCPDXRL4PZLBLNMX6T6FV67N7S3BL75DZQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.082659149 (181h45m44.63412961s) [noise=true , address=77.90.31.14:4663 , piece_id=W4MIXY47CPR4IYA6WCREO4VC5BVGAFBJB2BJV5ZNODIIHEXJT2VQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.083189538 (181h45m44.634659989s) [noise=true , address=145.79.227.31:4798 , piece_id=MBH7EMPFU4UKRIDVAHD6B7COGTRYBUZ3XS2L2UVCJ5ONXEJE4POQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.083640587 (181h45m44.635111048s) [noise=true , address=145.79.227.14:4837 , piece_id=YBBFYSC22KSQK5GENOT2NU624PDLUNX2TWQ24RNRW3WMGJT6P7RA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.084303179 (181h45m44.63577364s) [noise=true , address=145.79.227.17:4971 , piece_id=LBNF3Z5A3NZGUQKCWAIW4XPONBC2DD324VXP44FFCQ5FEXA5FZOA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.085550574 (181h45m44.637021025s) [noise=true , address=145.79.227.20:4439 , piece_id=PLNGHRV2N3GPQYYSGU7TKL2FMFCU4JQWXPM4K4HUHH3WKKLLQYVQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.091014480 (181h45m44.642484931s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:47.811332199 (181h45m46.36280266s) [upload=160073]: Next returned nil segment Nov 16 21:49:47.811354462 (181h45m46.362824913s) [upload=160073]: Waiting for error group managing segments... Nov 16 21:49:48.087848677 (181h45m46.639319128s) [noise=true , address=145.79.227.32:4834 , piece_id=4GON7GI63RXLBPJAJM32MO5BRMDTUFKJCRIJQDDWYDVOZA64LXGQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.105693638 (181h45m46.657164099s) [noise=true , address=145.79.227.25:4617 , piece_id=4H4BTA7PSPVFR7FJ7TCQCUOQPRVJCICXSBXUR6XJEZV5NSIBRZWQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.106011864 (181h45m46.657482335s) [noise=true , address=145.79.227.10:4401 , piece_id=WIRM4I7DVYSHDQOEVCH23GRAC2G6HTQ2USSVL2HDTL6MO3VQQQJA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.106913180 (181h45m46.658383641s) [noise=true , address=145.79.227.15:4876 , piece_id=RSNHRRILCRJPBG6KJZTVFCQAROWHF5C6RBIB7XGSQH2SV3G6HUYQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.107526719 (181h45m46.65899718s) [noise=true , address=145.79.227.15:4873 , piece_id=OHIKS4CCPNMIVJAVNDZCPDXRL4PZLBLNMX6T6FV67N7S3BL75DZQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.114554281 (181h45m46.666024732s) [noise=true , address=145.79.227.30:4761 , piece_id=I256J6YH4LHPL4DU6H4P7NYFVJRI2XBV2AFJMASE2HPPDMO6QXLA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.119027128 (181h45m46.670497589s) [noise=true , address=145.79.227.11:4730 , piece_id=CPUKLG5IGGB3RLTVTPSXHCW6T25KJ3P7XNAMKOCRMOJ7UC4EVKDA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.133964262 (181h45m46.685434713s) [noise=true , address=145.79.227.20:4439 , piece_id=PLNGHRV2N3GPQYYSGU7TKL2FMFCU4JQWXPM4K4HUHH3WKKLLQYVQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.146016122 (181h45m46.697486573s) [noise=true , address=145.79.227.17:4971 , piece_id=LBNF3Z5A3NZGUQKCWAIW4XPONBC2DD324VXP44FFCQ5FEXA5FZOA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.153123927 (181h45m46.704594388s) [noise=true , address=145.79.227.31:4798 , piece_id=MBH7EMPFU4UKRIDVAHD6B7COGTRYBUZ3XS2L2UVCJ5ONXEJE4POQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.174476267 (181h45m46.725946728s) [noise=true , address=145.79.227.9:4366 , piece_id=SI7YJXWUTUXDVPNBYCLDPPVRE66RXHLPBLB43A45TND34UO4TF7Q, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.197067667 (181h45m46.748538128s) [noise=true , address=145.79.227.14:4837 , piece_id=YBBFYSC22KSQK5GENOT2NU624PDLUNX2TWQ24RNRW3WMGJT6P7RA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.238467652 (181h45m46.789938113s) [noise=true , address=77.90.31.14:4663 , piece_id=W4MIXY47CPR4IYA6WCREO4VC5BVGAFBJB2BJV5ZNODIIHEXJT2VQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.238492430 (181h45m46.789962891s) [seg_pos={0 0}, upload=160073]: Segment reached optimal threshold of 13 pieces. Nov 16 21:49:48.238743558 (181h45m46.790214019s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.238753988 (181h45m46.790224449s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.238823801 (181h45m46.790294262s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.238844991 (181h45m46.790315442s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.239032308 (181h45m46.790502769s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.239041596 (181h45m46.790512047s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.239195508 (181h45m46.790665959s) [seg_pos={0 0}, upload=160073]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 21:49:48.239303494 (181h45m46.790773955s) [upload=160073]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 21:49:48.239313143 (181h45m46.790783594s) [upload=160073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 21:49:53.369411932 (181h45m51.920882393s) [upload=160073]: Done uploading segments... Nov 16 21:49:53.369436949 (181h45m51.92090741s) [upload=160073]: Upload finished. err: <nil> Duration: 5.869935307s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.308352649s trace: https://jaeger.dev.storj.io/trace/1533f838bd5c565 FAIL (2025-11-16 22:34:36 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.308374379s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............X.............................................] duration: 3.524029999s trace: https://jaeger.dev.storj.io/trace/251e9c48735b3b0a FAIL (2025-11-16 22:34:23 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.524079794s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 6.112873895s trace: https://jaeger.dev.storj.io/trace/1533f838bd5c565 FAIL (2025-11-16 22:34:20 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.112903502s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.981679714s trace: https://jaeger.dev.storj.io/trace/6c6e7bbb319587a2 FAIL (2025-11-16 22:34:17 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 16 21:49:44.857632528 (181h45m43.409102979s) [upload=160073]: Starting upload Nov 16 21:49:44.857711077 (181h45m43.409181538s) [upload=160073]: Done starting upload Nov 16 21:49:44.857867525 (181h45m43.409337986s) [upload=160073]: Uploading segments... Nov 16 21:49:44.858117080 (181h45m43.409587531s) [upload=160073]: Got next segment. Inline: false Nov 16 21:49:44.858184298 (181h45m43.409654749s) [upload=160073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 21:49:44.858191282 (181h45m43.409661733s) [upload=160073]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 21:49:46.079436121 (181h45m44.630906582s) [seg_pos={0 0}, upload=160073]: Begin upload segment... Nov 16 21:49:46.079498500 (181h45m44.630968961s) [seg_pos={0 0}, upload=160073]: Done begin upload segment. Nov 16 21:49:46.079551421 (181h45m44.631021882s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.079592329 (181h45m44.63106279s) [noise=true , address=145.79.227.25:4617 , piece_id=4H4BTA7PSPVFR7FJ7TCQCUOQPRVJCICXSBXUR6XJEZV5NSIBRZWQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.079721945 (181h45m44.631192406s) [noise=true , address=145.79.227.9:4366 , piece_id=SI7YJXWUTUXDVPNBYCLDPPVRE66RXHLPBLB43A45TND34UO4TF7Q, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080339301 (181h45m44.631809762s) [noise=true , address=145.79.227.10:4401 , piece_id=WIRM4I7DVYSHDQOEVCH23GRAC2G6HTQ2USSVL2HDTL6MO3VQQQJA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080638140 (181h45m44.632108601s) [noise=true , address=145.79.227.11:4730 , piece_id=CPUKLG5IGGB3RLTVTPSXHCW6T25KJ3P7XNAMKOCRMOJ7UC4EVKDA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080980683 (181h45m44.632451144s) [noise=true , address=145.79.227.30:4761 , piece_id=I256J6YH4LHPL4DU6H4P7NYFVJRI2XBV2AFJMASE2HPPDMO6QXLA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080993116 (181h45m44.632463577s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.081391766 (181h45m44.632862227s) [noise=true , address=145.79.227.15:4876 , piece_id=RSNHRRILCRJPBG6KJZTVFCQAROWHF5C6RBIB7XGSQH2SV3G6HUYQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.081400582 (181h45m44.632871043s) [noise=true , address=145.79.227.32:4834 , piece_id=4GON7GI63RXLBPJAJM32MO5BRMDTUFKJCRIJQDDWYDVOZA64LXGQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.082534582 (181h45m44.634005043s) [noise=true , address=145.79.227.15:4873 , piece_id=OHIKS4CCPNMIVJAVNDZCPDXRL4PZLBLNMX6T6FV67N7S3BL75DZQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.082659149 (181h45m44.63412961s) [noise=true , address=77.90.31.14:4663 , piece_id=W4MIXY47CPR4IYA6WCREO4VC5BVGAFBJB2BJV5ZNODIIHEXJT2VQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.083189538 (181h45m44.634659989s) [noise=true , address=145.79.227.31:4798 , piece_id=MBH7EMPFU4UKRIDVAHD6B7COGTRYBUZ3XS2L2UVCJ5ONXEJE4POQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.083640587 (181h45m44.635111048s) [noise=true , address=145.79.227.14:4837 , piece_id=YBBFYSC22KSQK5GENOT2NU624PDLUNX2TWQ24RNRW3WMGJT6P7RA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.084303179 (181h45m44.63577364s) [noise=true , address=145.79.227.17:4971 , piece_id=LBNF3Z5A3NZGUQKCWAIW4XPONBC2DD324VXP44FFCQ5FEXA5FZOA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.085550574 (181h45m44.637021025s) [noise=true , address=145.79.227.20:4439 , piece_id=PLNGHRV2N3GPQYYSGU7TKL2FMFCU4JQWXPM4K4HUHH3WKKLLQYVQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.091014480 (181h45m44.642484931s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:47.811332199 (181h45m46.36280266s) [upload=160073]: Next returned nil segment Nov 16 21:49:47.811354462 (181h45m46.362824913s) [upload=160073]: Waiting for error group managing segments... Nov 16 21:49:48.087848677 (181h45m46.639319128s) [noise=true , address=145.79.227.32:4834 , piece_id=4GON7GI63RXLBPJAJM32MO5BRMDTUFKJCRIJQDDWYDVOZA64LXGQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.105693638 (181h45m46.657164099s) [noise=true , address=145.79.227.25:4617 , piece_id=4H4BTA7PSPVFR7FJ7TCQCUOQPRVJCICXSBXUR6XJEZV5NSIBRZWQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.106011864 (181h45m46.657482335s) [noise=true , address=145.79.227.10:4401 , piece_id=WIRM4I7DVYSHDQOEVCH23GRAC2G6HTQ2USSVL2HDTL6MO3VQQQJA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.106913180 (181h45m46.658383641s) [noise=true , address=145.79.227.15:4876 , piece_id=RSNHRRILCRJPBG6KJZTVFCQAROWHF5C6RBIB7XGSQH2SV3G6HUYQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.107526719 (181h45m46.65899718s) [noise=true , address=145.79.227.15:4873 , piece_id=OHIKS4CCPNMIVJAVNDZCPDXRL4PZLBLNMX6T6FV67N7S3BL75DZQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.114554281 (181h45m46.666024732s) [noise=true , address=145.79.227.30:4761 , piece_id=I256J6YH4LHPL4DU6H4P7NYFVJRI2XBV2AFJMASE2HPPDMO6QXLA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.119027128 (181h45m46.670497589s) [noise=true , address=145.79.227.11:4730 , piece_id=CPUKLG5IGGB3RLTVTPSXHCW6T25KJ3P7XNAMKOCRMOJ7UC4EVKDA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.133964262 (181h45m46.685434713s) [noise=true , address=145.79.227.20:4439 , piece_id=PLNGHRV2N3GPQYYSGU7TKL2FMFCU4JQWXPM4K4HUHH3WKKLLQYVQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.146016122 (181h45m46.697486573s) [noise=true , address=145.79.227.17:4971 , piece_id=LBNF3Z5A3NZGUQKCWAIW4XPONBC2DD324VXP44FFCQ5FEXA5FZOA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.153123927 (181h45m46.704594388s) [noise=true , address=145.79.227.31:4798 , piece_id=MBH7EMPFU4UKRIDVAHD6B7COGTRYBUZ3XS2L2UVCJ5ONXEJE4POQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.174476267 (181h45m46.725946728s) [noise=true , address=145.79.227.9:4366 , piece_id=SI7YJXWUTUXDVPNBYCLDPPVRE66RXHLPBLB43A45TND34UO4TF7Q, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.197067667 (181h45m46.748538128s) [noise=true , address=145.79.227.14:4837 , piece_id=YBBFYSC22KSQK5GENOT2NU624PDLUNX2TWQ24RNRW3WMGJT6P7RA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.238467652 (181h45m46.789938113s) [noise=true , address=77.90.31.14:4663 , piece_id=W4MIXY47CPR4IYA6WCREO4VC5BVGAFBJB2BJV5ZNODIIHEXJT2VQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.238492430 (181h45m46.789962891s) [seg_pos={0 0}, upload=160073]: Segment reached optimal threshold of 13 pieces. Nov 16 21:49:48.238743558 (181h45m46.790214019s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.238753988 (181h45m46.790224449s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.238823801 (181h45m46.790294262s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.238844991 (181h45m46.790315442s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.239032308 (181h45m46.790502769s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.239041596 (181h45m46.790512047s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.239195508 (181h45m46.790665959s) [seg_pos={0 0}, upload=160073]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 21:49:48.239303494 (181h45m46.790773955s) [upload=160073]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 21:49:48.239313143 (181h45m46.790783594s) [upload=160073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 21:49:53.369411932 (181h45m51.920882393s) [upload=160073]: Done uploading segments... Nov 16 21:49:53.369436949 (181h45m51.92090741s) [upload=160073]: Upload finished. err: <nil> Duration: 5.981715883s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............X............................................] duration: 3.551847211s trace: https://jaeger.dev.storj.io/trace/251e9c48735b3b0a FAIL (2025-11-16 22:33:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.551890394s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 6.145169504s trace: https://jaeger.dev.storj.io/trace/1533f838bd5c565 FAIL (2025-11-16 22:33:38 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.145204299s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.978624939s trace: https://jaeger.dev.storj.io/trace/1533f838bd5c565 FAIL (2025-11-16 22:33:16 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.978658083s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.934857958s trace: https://jaeger.dev.storj.io/trace/6c6e7bbb319587a2 FAIL (2025-11-16 22:33:04 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 16 21:49:44.857632528 (181h45m43.409102979s) [upload=160073]: Starting upload Nov 16 21:49:44.857711077 (181h45m43.409181538s) [upload=160073]: Done starting upload Nov 16 21:49:44.857867525 (181h45m43.409337986s) [upload=160073]: Uploading segments... Nov 16 21:49:44.858117080 (181h45m43.409587531s) [upload=160073]: Got next segment. Inline: false Nov 16 21:49:44.858184298 (181h45m43.409654749s) [upload=160073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 21:49:44.858191282 (181h45m43.409661733s) [upload=160073]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 21:49:46.079436121 (181h45m44.630906582s) [seg_pos={0 0}, upload=160073]: Begin upload segment... Nov 16 21:49:46.079498500 (181h45m44.630968961s) [seg_pos={0 0}, upload=160073]: Done begin upload segment. Nov 16 21:49:46.079551421 (181h45m44.631021882s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.079592329 (181h45m44.63106279s) [noise=true , address=145.79.227.25:4617 , piece_id=4H4BTA7PSPVFR7FJ7TCQCUOQPRVJCICXSBXUR6XJEZV5NSIBRZWQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.079721945 (181h45m44.631192406s) [noise=true , address=145.79.227.9:4366 , piece_id=SI7YJXWUTUXDVPNBYCLDPPVRE66RXHLPBLB43A45TND34UO4TF7Q, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080339301 (181h45m44.631809762s) [noise=true , address=145.79.227.10:4401 , piece_id=WIRM4I7DVYSHDQOEVCH23GRAC2G6HTQ2USSVL2HDTL6MO3VQQQJA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080638140 (181h45m44.632108601s) [noise=true , address=145.79.227.11:4730 , piece_id=CPUKLG5IGGB3RLTVTPSXHCW6T25KJ3P7XNAMKOCRMOJ7UC4EVKDA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080980683 (181h45m44.632451144s) [noise=true , address=145.79.227.30:4761 , piece_id=I256J6YH4LHPL4DU6H4P7NYFVJRI2XBV2AFJMASE2HPPDMO6QXLA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080993116 (181h45m44.632463577s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.081391766 (181h45m44.632862227s) [noise=true , address=145.79.227.15:4876 , piece_id=RSNHRRILCRJPBG6KJZTVFCQAROWHF5C6RBIB7XGSQH2SV3G6HUYQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.081400582 (181h45m44.632871043s) [noise=true , address=145.79.227.32:4834 , piece_id=4GON7GI63RXLBPJAJM32MO5BRMDTUFKJCRIJQDDWYDVOZA64LXGQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.082534582 (181h45m44.634005043s) [noise=true , address=145.79.227.15:4873 , piece_id=OHIKS4CCPNMIVJAVNDZCPDXRL4PZLBLNMX6T6FV67N7S3BL75DZQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.082659149 (181h45m44.63412961s) [noise=true , address=77.90.31.14:4663 , piece_id=W4MIXY47CPR4IYA6WCREO4VC5BVGAFBJB2BJV5ZNODIIHEXJT2VQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.083189538 (181h45m44.634659989s) [noise=true , address=145.79.227.31:4798 , piece_id=MBH7EMPFU4UKRIDVAHD6B7COGTRYBUZ3XS2L2UVCJ5ONXEJE4POQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.083640587 (181h45m44.635111048s) [noise=true , address=145.79.227.14:4837 , piece_id=YBBFYSC22KSQK5GENOT2NU624PDLUNX2TWQ24RNRW3WMGJT6P7RA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.084303179 (181h45m44.63577364s) [noise=true , address=145.79.227.17:4971 , piece_id=LBNF3Z5A3NZGUQKCWAIW4XPONBC2DD324VXP44FFCQ5FEXA5FZOA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.085550574 (181h45m44.637021025s) [noise=true , address=145.79.227.20:4439 , piece_id=PLNGHRV2N3GPQYYSGU7TKL2FMFCU4JQWXPM4K4HUHH3WKKLLQYVQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.091014480 (181h45m44.642484931s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:47.811332199 (181h45m46.36280266s) [upload=160073]: Next returned nil segment Nov 16 21:49:47.811354462 (181h45m46.362824913s) [upload=160073]: Waiting for error group managing segments... Nov 16 21:49:48.087848677 (181h45m46.639319128s) [noise=true , address=145.79.227.32:4834 , piece_id=4GON7GI63RXLBPJAJM32MO5BRMDTUFKJCRIJQDDWYDVOZA64LXGQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.105693638 (181h45m46.657164099s) [noise=true , address=145.79.227.25:4617 , piece_id=4H4BTA7PSPVFR7FJ7TCQCUOQPRVJCICXSBXUR6XJEZV5NSIBRZWQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.106011864 (181h45m46.657482335s) [noise=true , address=145.79.227.10:4401 , piece_id=WIRM4I7DVYSHDQOEVCH23GRAC2G6HTQ2USSVL2HDTL6MO3VQQQJA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.106913180 (181h45m46.658383641s) [noise=true , address=145.79.227.15:4876 , piece_id=RSNHRRILCRJPBG6KJZTVFCQAROWHF5C6RBIB7XGSQH2SV3G6HUYQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.107526719 (181h45m46.65899718s) [noise=true , address=145.79.227.15:4873 , piece_id=OHIKS4CCPNMIVJAVNDZCPDXRL4PZLBLNMX6T6FV67N7S3BL75DZQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.114554281 (181h45m46.666024732s) [noise=true , address=145.79.227.30:4761 , piece_id=I256J6YH4LHPL4DU6H4P7NYFVJRI2XBV2AFJMASE2HPPDMO6QXLA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.119027128 (181h45m46.670497589s) [noise=true , address=145.79.227.11:4730 , piece_id=CPUKLG5IGGB3RLTVTPSXHCW6T25KJ3P7XNAMKOCRMOJ7UC4EVKDA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.133964262 (181h45m46.685434713s) [noise=true , address=145.79.227.20:4439 , piece_id=PLNGHRV2N3GPQYYSGU7TKL2FMFCU4JQWXPM4K4HUHH3WKKLLQYVQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.146016122 (181h45m46.697486573s) [noise=true , address=145.79.227.17:4971 , piece_id=LBNF3Z5A3NZGUQKCWAIW4XPONBC2DD324VXP44FFCQ5FEXA5FZOA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.153123927 (181h45m46.704594388s) [noise=true , address=145.79.227.31:4798 , piece_id=MBH7EMPFU4UKRIDVAHD6B7COGTRYBUZ3XS2L2UVCJ5ONXEJE4POQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.174476267 (181h45m46.725946728s) [noise=true , address=145.79.227.9:4366 , piece_id=SI7YJXWUTUXDVPNBYCLDPPVRE66RXHLPBLB43A45TND34UO4TF7Q, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.197067667 (181h45m46.748538128s) [noise=true , address=145.79.227.14:4837 , piece_id=YBBFYSC22KSQK5GENOT2NU624PDLUNX2TWQ24RNRW3WMGJT6P7RA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.238467652 (181h45m46.789938113s) [noise=true , address=77.90.31.14:4663 , piece_id=W4MIXY47CPR4IYA6WCREO4VC5BVGAFBJB2BJV5ZNODIIHEXJT2VQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.238492430 (181h45m46.789962891s) [seg_pos={0 0}, upload=160073]: Segment reached optimal threshold of 13 pieces. Nov 16 21:49:48.238743558 (181h45m46.790214019s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.238753988 (181h45m46.790224449s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.238823801 (181h45m46.790294262s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.238844991 (181h45m46.790315442s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.239032308 (181h45m46.790502769s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.239041596 (181h45m46.790512047s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.239195508 (181h45m46.790665959s) [seg_pos={0 0}, upload=160073]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 21:49:48.239303494 (181h45m46.790773955s) [upload=160073]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 21:49:48.239313143 (181h45m46.790783594s) [upload=160073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 21:49:53.369411932 (181h45m51.920882393s) [upload=160073]: Done uploading segments... Nov 16 21:49:53.369436949 (181h45m51.92090741s) [upload=160073]: Upload finished. err: <nil> Duration: 5.934891841s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................X...........................................] duration: 3.429117012s trace: https://jaeger.dev.storj.io/trace/251e9c48735b3b0a FAIL (2025-11-16 22:32:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.429146819s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 6.117958117s trace: https://jaeger.dev.storj.io/trace/1533f838bd5c565 FAIL (2025-11-16 22:32:19 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.117993133s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.918394904s trace: https://jaeger.dev.storj.io/trace/6c6e7bbb319587a2 FAIL (2025-11-16 22:32:00 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 16 21:49:44.857632528 (181h45m43.409102979s) [upload=160073]: Starting upload Nov 16 21:49:44.857711077 (181h45m43.409181538s) [upload=160073]: Done starting upload Nov 16 21:49:44.857867525 (181h45m43.409337986s) [upload=160073]: Uploading segments... Nov 16 21:49:44.858117080 (181h45m43.409587531s) [upload=160073]: Got next segment. Inline: false Nov 16 21:49:44.858184298 (181h45m43.409654749s) [upload=160073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 21:49:44.858191282 (181h45m43.409661733s) [upload=160073]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 21:49:46.079436121 (181h45m44.630906582s) [seg_pos={0 0}, upload=160073]: Begin upload segment... Nov 16 21:49:46.079498500 (181h45m44.630968961s) [seg_pos={0 0}, upload=160073]: Done begin upload segment. Nov 16 21:49:46.079551421 (181h45m44.631021882s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.079592329 (181h45m44.63106279s) [noise=true , address=145.79.227.25:4617 , piece_id=4H4BTA7PSPVFR7FJ7TCQCUOQPRVJCICXSBXUR6XJEZV5NSIBRZWQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.079721945 (181h45m44.631192406s) [noise=true , address=145.79.227.9:4366 , piece_id=SI7YJXWUTUXDVPNBYCLDPPVRE66RXHLPBLB43A45TND34UO4TF7Q, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080339301 (181h45m44.631809762s) [noise=true , address=145.79.227.10:4401 , piece_id=WIRM4I7DVYSHDQOEVCH23GRAC2G6HTQ2USSVL2HDTL6MO3VQQQJA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080638140 (181h45m44.632108601s) [noise=true , address=145.79.227.11:4730 , piece_id=CPUKLG5IGGB3RLTVTPSXHCW6T25KJ3P7XNAMKOCRMOJ7UC4EVKDA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080980683 (181h45m44.632451144s) [noise=true , address=145.79.227.30:4761 , piece_id=I256J6YH4LHPL4DU6H4P7NYFVJRI2XBV2AFJMASE2HPPDMO6QXLA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080993116 (181h45m44.632463577s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.081391766 (181h45m44.632862227s) [noise=true , address=145.79.227.15:4876 , piece_id=RSNHRRILCRJPBG6KJZTVFCQAROWHF5C6RBIB7XGSQH2SV3G6HUYQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.081400582 (181h45m44.632871043s) [noise=true , address=145.79.227.32:4834 , piece_id=4GON7GI63RXLBPJAJM32MO5BRMDTUFKJCRIJQDDWYDVOZA64LXGQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.082534582 (181h45m44.634005043s) [noise=true , address=145.79.227.15:4873 , piece_id=OHIKS4CCPNMIVJAVNDZCPDXRL4PZLBLNMX6T6FV67N7S3BL75DZQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.082659149 (181h45m44.63412961s) [noise=true , address=77.90.31.14:4663 , piece_id=W4MIXY47CPR4IYA6WCREO4VC5BVGAFBJB2BJV5ZNODIIHEXJT2VQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.083189538 (181h45m44.634659989s) [noise=true , address=145.79.227.31:4798 , piece_id=MBH7EMPFU4UKRIDVAHD6B7COGTRYBUZ3XS2L2UVCJ5ONXEJE4POQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.083640587 (181h45m44.635111048s) [noise=true , address=145.79.227.14:4837 , piece_id=YBBFYSC22KSQK5GENOT2NU624PDLUNX2TWQ24RNRW3WMGJT6P7RA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.084303179 (181h45m44.63577364s) [noise=true , address=145.79.227.17:4971 , piece_id=LBNF3Z5A3NZGUQKCWAIW4XPONBC2DD324VXP44FFCQ5FEXA5FZOA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.085550574 (181h45m44.637021025s) [noise=true , address=145.79.227.20:4439 , piece_id=PLNGHRV2N3GPQYYSGU7TKL2FMFCU4JQWXPM4K4HUHH3WKKLLQYVQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.091014480 (181h45m44.642484931s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:47.811332199 (181h45m46.36280266s) [upload=160073]: Next returned nil segment Nov 16 21:49:47.811354462 (181h45m46.362824913s) [upload=160073]: Waiting for error group managing segments... Nov 16 21:49:48.087848677 (181h45m46.639319128s) [noise=true , address=145.79.227.32:4834 , piece_id=4GON7GI63RXLBPJAJM32MO5BRMDTUFKJCRIJQDDWYDVOZA64LXGQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.105693638 (181h45m46.657164099s) [noise=true , address=145.79.227.25:4617 , piece_id=4H4BTA7PSPVFR7FJ7TCQCUOQPRVJCICXSBXUR6XJEZV5NSIBRZWQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.106011864 (181h45m46.657482335s) [noise=true , address=145.79.227.10:4401 , piece_id=WIRM4I7DVYSHDQOEVCH23GRAC2G6HTQ2USSVL2HDTL6MO3VQQQJA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.106913180 (181h45m46.658383641s) [noise=true , address=145.79.227.15:4876 , piece_id=RSNHRRILCRJPBG6KJZTVFCQAROWHF5C6RBIB7XGSQH2SV3G6HUYQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.107526719 (181h45m46.65899718s) [noise=true , address=145.79.227.15:4873 , piece_id=OHIKS4CCPNMIVJAVNDZCPDXRL4PZLBLNMX6T6FV67N7S3BL75DZQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.114554281 (181h45m46.666024732s) [noise=true , address=145.79.227.30:4761 , piece_id=I256J6YH4LHPL4DU6H4P7NYFVJRI2XBV2AFJMASE2HPPDMO6QXLA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.119027128 (181h45m46.670497589s) [noise=true , address=145.79.227.11:4730 , piece_id=CPUKLG5IGGB3RLTVTPSXHCW6T25KJ3P7XNAMKOCRMOJ7UC4EVKDA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.133964262 (181h45m46.685434713s) [noise=true , address=145.79.227.20:4439 , piece_id=PLNGHRV2N3GPQYYSGU7TKL2FMFCU4JQWXPM4K4HUHH3WKKLLQYVQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.146016122 (181h45m46.697486573s) [noise=true , address=145.79.227.17:4971 , piece_id=LBNF3Z5A3NZGUQKCWAIW4XPONBC2DD324VXP44FFCQ5FEXA5FZOA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.153123927 (181h45m46.704594388s) [noise=true , address=145.79.227.31:4798 , piece_id=MBH7EMPFU4UKRIDVAHD6B7COGTRYBUZ3XS2L2UVCJ5ONXEJE4POQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.174476267 (181h45m46.725946728s) [noise=true , address=145.79.227.9:4366 , piece_id=SI7YJXWUTUXDVPNBYCLDPPVRE66RXHLPBLB43A45TND34UO4TF7Q, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.197067667 (181h45m46.748538128s) [noise=true , address=145.79.227.14:4837 , piece_id=YBBFYSC22KSQK5GENOT2NU624PDLUNX2TWQ24RNRW3WMGJT6P7RA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.238467652 (181h45m46.789938113s) [noise=true , address=77.90.31.14:4663 , piece_id=W4MIXY47CPR4IYA6WCREO4VC5BVGAFBJB2BJV5ZNODIIHEXJT2VQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.238492430 (181h45m46.789962891s) [seg_pos={0 0}, upload=160073]: Segment reached optimal threshold of 13 pieces. Nov 16 21:49:48.238743558 (181h45m46.790214019s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.238753988 (181h45m46.790224449s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.238823801 (181h45m46.790294262s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.238844991 (181h45m46.790315442s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.239032308 (181h45m46.790502769s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.239041596 (181h45m46.790512047s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.239195508 (181h45m46.790665959s) [seg_pos={0 0}, upload=160073]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 21:49:48.239303494 (181h45m46.790773955s) [upload=160073]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 21:49:48.239313143 (181h45m46.790783594s) [upload=160073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 21:49:53.369411932 (181h45m51.920882393s) [upload=160073]: Done uploading segments... Nov 16 21:49:53.369436949 (181h45m51.92090741s) [upload=160073]: Upload finished. err: <nil> Duration: 5.918418951s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.968574668s trace: https://jaeger.dev.storj.io/trace/1533f838bd5c565 FAIL (2025-11-16 22:31:59 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.968603223s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 1.424962891s trace: https://jaeger.dev.storj.io/trace/ef0652f4a47c08 FAIL (2025-11-16 22:31:54 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.424997707s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................X..........................................] duration: 3.458855553s trace: https://jaeger.dev.storj.io/trace/251e9c48735b3b0a FAIL (2025-11-16 22:31:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.45888523s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 6.087850176s trace: https://jaeger.dev.storj.io/trace/1533f838bd5c565 FAIL (2025-11-16 22:31:40 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.087885934s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 1.4618751s trace: https://jaeger.dev.storj.io/trace/ef0652f4a47c08 FAIL (2025-11-16 22:31:24 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.461903324s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.01199347s trace: https://jaeger.dev.storj.io/trace/6c6e7bbb319587a2 FAIL (2025-11-16 22:31:18 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 16 21:49:44.857632528 (181h45m43.409102979s) [upload=160073]: Starting upload Nov 16 21:49:44.857711077 (181h45m43.409181538s) [upload=160073]: Done starting upload Nov 16 21:49:44.857867525 (181h45m43.409337986s) [upload=160073]: Uploading segments... Nov 16 21:49:44.858117080 (181h45m43.409587531s) [upload=160073]: Got next segment. Inline: false Nov 16 21:49:44.858184298 (181h45m43.409654749s) [upload=160073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 21:49:44.858191282 (181h45m43.409661733s) [upload=160073]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 21:49:46.079436121 (181h45m44.630906582s) [seg_pos={0 0}, upload=160073]: Begin upload segment... Nov 16 21:49:46.079498500 (181h45m44.630968961s) [seg_pos={0 0}, upload=160073]: Done begin upload segment. Nov 16 21:49:46.079551421 (181h45m44.631021882s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.079592329 (181h45m44.63106279s) [noise=true , address=145.79.227.25:4617 , piece_id=4H4BTA7PSPVFR7FJ7TCQCUOQPRVJCICXSBXUR6XJEZV5NSIBRZWQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.079721945 (181h45m44.631192406s) [noise=true , address=145.79.227.9:4366 , piece_id=SI7YJXWUTUXDVPNBYCLDPPVRE66RXHLPBLB43A45TND34UO4TF7Q, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080339301 (181h45m44.631809762s) [noise=true , address=145.79.227.10:4401 , piece_id=WIRM4I7DVYSHDQOEVCH23GRAC2G6HTQ2USSVL2HDTL6MO3VQQQJA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080638140 (181h45m44.632108601s) [noise=true , address=145.79.227.11:4730 , piece_id=CPUKLG5IGGB3RLTVTPSXHCW6T25KJ3P7XNAMKOCRMOJ7UC4EVKDA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080980683 (181h45m44.632451144s) [noise=true , address=145.79.227.30:4761 , piece_id=I256J6YH4LHPL4DU6H4P7NYFVJRI2XBV2AFJMASE2HPPDMO6QXLA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080993116 (181h45m44.632463577s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.081391766 (181h45m44.632862227s) [noise=true , address=145.79.227.15:4876 , piece_id=RSNHRRILCRJPBG6KJZTVFCQAROWHF5C6RBIB7XGSQH2SV3G6HUYQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.081400582 (181h45m44.632871043s) [noise=true , address=145.79.227.32:4834 , piece_id=4GON7GI63RXLBPJAJM32MO5BRMDTUFKJCRIJQDDWYDVOZA64LXGQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.082534582 (181h45m44.634005043s) [noise=true , address=145.79.227.15:4873 , piece_id=OHIKS4CCPNMIVJAVNDZCPDXRL4PZLBLNMX6T6FV67N7S3BL75DZQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.082659149 (181h45m44.63412961s) [noise=true , address=77.90.31.14:4663 , piece_id=W4MIXY47CPR4IYA6WCREO4VC5BVGAFBJB2BJV5ZNODIIHEXJT2VQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.083189538 (181h45m44.634659989s) [noise=true , address=145.79.227.31:4798 , piece_id=MBH7EMPFU4UKRIDVAHD6B7COGTRYBUZ3XS2L2UVCJ5ONXEJE4POQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.083640587 (181h45m44.635111048s) [noise=true , address=145.79.227.14:4837 , piece_id=YBBFYSC22KSQK5GENOT2NU624PDLUNX2TWQ24RNRW3WMGJT6P7RA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.084303179 (181h45m44.63577364s) [noise=true , address=145.79.227.17:4971 , piece_id=LBNF3Z5A3NZGUQKCWAIW4XPONBC2DD324VXP44FFCQ5FEXA5FZOA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.085550574 (181h45m44.637021025s) [noise=true , address=145.79.227.20:4439 , piece_id=PLNGHRV2N3GPQYYSGU7TKL2FMFCU4JQWXPM4K4HUHH3WKKLLQYVQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.091014480 (181h45m44.642484931s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:47.811332199 (181h45m46.36280266s) [upload=160073]: Next returned nil segment Nov 16 21:49:47.811354462 (181h45m46.362824913s) [upload=160073]: Waiting for error group managing segments... Nov 16 21:49:48.087848677 (181h45m46.639319128s) [noise=true , address=145.79.227.32:4834 , piece_id=4GON7GI63RXLBPJAJM32MO5BRMDTUFKJCRIJQDDWYDVOZA64LXGQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.105693638 (181h45m46.657164099s) [noise=true , address=145.79.227.25:4617 , piece_id=4H4BTA7PSPVFR7FJ7TCQCUOQPRVJCICXSBXUR6XJEZV5NSIBRZWQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.106011864 (181h45m46.657482335s) [noise=true , address=145.79.227.10:4401 , piece_id=WIRM4I7DVYSHDQOEVCH23GRAC2G6HTQ2USSVL2HDTL6MO3VQQQJA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.106913180 (181h45m46.658383641s) [noise=true , address=145.79.227.15:4876 , piece_id=RSNHRRILCRJPBG6KJZTVFCQAROWHF5C6RBIB7XGSQH2SV3G6HUYQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.107526719 (181h45m46.65899718s) [noise=true , address=145.79.227.15:4873 , piece_id=OHIKS4CCPNMIVJAVNDZCPDXRL4PZLBLNMX6T6FV67N7S3BL75DZQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.114554281 (181h45m46.666024732s) [noise=true , address=145.79.227.30:4761 , piece_id=I256J6YH4LHPL4DU6H4P7NYFVJRI2XBV2AFJMASE2HPPDMO6QXLA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.119027128 (181h45m46.670497589s) [noise=true , address=145.79.227.11:4730 , piece_id=CPUKLG5IGGB3RLTVTPSXHCW6T25KJ3P7XNAMKOCRMOJ7UC4EVKDA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.133964262 (181h45m46.685434713s) [noise=true , address=145.79.227.20:4439 , piece_id=PLNGHRV2N3GPQYYSGU7TKL2FMFCU4JQWXPM4K4HUHH3WKKLLQYVQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.146016122 (181h45m46.697486573s) [noise=true , address=145.79.227.17:4971 , piece_id=LBNF3Z5A3NZGUQKCWAIW4XPONBC2DD324VXP44FFCQ5FEXA5FZOA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.153123927 (181h45m46.704594388s) [noise=true , address=145.79.227.31:4798 , piece_id=MBH7EMPFU4UKRIDVAHD6B7COGTRYBUZ3XS2L2UVCJ5ONXEJE4POQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.174476267 (181h45m46.725946728s) [noise=true , address=145.79.227.9:4366 , piece_id=SI7YJXWUTUXDVPNBYCLDPPVRE66RXHLPBLB43A45TND34UO4TF7Q, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.197067667 (181h45m46.748538128s) [noise=true , address=145.79.227.14:4837 , piece_id=YBBFYSC22KSQK5GENOT2NU624PDLUNX2TWQ24RNRW3WMGJT6P7RA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.238467652 (181h45m46.789938113s) [noise=true , address=77.90.31.14:4663 , piece_id=W4MIXY47CPR4IYA6WCREO4VC5BVGAFBJB2BJV5ZNODIIHEXJT2VQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.238492430 (181h45m46.789962891s) [seg_pos={0 0}, upload=160073]: Segment reached optimal threshold of 13 pieces. Nov 16 21:49:48.238743558 (181h45m46.790214019s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.238753988 (181h45m46.790224449s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.238823801 (181h45m46.790294262s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.238844991 (181h45m46.790315442s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.239032308 (181h45m46.790502769s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.239041596 (181h45m46.790512047s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.239195508 (181h45m46.790665959s) [seg_pos={0 0}, upload=160073]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 21:49:48.239303494 (181h45m46.790773955s) [upload=160073]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 21:49:48.239313143 (181h45m46.790783594s) [upload=160073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 21:49:53.369411932 (181h45m51.920882393s) [upload=160073]: Done uploading segments... Nov 16 21:49:53.369436949 (181h45m51.92090741s) [upload=160073]: Upload finished. err: <nil> Duration: 6.012054435s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 6.064875434s trace: https://jaeger.dev.storj.io/trace/6c6e7bbb319587a2 FAIL (2025-11-16 22:30:56 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 16 21:49:44.857632528 (181h45m43.409102979s) [upload=160073]: Starting upload Nov 16 21:49:44.857711077 (181h45m43.409181538s) [upload=160073]: Done starting upload Nov 16 21:49:44.857867525 (181h45m43.409337986s) [upload=160073]: Uploading segments... Nov 16 21:49:44.858117080 (181h45m43.409587531s) [upload=160073]: Got next segment. Inline: false Nov 16 21:49:44.858184298 (181h45m43.409654749s) [upload=160073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 21:49:44.858191282 (181h45m43.409661733s) [upload=160073]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 21:49:46.079436121 (181h45m44.630906582s) [seg_pos={0 0}, upload=160073]: Begin upload segment... Nov 16 21:49:46.079498500 (181h45m44.630968961s) [seg_pos={0 0}, upload=160073]: Done begin upload segment. Nov 16 21:49:46.079551421 (181h45m44.631021882s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.079592329 (181h45m44.63106279s) [noise=true , address=145.79.227.25:4617 , piece_id=4H4BTA7PSPVFR7FJ7TCQCUOQPRVJCICXSBXUR6XJEZV5NSIBRZWQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.079721945 (181h45m44.631192406s) [noise=true , address=145.79.227.9:4366 , piece_id=SI7YJXWUTUXDVPNBYCLDPPVRE66RXHLPBLB43A45TND34UO4TF7Q, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080339301 (181h45m44.631809762s) [noise=true , address=145.79.227.10:4401 , piece_id=WIRM4I7DVYSHDQOEVCH23GRAC2G6HTQ2USSVL2HDTL6MO3VQQQJA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080638140 (181h45m44.632108601s) [noise=true , address=145.79.227.11:4730 , piece_id=CPUKLG5IGGB3RLTVTPSXHCW6T25KJ3P7XNAMKOCRMOJ7UC4EVKDA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080980683 (181h45m44.632451144s) [noise=true , address=145.79.227.30:4761 , piece_id=I256J6YH4LHPL4DU6H4P7NYFVJRI2XBV2AFJMASE2HPPDMO6QXLA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080993116 (181h45m44.632463577s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.081391766 (181h45m44.632862227s) [noise=true , address=145.79.227.15:4876 , piece_id=RSNHRRILCRJPBG6KJZTVFCQAROWHF5C6RBIB7XGSQH2SV3G6HUYQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.081400582 (181h45m44.632871043s) [noise=true , address=145.79.227.32:4834 , piece_id=4GON7GI63RXLBPJAJM32MO5BRMDTUFKJCRIJQDDWYDVOZA64LXGQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.082534582 (181h45m44.634005043s) [noise=true , address=145.79.227.15:4873 , piece_id=OHIKS4CCPNMIVJAVNDZCPDXRL4PZLBLNMX6T6FV67N7S3BL75DZQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.082659149 (181h45m44.63412961s) [noise=true , address=77.90.31.14:4663 , piece_id=W4MIXY47CPR4IYA6WCREO4VC5BVGAFBJB2BJV5ZNODIIHEXJT2VQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.083189538 (181h45m44.634659989s) [noise=true , address=145.79.227.31:4798 , piece_id=MBH7EMPFU4UKRIDVAHD6B7COGTRYBUZ3XS2L2UVCJ5ONXEJE4POQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.083640587 (181h45m44.635111048s) [noise=true , address=145.79.227.14:4837 , piece_id=YBBFYSC22KSQK5GENOT2NU624PDLUNX2TWQ24RNRW3WMGJT6P7RA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.084303179 (181h45m44.63577364s) [noise=true , address=145.79.227.17:4971 , piece_id=LBNF3Z5A3NZGUQKCWAIW4XPONBC2DD324VXP44FFCQ5FEXA5FZOA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.085550574 (181h45m44.637021025s) [noise=true , address=145.79.227.20:4439 , piece_id=PLNGHRV2N3GPQYYSGU7TKL2FMFCU4JQWXPM4K4HUHH3WKKLLQYVQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.091014480 (181h45m44.642484931s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:47.811332199 (181h45m46.36280266s) [upload=160073]: Next returned nil segment Nov 16 21:49:47.811354462 (181h45m46.362824913s) [upload=160073]: Waiting for error group managing segments... Nov 16 21:49:48.087848677 (181h45m46.639319128s) [noise=true , address=145.79.227.32:4834 , piece_id=4GON7GI63RXLBPJAJM32MO5BRMDTUFKJCRIJQDDWYDVOZA64LXGQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.105693638 (181h45m46.657164099s) [noise=true , address=145.79.227.25:4617 , piece_id=4H4BTA7PSPVFR7FJ7TCQCUOQPRVJCICXSBXUR6XJEZV5NSIBRZWQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.106011864 (181h45m46.657482335s) [noise=true , address=145.79.227.10:4401 , piece_id=WIRM4I7DVYSHDQOEVCH23GRAC2G6HTQ2USSVL2HDTL6MO3VQQQJA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.106913180 (181h45m46.658383641s) [noise=true , address=145.79.227.15:4876 , piece_id=RSNHRRILCRJPBG6KJZTVFCQAROWHF5C6RBIB7XGSQH2SV3G6HUYQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.107526719 (181h45m46.65899718s) [noise=true , address=145.79.227.15:4873 , piece_id=OHIKS4CCPNMIVJAVNDZCPDXRL4PZLBLNMX6T6FV67N7S3BL75DZQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.114554281 (181h45m46.666024732s) [noise=true , address=145.79.227.30:4761 , piece_id=I256J6YH4LHPL4DU6H4P7NYFVJRI2XBV2AFJMASE2HPPDMO6QXLA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.119027128 (181h45m46.670497589s) [noise=true , address=145.79.227.11:4730 , piece_id=CPUKLG5IGGB3RLTVTPSXHCW6T25KJ3P7XNAMKOCRMOJ7UC4EVKDA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.133964262 (181h45m46.685434713s) [noise=true , address=145.79.227.20:4439 , piece_id=PLNGHRV2N3GPQYYSGU7TKL2FMFCU4JQWXPM4K4HUHH3WKKLLQYVQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.146016122 (181h45m46.697486573s) [noise=true , address=145.79.227.17:4971 , piece_id=LBNF3Z5A3NZGUQKCWAIW4XPONBC2DD324VXP44FFCQ5FEXA5FZOA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.153123927 (181h45m46.704594388s) [noise=true , address=145.79.227.31:4798 , piece_id=MBH7EMPFU4UKRIDVAHD6B7COGTRYBUZ3XS2L2UVCJ5ONXEJE4POQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.174476267 (181h45m46.725946728s) [noise=true , address=145.79.227.9:4366 , piece_id=SI7YJXWUTUXDVPNBYCLDPPVRE66RXHLPBLB43A45TND34UO4TF7Q, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.197067667 (181h45m46.748538128s) [noise=true , address=145.79.227.14:4837 , piece_id=YBBFYSC22KSQK5GENOT2NU624PDLUNX2TWQ24RNRW3WMGJT6P7RA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.238467652 (181h45m46.789938113s) [noise=true , address=77.90.31.14:4663 , piece_id=W4MIXY47CPR4IYA6WCREO4VC5BVGAFBJB2BJV5ZNODIIHEXJT2VQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.238492430 (181h45m46.789962891s) [seg_pos={0 0}, upload=160073]: Segment reached optimal threshold of 13 pieces. Nov 16 21:49:48.238743558 (181h45m46.790214019s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.238753988 (181h45m46.790224449s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.238823801 (181h45m46.790294262s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.238844991 (181h45m46.790315442s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.239032308 (181h45m46.790502769s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.239041596 (181h45m46.790512047s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.239195508 (181h45m46.790665959s) [seg_pos={0 0}, upload=160073]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 21:49:48.239303494 (181h45m46.790773955s) [upload=160073]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 21:49:48.239313143 (181h45m46.790783594s) [upload=160073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 21:49:53.369411932 (181h45m51.920882393s) [upload=160073]: Done uploading segments... Nov 16 21:49:53.369436949 (181h45m51.92090741s) [upload=160073]: Upload finished. err: <nil> Duration: 6.064921413s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................X.........................................] duration: 3.478205113s trace: https://jaeger.dev.storj.io/trace/251e9c48735b3b0a FAIL (2025-11-16 22:30:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.478236814s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 1.424730458s trace: https://jaeger.dev.storj.io/trace/ef0652f4a47c08 FAIL (2025-11-16 22:30:48 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.424775344s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 6.169613656s trace: https://jaeger.dev.storj.io/trace/1533f838bd5c565 FAIL (2025-11-16 22:30:31 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.169643413s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 1.456899687s trace: https://jaeger.dev.storj.io/trace/ef0652f4a47c08 FAIL (2025-11-16 22:30:18 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.456936197s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.462994589s trace: https://jaeger.dev.storj.io/trace/1533f838bd5c565 FAIL (2025-11-16 22:30:15 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.463023574s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 6.008046471s trace: https://jaeger.dev.storj.io/trace/6c6e7bbb319587a2 FAIL (2025-11-16 22:30:11 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Nov 16 21:49:44.857632528 (181h45m43.409102979s) [upload=160073]: Starting upload Nov 16 21:49:44.857711077 (181h45m43.409181538s) [upload=160073]: Done starting upload Nov 16 21:49:44.857867525 (181h45m43.409337986s) [upload=160073]: Uploading segments... Nov 16 21:49:44.858117080 (181h45m43.409587531s) [upload=160073]: Got next segment. Inline: false Nov 16 21:49:44.858184298 (181h45m43.409654749s) [upload=160073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 21:49:44.858191282 (181h45m43.409661733s) [upload=160073]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 21:49:46.079436121 (181h45m44.630906582s) [seg_pos={0 0}, upload=160073]: Begin upload segment... Nov 16 21:49:46.079498500 (181h45m44.630968961s) [seg_pos={0 0}, upload=160073]: Done begin upload segment. Nov 16 21:49:46.079551421 (181h45m44.631021882s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.079592329 (181h45m44.63106279s) [noise=true , address=145.79.227.25:4617 , piece_id=4H4BTA7PSPVFR7FJ7TCQCUOQPRVJCICXSBXUR6XJEZV5NSIBRZWQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.079721945 (181h45m44.631192406s) [noise=true , address=145.79.227.9:4366 , piece_id=SI7YJXWUTUXDVPNBYCLDPPVRE66RXHLPBLB43A45TND34UO4TF7Q, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080339301 (181h45m44.631809762s) [noise=true , address=145.79.227.10:4401 , piece_id=WIRM4I7DVYSHDQOEVCH23GRAC2G6HTQ2USSVL2HDTL6MO3VQQQJA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080638140 (181h45m44.632108601s) [noise=true , address=145.79.227.11:4730 , piece_id=CPUKLG5IGGB3RLTVTPSXHCW6T25KJ3P7XNAMKOCRMOJ7UC4EVKDA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080980683 (181h45m44.632451144s) [noise=true , address=145.79.227.30:4761 , piece_id=I256J6YH4LHPL4DU6H4P7NYFVJRI2XBV2AFJMASE2HPPDMO6QXLA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.080993116 (181h45m44.632463577s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.081391766 (181h45m44.632862227s) [noise=true , address=145.79.227.15:4876 , piece_id=RSNHRRILCRJPBG6KJZTVFCQAROWHF5C6RBIB7XGSQH2SV3G6HUYQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.081400582 (181h45m44.632871043s) [noise=true , address=145.79.227.32:4834 , piece_id=4GON7GI63RXLBPJAJM32MO5BRMDTUFKJCRIJQDDWYDVOZA64LXGQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.082534582 (181h45m44.634005043s) [noise=true , address=145.79.227.15:4873 , piece_id=OHIKS4CCPNMIVJAVNDZCPDXRL4PZLBLNMX6T6FV67N7S3BL75DZQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.082659149 (181h45m44.63412961s) [noise=true , address=77.90.31.14:4663 , piece_id=W4MIXY47CPR4IYA6WCREO4VC5BVGAFBJB2BJV5ZNODIIHEXJT2VQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.083189538 (181h45m44.634659989s) [noise=true , address=145.79.227.31:4798 , piece_id=MBH7EMPFU4UKRIDVAHD6B7COGTRYBUZ3XS2L2UVCJ5ONXEJE4POQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.083640587 (181h45m44.635111048s) [noise=true , address=145.79.227.14:4837 , piece_id=YBBFYSC22KSQK5GENOT2NU624PDLUNX2TWQ24RNRW3WMGJT6P7RA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.084303179 (181h45m44.63577364s) [noise=true , address=145.79.227.17:4971 , piece_id=LBNF3Z5A3NZGUQKCWAIW4XPONBC2DD324VXP44FFCQ5FEXA5FZOA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.085550574 (181h45m44.637021025s) [noise=true , address=145.79.227.20:4439 , piece_id=PLNGHRV2N3GPQYYSGU7TKL2FMFCU4JQWXPM4K4HUHH3WKKLLQYVQ, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:46.091014480 (181h45m44.642484931s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Uploading piece... Nov 16 21:49:47.811332199 (181h45m46.36280266s) [upload=160073]: Next returned nil segment Nov 16 21:49:47.811354462 (181h45m46.362824913s) [upload=160073]: Waiting for error group managing segments... Nov 16 21:49:48.087848677 (181h45m46.639319128s) [noise=true , address=145.79.227.32:4834 , piece_id=4GON7GI63RXLBPJAJM32MO5BRMDTUFKJCRIJQDDWYDVOZA64LXGQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.105693638 (181h45m46.657164099s) [noise=true , address=145.79.227.25:4617 , piece_id=4H4BTA7PSPVFR7FJ7TCQCUOQPRVJCICXSBXUR6XJEZV5NSIBRZWQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.106011864 (181h45m46.657482335s) [noise=true , address=145.79.227.10:4401 , piece_id=WIRM4I7DVYSHDQOEVCH23GRAC2G6HTQ2USSVL2HDTL6MO3VQQQJA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.106913180 (181h45m46.658383641s) [noise=true , address=145.79.227.15:4876 , piece_id=RSNHRRILCRJPBG6KJZTVFCQAROWHF5C6RBIB7XGSQH2SV3G6HUYQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.107526719 (181h45m46.65899718s) [noise=true , address=145.79.227.15:4873 , piece_id=OHIKS4CCPNMIVJAVNDZCPDXRL4PZLBLNMX6T6FV67N7S3BL75DZQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.114554281 (181h45m46.666024732s) [noise=true , address=145.79.227.30:4761 , piece_id=I256J6YH4LHPL4DU6H4P7NYFVJRI2XBV2AFJMASE2HPPDMO6QXLA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.119027128 (181h45m46.670497589s) [noise=true , address=145.79.227.11:4730 , piece_id=CPUKLG5IGGB3RLTVTPSXHCW6T25KJ3P7XNAMKOCRMOJ7UC4EVKDA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.133964262 (181h45m46.685434713s) [noise=true , address=145.79.227.20:4439 , piece_id=PLNGHRV2N3GPQYYSGU7TKL2FMFCU4JQWXPM4K4HUHH3WKKLLQYVQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.146016122 (181h45m46.697486573s) [noise=true , address=145.79.227.17:4971 , piece_id=LBNF3Z5A3NZGUQKCWAIW4XPONBC2DD324VXP44FFCQ5FEXA5FZOA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.153123927 (181h45m46.704594388s) [noise=true , address=145.79.227.31:4798 , piece_id=MBH7EMPFU4UKRIDVAHD6B7COGTRYBUZ3XS2L2UVCJ5ONXEJE4POQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.174476267 (181h45m46.725946728s) [noise=true , address=145.79.227.9:4366 , piece_id=SI7YJXWUTUXDVPNBYCLDPPVRE66RXHLPBLB43A45TND34UO4TF7Q, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.197067667 (181h45m46.748538128s) [noise=true , address=145.79.227.14:4837 , piece_id=YBBFYSC22KSQK5GENOT2NU624PDLUNX2TWQ24RNRW3WMGJT6P7RA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.238467652 (181h45m46.789938113s) [noise=true , address=77.90.31.14:4663 , piece_id=W4MIXY47CPR4IYA6WCREO4VC5BVGAFBJB2BJV5ZNODIIHEXJT2VQ, seg_pos={0 0}, upload=160073]: Done uploading piece. err: <nil> Nov 16 21:49:48.238492430 (181h45m46.789962891s) [seg_pos={0 0}, upload=160073]: Segment reached optimal threshold of 13 pieces. Nov 16 21:49:48.238743558 (181h45m46.790214019s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.238753988 (181h45m46.790224449s) [noise=true , address=45.91.116.14:4190 , piece_id=NWVY4TNZTT2Z3OBXPPYXOAQQALIHMGWML2QQLIREWBDHT3GBUDOA, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.238823801 (181h45m46.790294262s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.238844991 (181h45m46.790315442s) [noise=true , address=45.91.116.4:4939 , piece_id=OFWZ2DS67MBSCO7MJ3NYCCUOEBFNDDZ64GSB3XNZ2J4UZJW6BX7Q, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.239032308 (181h45m46.790502769s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Nov 16 21:49:48.239041596 (181h45m46.790512047s) [noise=true , address=45.91.116.6:4975 , piece_id=6PPD4GO4CNNX3A246GZQRXMBUFCKUPHI2L5JAOSNYWXUFBRM4SEA, seg_pos={0 0}, upload=160073]: Long tail cancellation due to optimal threshold Nov 16 21:49:48.239195508 (181h45m46.790665959s) [seg_pos={0 0}, upload=160073]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 21:49:48.239303494 (181h45m46.790773955s) [upload=160073]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 21:49:48.239313143 (181h45m46.790783594s) [upload=160073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 21:49:53.369411932 (181h45m51.920882393s) [upload=160073]: Done uploading segments... Nov 16 21:49:53.369436949 (181h45m51.92090741s) [upload=160073]: Upload finished. err: <nil> Duration: 6.008088792s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.914211591s trace: https://jaeger.dev.storj.io/trace/1533f838bd5c565 FAIL (2025-11-16 22:29:57 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.914240195s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................X........................................] duration: 3.448213299s trace: https://jaeger.dev.storj.io/trace/251e9c48735b3b0a FAIL (2025-11-16 22:29:49 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.44823456s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 1.437122065s trace: https://jaeger.dev.storj.io/trace/ef0652f4a47c08 FAIL (2025-11-16 22:29:47 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.437148267s core uplink eu1

Orbiter Check History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)