Work

Multi-criteria ranking of open automatic speech recognition models under alternative deployment priorities

Year: 2026
Type: article
Abstract: Automatic speech recognition (ASR) models differ in accuracy, throughput, and parameter count, so choosing a model involves trade-offs. This study ranks 69 license-eligible open ASR model configurations from the July 2026 open ASR leaderboard, which covers eight short-form English test sets. Criterion weights were derived with the fuzzy analytic hierarchy process (fuzzy AHP) under three illustrative scenarios, each prioritizing one criterion: word error rate (WER), inverse real-time factor (RTFx), or parameter count. The models were then ranked with the technique for order preference by similarity to ideal solution (TOPSIS). NVIDIA Parakeet-TDT-CTC-110M ranked first in the WER-prioritized scenario, and NVIDIA FastConformer-CTC-large ranked first in the other two. The sensitivity analysis tested three further ranking methods (VIKOR, MABAC, and ARAS), classical AHP and entropy weights, stronger dominance ratings, min–max normalization, 171 weight combinations, and alternative WER measures. Across the four ranking methods and three scenarios, the top rank was always held by one of three NVIDIA models: Parakeet-TDT-CTC-110M, FastConformer-CTC-Large, or Conformer-CTC-Small. Classical AHP weights produced nearly identical rankings (Spearman rank correlation of at least 0.998), and Parakeet-TDT-CTC-110M or FastConformer-CTC-Large ranked first in 163 of the 171 TOPSIS weight combinations. However, under the strongest WER priority tested, the more accurate but larger NVIDIA Parakeet-TDT-0.6B-v2 ranked first in the WER-prioritized scenario, and ARAS favored the smallest models. The rankings are specific to this benchmark rather than validated recommendations for particular sectors, and any model’s suitability must be verified on representative audio and target hardware before deployment. The framework offers a transparent, reproducible way to compare ASR models under different deployment priorities.
Language: English
Open Access status: gold
Estimated APC paid: $2,690
OpenAlex ID: W7225915378
FWCI:
Cites: 34
Cited by:
Related to: 10