Summary
Automatic speech recognition (ASR) systems are traditionally evaluated with word error rate (WER), a metric that is insensitive to semantics. Embedding-based semantic metrics correlate better with human perception, but decoder-based large language models (LLMs) remain underexplored for this task. A paper by Thibault Bañeras-Roux and colleagues (arXiv:2604.21928) assesses LLM relevance to ASR evaluation through three approaches: selecting the best hypothesis between two candidates, computing semantic distance using generative embeddings, and qualitative classification of errors. On the HATS dataset, the best LLM achieved 92–94% agreement with human annotators in hypothesis selection, compared with only 63% for WER, while also outperforming semantic embedding metrics. Embeddings from decoder-based LLMs performed on par with encoder-based models. The authors conclude that decoder LLMs offer a promising direction for interpretable, semantics-aware ASR evaluation.
Paper Overview
- Field: NLP
- Authors: Thibault Bañeras-Roux, Shashi Kumar, Driss Khalil, Sergio Burdisso, Petr Motlicek, Shiran Liu, Mickael Rouvier, Jane Wottawa, Richard Dufour
- Published: 2026-04-23
- arXiv: 2604.21928
Summary
Automatic speech recognition (ASR) has traditionally been evaluated using word error rate (WER), but this metric is insensitive to semantics. Embedding-based semantic metrics correlate better with human perception, yet decoder-based large language models (LLMs) remain underexplored for this task.
This paper evaluates their relevance through three approaches:
1. Hypothesis selection — choosing the best hypothesis between two candidates.
2. Semantic distance — computing semantic distances using generative embeddings.
3. Error analysis — qualitative classification of recognition errors.
Key Findings
- On the HATS dataset, the best LLM achieved 92–94% agreement with human annotators in hypothesis selection, while WER reached only 63%.
- The LLM-based approach also outperformed existing semantic embedding metrics.
- Embeddings from decoder-based LLMs performed on par with encoder-based models.
- The authors conclude that decoder LLMs provide a promising direction for interpretable, semantics-aware ASR evaluation.
---
*Auto-collected on 2026-04-27*
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/177618793