Summary
A forum post introduces an arXiv paper (2604.21928) on evaluating automatic speech recognition (ASR) systems beyond word error rate (WER). WER is widely used but is semantically insensitive; embedding-based semantic metrics correlate better with human perception, yet decoder-based large language models (LLMs) remain underexplored for this task. The paper assesses LLM relevance through three approaches: (1) selecting the best hypothesis between two candidates, (2) computing semantic distance using generative embeddings, and (3) qualitative classification of errors. On the HATS dataset, the best LLM achieved 92-94% agreement with human annotators in hypothesis selection, compared to only 63% for WER, while also outperforming semantic metrics. Decoder-LLM embeddings performed on par with encoder-based models, suggesting LLMs offer a promising, interpretable, and semantic direction for ASR evaluation.
Paper Overview
Field: NLP
Authors: Thibault Bañeras-Roux, Shashi Kumar, Driss Khalil, Sergio Burdisso, Petr Motlicek, Shiran Liu, Mickael Rouvier, Jane Wottawa, Richard Dufour
Published: 2026-04-23
arXiv: 2604.21928
Abstract (translated summary)
Automatic speech recognition (ASR) is traditionally evaluated using word error rate (WER), but this metric is semantically insensitive. Embedding-based semantic metrics correlate better with human perception, yet decoder-based large language models (LLMs) remain underexplored for this task. This paper evaluates their relevance through three approaches:
1. Selecting the best hypothesis between two candidate hypotheses
2. Computing semantic distance using generative embeddings
3. Qualitative classification of errors
On the HATS dataset, the best LLM reached 92-94% agreement with human annotators in hypothesis selection, while WER achieved only 63%, and the LLM also outperformed semantic metrics. Embeddings from decoder-based LLMs showed performance comparable to encoder models. Finally, LLMs represent a promising direction for interpretable and semantic ASR evaluation.
---
*Auto-collected on 2026-04-27*
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/177618793