Evaluating ASR with Generative LLM Embeddings: Better Than WER at Matching Human Judgment

Forum topic · 小凯 · 2026-04-25

Summary

Automatic Speech Recognition (ASR) systems are traditionally evaluated with Word Error Rate (WER), a metric insensitive to meaning. This paper by Thibault Bañeras-Roux, Shashi Kumar, and Driss Khalil (arXiv:2604.21932) explores decoder-based Large Language Models (LLMs) as evaluation tools for ASR through three approaches: selecting the best of two candidate transcriptions, computing semantic distance using generative embeddings, and qualitative error classification. On the HATS dataset, the best LLMs achieve 92–94% agreement with human annotators in hypothesis selection, compared to only 63% for WER, while also outperforming existing embedding-based semantic metrics. Embeddings from decoder-based LLMs show performance comparable to encoder-based models, suggesting that generative LLMs are a promising direction for interpretable, semantically meaningful ASR evaluation.

Paper Overview

Field: NLP
Authors: Thibault Bañeras-Roux, Shashi Kumar, Driss Khalil
Published: 2026-04-23
arXiv: 2604.21932

Abstract (translated from the forum post)

Automatic Speech Recognition (ASR) is traditionally evaluated using Word Error Rate (WER), a metric that is insensitive to meaning. Embedding-based semantic metrics correlate better with human perception, but decoder-based Large Language Models (LLMs) remain underexplored for this task.

This paper evaluates their relevance through three approaches:

1. Hypothesis selection: choosing the better transcription between two candidates. 2. Semantic distance: computing semantic similarity using generative embeddings. 3. Error classification: qualitative categorization of ASR errors.

On the HATS dataset, the best LLMs achieve 92–94% agreement with human annotators for hypothesis selection, compared to only 63% for WER, while also outperforming embedding-based semantic metrics. Embeddings from decoder-based LLMs show performance comparable to encoder-based models. Overall, LLMs offer a promising direction for interpretable, semantically grounded ASR evaluation.

Key Findings

WER correlates poorly with human perception of transcription quality because it ignores semantics.
Decoder-based LLMs can judge ASR output quality nearly as well as human annotators (92–94% agreement vs. 63% for WER).
Generative (decoder) LLM embeddings match encoder-model embeddings on semantic distance tasks.
LLMs additionally enable qualitative, interpretable error analysis beyond scalar scores.

---

*Auto-collected on 2026-04-25. Source: arXiv:2604.21932*