SPEARBench: A Benchmark for Evaluating Naturalness in Streaming Speech-to-Speech Language Models

Forum topic · 小凯 · 2026-07-08

Summary

SPEARBench is a new benchmark proposed to evaluate the naturalness of streaming speech-to-speech language models that answer spoken queries directly with synthesized speech. Standard speech and text benchmarks fail to capture whether such systems sound natural in real conversations, where timing, turn-taking, prosody, interpersonal stance, language and dialect consistency, and relationship-aware appropriateness jointly shape perceived quality. Built from controlled conversational prompts derived from the Seamless Interaction corpus, SPEARBench runs inference across multiple models and evaluates generated responses with a multi-dimensional protocol covering response latency, interruptions, speech quality, ASR robustness, language and dialect consistency, emotional naturalness, interpersonal stance, and interpretable distribution baselines. Results show that although current models achieve high signal-level quality and low ASR error, they still differ from human conversational behavior in latency, overlap handling, dialect preservation, emotional adaptation, and interpersonal stance dynamics. The paper is authored by Thomas Thebaud, Yuzhe Wang, Hao Zhang, and colleagues from Amazon, and is available on arXiv as 2607.05365.

Paper Overview

Research Area: NLP Authors: Thomas Thebaud, Yuzhe Wang, Hao Zhang, Sathvik Manikantan Napa Ugandhar, Ashish Hallur, Georgi Tinchev, Venkatesh Ravichandran, Laureano Moro-Velazquez Published: 2026-07-06 arXiv: 2607.05365

Abstract

Streaming speech-to-speech language models aim to answer spoken queries directly with synthesized speech. However, standard speech and text benchmarks fail to capture whether these systems sound natural in conversation, where timing, turn-taking, prosody, interpersonal stance, language and dialect consistency, and relationship-aware appropriateness jointly shape perceived quality.

This paper introduces SPEARBench, a benchmark for evaluating the naturalness of speech-to-speech language models from question-answering interactions. SPEARBench constructs controlled conversational prompts from the Seamless Interaction corpus, runs inference across multiple models, and evaluates the generated responses using a multi-dimensional protocol.

Evaluation Dimensions

The evaluation protocol covers:

Response latency
Interruptions / barge-in behavior
Speech quality
ASR robustness
Language and dialect consistency
Emotional naturalness
Interpersonal stance
Interpretable distribution baselines

Key Findings

Results show that current models can achieve high signal-level quality and low ASR error, yet they still differ from human conversational behavior in terms of latency, overlap handling, dialect preservation, emotional adaptation, and interpersonal stance dynamics.

--- *Auto-collected on 2026-07-06*