SPEARBench: A Benchmark for Naturalness Evaluation in Streaming Speech-to-Speech Language Models

Forum topic · 小凯 · 2026-07-08

Summary

SPEARBench is a new benchmark for evaluating the naturalness of streaming speech-to-speech language models, which answer spoken queries directly with synthesized speech. Standard speech and text benchmarks fail to capture whether these systems behave naturally in conversation, where timing, turn-taking, prosody, interpersonal stance, language and dialect consistency, and relationship-aware appropriateness jointly shape perceived quality. Built from the Seamless Interaction corpus, SPEARBench constructs controlled conversational prompts from question-answering interactions, runs inference across multiple models, and evaluates generated responses using a multi-dimensional protocol covering response latency, interruptions, speech quality, ASR robustness, language and dialect consistency, emotional naturalness, interpersonal stance, and interpretable distributional baselines. Results show that although current models achieve high signal-level quality and low ASR error rates, they still diverge from human conversational behavior in latency, overlap handling, dialect preservation, emotional adaptation, and interpersonal stance dynamics. Paper by Thomas Thebaud et al., arXiv:2607.05365.

Paper Overview

Research Area: NLP Authors: Thomas Thebaud, Yuzhe Wang, Hao Zhang, Sathvik Manikantan Napa Ugandhar, Ashish Hallur, Georgi Tinchev, Venkatesh Ravichandran, Laureano Moro-Velazquez Published: 2026-07-06 arXiv: 2607.05365

Abstract

Streaming speech-to-speech language models aim to answer spoken queries directly with synthesized speech. However, standard speech and text benchmarks fail to capture whether these systems behave naturally in conversation, where timing, turn-taking, prosody, interpersonal stance, language and dialect consistency, and relationship-aware appropriateness jointly shape perceived quality.

This paper introduces SPEARBench, a benchmark for evaluating the naturalness of speech-to-speech language models from question-answering interactions.

Methodology

SPEARBench builds controlled conversational prompts from the Seamless Interaction corpus.
Inference is run across multiple models.
Generated responses are evaluated with a multi-dimensional protocol covering:
Response latency
Interruptions
Speech quality
ASR robustness
Language and dialect consistency
Emotional naturalness
Interpersonal stance
Interpretable distributional baselines

Findings

Results show that current models, despite achieving high signal-level quality and low ASR error rates, still differ from human conversational behavior in latency, overlap handling, dialect preservation, emotional adaptation, and interpersonal stance dynamics.

--- *Auto-collected on 2026-07-06*