DuplexSpeechBench-IFEval: Evaluating Implicit Instruction Following in Full-Duplex Voice Agents

Forum topic · 小凯 · 2026-09-06

Summary

DuplexSpeechBench-IFEval (DSB-IFEval), introduced by Puneet Mathur and Dinesh Manocha (arXiv:2509.00008), is a benchmark for evaluating implicit instruction following in full-duplex, real-time voice agents. Unlike prior benchmarks that test turn management through explicit instructions, DSB-IFEval reflects how deployed agents are actually configured: via roles and personas from which conversational behavior must be inferred. The benchmark contains 1,038 test cases spanning eight assistant roles and five conditioning protocols: default behavior, explicit behavioral instructions, persona-implied behavior, combined persona-rule conditioning, and instruction conflict. It uses a deterministic Instruction Adherence Score (IAS) for floor management and an LLM-judged Persona Adherence Score (PAS) for persona-consistent content. Across six real-time speech systems, full-duplex models like F-Actor and PersonaPlex showed 9.7% and 4.5% adherence drops when behavior had to be inferred from persona alone, while GPT-Realtime, MiniCPM-o, and Fun-Audio-Chat followed persona-consistent content strongly but failed to adapt turn-taking behavior, remaining limited on proactive actions. Systems also struggled to override instructions under safety conflicts.

Paper Overview

Field: AI/ML Authors: Puneet Mathur, Dinesh Manocha arXiv: 2509.00008

Abstract

Full-duplex voice agents must continuously decide when to listen, backchannel, interrupt, handle speech overlaps, take the floor, and yield. Existing benchmarks largely test these behaviors through explicit turn-management instructions, while deployed agents are often configured through roles or personas from which the appropriate conversational behavior must be inferred. The authors introduce DuplexSpeechBench-IFEval (DSB-IFEval) for evaluating implicit instruction-following in real-time spoken interaction.

Benchmark Design

1,038 test cases spanning eight diverse assistant roles
Five conditioning protocols for instruction-following:
Deterministic Instruction Adherence Score (IAS) for real-time floor management
LLM-judged Persona Adherence Score (PAS) for persona-consistent content

Key Findings

Evaluations across six real-time speech systems reveal architecture-dependent trade-offs.
Full-duplex models such as F-Actor and PersonaPlex are sensitive to whether conversational behavior is explicitly stated or must be inferred from the persona, with adherence dropping by 9.7% and 4.5% respectively under persona-only conditioning.
In contrast, GPT-Realtime, MiniCPM-o, and Fun-Audio-Chat strongly follow persona-consistent content, but their turn-taking behavior does not adapt between explicit and persona-only instructions, and they remain limited on several proactive actions.
Even when systems reliably follow instructions that conflict with their assigned persona, they struggle to override these instructions under safety conflicts.

Conclusion

Inferring role-implied behavior, executing it at the appropriate conversational moment, and resolving competing instructions remain distinct, unsolved challenges for full-duplex voice agents.