DuplexSpeechBench-IFEval: A Benchmark for Implicit Instruction Following in Full-Duplex Voice Agents

Forum topic · 小凯 · 2026-09-06

Summary

DuplexSpeechBench-IFEval (DSB-IFEval) is a new benchmark from researchers Puneet Mathur and Dinesh Manocha for evaluating implicit instruction following in real-time full-duplex spoken AI agents. Unlike existing benchmarks that rely on explicit turn-management instructions, DSB-IFEval tests whether agents can infer appropriate conversational behavior from personas. The benchmark contains 1,038 test cases across eight assistant roles and five conditioning protocols: default behavior, explicit behavioral instructions, persona-implied behavior, combined persona-rule conditioning, and instruction conflict. Evaluation uses a deterministic Instruction Adherence Score (IAS) for turn-taking management and an LLM-judged Persona Adherence Score (PAS) for persona-consistent content. Testing six real-time voice systems revealed architecture-dependent tradeoffs: full-duplex models like F-Actor and PersonaPlex are sensitive to whether behavior is stated explicitly or inferred, dropping 9.7% and 4.5% respectively in persona-only conditions, while GPT-Realtime, MiniCPM-o, and Fun-Audio-Chat strongly follow persona-consistent content but show inflexible turn-taking behavior. arXiv: 2509.00008.

Paper Overview

Field: AI/ML Authors: Puneet Mathur, Dinesh Manocha Published: 2026-09-06 arXiv: 2509.00008

Abstract (translated)

Full-duplex voice agents must continuously decide when to listen, backchannel, interrupt, handle speech overlaps, take the floor, and yield. Existing benchmarks largely test these behaviors through explicit turn-management instructions, while deployed agents are often configured through roles or personas from which the appropriate conversational behavior must be inferred. The authors introduce DuplexSpeechBench-IFEval (DSB-IFEval) for evaluating implicit instruction-following in real-time spoken interaction.

DSB-IFEval comprises 1,038 test cases spanning eight diverse assistant roles and evaluates five conditioning protocols for instruction-following: default behavior, explicit behavioral instructions, persona-implied behavior, combined persona-rule conditioning, and instruction conflict. The benchmark uses a deterministic Instruction Adherence Score (IAS) to measure real-time floor management, and an LLM-judged Persona Adherence Score (PAS) for persona-consistent content.

Key Findings

Across six real-time voice systems, the evaluation revealed architecture-dependent tradeoffs:

Full-duplex models (F-Actor, PersonaPlex) are more sensitive to whether conversational behaviors are explicitly stated or must be inferred from personas, with adherence drops of 9.7% and 4.5% respectively in persona-only conditions.
GPT-Realtime, MiniCPM-o, and Fun-Audio-Chat strongly follow persona-consistent content, but their floor-taking behavior does not adapt between explicit and persona-only instructions, and they remain restricted on several proactive actions.
Even when systems reliably follow instructions that conflict with their assigned persona, they struggle to override those instructions under safety conflicts.

Conclusion

Inferring persona-implied behavior, executing it at appropriate conversational moments, and resolving competing instructions remain distinct challenges for full-duplex voice agents.

---

*Auto-collected on 2026-09-06*