On August 9, NVIDIA published NemotronLabs VoiceChat 11B on Hugging Face. Its target audience is not end consumers—it is a base model for teams building voice agents. Among open-source full-duplex solutions, prior options have generally fallen into three categories: can listen but not speak, can speak but not listen, or can do both but cannot call tools. VoiceChat 11B fills the third gap: it can call external tools mid-conversation, emit <TOOLCALL> blocks over a bypass channel, and deliver preconfigured "on-hold" phrases while the agent waits seconds for tool results, so the conversation never goes silent.
How it reaches 448 ms latency
By removing the cascaded stack's ASR, LLM, and TTS components entirely, replaced with one unified network:
- Fast Conformer encoder (from Nemotron-Speech-Streaming-En-0.6b, continuously encoding a 16 kHz audio stream)
- Nemotron Nano v2 9B LLM backbone (consumes audio tokens, predicts text tokens)
- NVIDIA's own TTS decoder and codec (outputs 22.05 kHz audio)
- Full-Duplex-Bench 1.0: smooth turn-taking TOR 0.82 / 448 ms; user-interruption TOR 1.00 / 480 ms; pause-handling TOR (lower is better) 0.153 synthetic / 0.255 Candor
- AU Harness BFCL-v3 speech: simple 58.5%, multiple 62.5%, parallel 42.5%, parallel-multiple 27.5%, irrelevance 89.6%, average 56.1%
- Full-Duplex-Bench v3 tool calling: tool selection 82.5%, argument accuracy 44.2%, pass@1 33%
- NVIDIA self-reports: rank 2 among open full-duplex models on VoiceBench and rank 2 on Full-Duplex-Bench 1.0 open category
- License: OpenMDW-1.1, officially stated as "research purposes only"; commercial licensing must be negotiated separately. NVIDIA provides no hosted API—just an NGC container and HF weights.
- Hardware: single 80 GB GPU (A100 / H100 / H200 / B100 / B200 / RTX-6000), x86_64 Linux, vLLM runtime—teams without GPU resources cannot touch it today.
- Context: 2-minute audio limit; degrades into unrecoverable gibberish after several turns; occasional runaway self-talk after turn ends; occasional dropped words in user speech transcription.
- Tool calling limits: hard cap of 5 tools per session; no parallel calls; user cannot interrupt during tool execution (to preserve
<TOOLCALL>/<TOOL_RESPONSE>causal closure). - ASCII-only, TTS-friendly system prompts and tool responses: prompts must avoid emoji, full-width symbols, and Unicode quotes—a clear weakness for Chinese scenarios, since the Fast Conformer is an English streaming model trained predominantly on English data.
- Fits: voice AI teams with 80 GB GPU R&D capacity (in-car voice, customer service CX, game NPCs, retail kiosks, accessibility)
- Not for: consumer products needing immediate production, businesses needing deep Chinese support, or fast integrators wanting a hosted API
- Official model card (Hugging Face): https://huggingface.co/nvidia/NVIDIA-NemotronLabs-VoiceChat-11B
- Official code repo (NeMo Speech branch): https://github.com/NVIDIA-NeMo/Speech/tree/nemotron-labs-voicechat
- Official NGC container: https://catalog.ngc.nvidia.com/orgs/nim/nvidia/containers/nemotron-labs-voicechat
- OpenMDW 1.1 license: https://github.com/OpenMDW/OpenMDW/blob/main/1.1/LICENSE.OpenMDW-1.1
- MarkTechPost technical analysis: https://www.marktechpost.com/2026/08/09/nvidia-releases-nemotronlabs-voicechat-11b-an-open-full-duplex-speech-to-speech-model-with-450-ms-turn-taking-and-live-tool-calling
- Full-Duplex-Bench 1.0 paper: https://arxiv.org/abs/2503.04721
- AU Harness BFCL-v3: https://github.com/ServiceNow/AU-Harness
- VoiceBench paper: https://arxiv.org/abs/2410.17196
There is no token conversion or API handshake in between; measured end-to-end smooth turn-taking latency is 448 ms.
Key engineering details
First open full-duplex model with tool calling
Tool calls are output on a dedicated channel in <TOOLCALL>[{"name":..., "arguments":...}]</TOOLCALL> format, with results returned via <TOOL_RESPONSE>[...]</TOOL_RESPONSE>. The key design is "on-hold phrases": each tool can have a preset line spoken at call time, so users do not think the call dropped during a 2-second API wait. This fills conversational gaps while awaiting async results—a real step forward in dialogue system usability.
Architecture
Four components: Fast Conformer encoder + Nemotron Nano v2 backbone + TTS decoder + an independent tool-call output channel. Training data is roughly 550,000 hours of audio (real + synthetic), building on SALM-Duplex and Audio Flamingo 3 research.
Benchmarks
Deployment requirements and limitations
Three takeaways
1. This is a base model, not a product. With 448 ms latency, 80 GB single-GPU requirements, and a research-only license, it is a reference and baseline for engineering teams building voice agents—not for consumers. It pushes the open-source full-duplex line from "demoable" to "evaluable," a direct stimulus for researchers working on Moshi, Audio Flamingo 3, and SALM-Duplex.
2. On-hold phrases are an underrated design point. Traditional cascaded stacks can only "freeze and wait" during tool calls, breaking the conversation. VoiceChat 11B fills that gap with preset speech—it acknowledges that the usability bottleneck for voice agents is not latency itself, but the moment users think the other side hung up. This parallels a broader agent-era paradigm shift in speech and multimodal systems: beyond latency, agents need explicit rules for in-conversation behavior.
3. The full-duplex landscape for late 2026 is taking shape. On the open-source side, NVIDIA has filled the tool-calling gap via NemotronLabs; commercially, GPT-4o Realtime, Gemini Live, and Realtime API push hosted offerings. Open-source base models with tool calling and commercial hosted reliability will coexist. For teams in China, NVIDIA has exposed two portable pain points: the boundaries of OpenMDW commercial licensing, and the ASCII-only system prompt limitation for Chinese interactions.