Overview
NVIDIA released NemotronLabs VoiceChat 11B on August 9 via Hugging Face as a foundation model for builders of full-duplex voice agents, not for end consumers. It is the first open-source speech-to-speech model that can invoke external tools during conversation without giving up duplex behavior.
Why It Matters
Previous open-source full-duplex stacks fell into one of three buckets: listen-only, speak-only, or speak-and-listen without tool calling. VoiceChat 11B closes the third gap by emitting <TOOLCALL> blocks on a side channel while the agent delivers a preset "on-hold" utterance during the 2-second wait for an API response.
Architecture
A single unified network replaces the ASR + LLM + TTS cascade:
- Fast Conformer encoder from Nemotron-Speech-Streaming-En-0.6b, continuously encoding 16 kHz audio.
- Nemotron Nano v2 9B as the LLM backbone, consuming audio tokens and predicting text tokens.
- NVIDIA TTS decoder + codec producing 22.05 kHz audio.
- Independent tool-calling output channel.
- Smooth turn-taking: TOR 0.82 / 448 ms
- User-interruption: TOR 1.00 / 480 ms
- Pause-handling (lower is better): synthetic 0.153, Candor 0.255
- Simple: 58.5%
- Multiple: 62.5%
- Parallel: 42.5%
- Parallel-multiple: 27.5%
- Irrelevance: 89.6%
- Average: 56.1%
- Tool selection: 82.5%
- Argument accuracy: 44.2%
- Pass@1: 33%
- License: OpenMDW-1.1, research use only; commercial licensing is separate.
- No hosted API. NGC container and Hugging Face weights only.
- Hardware: single 80 GB GPU (A100 / H100 / H200 / B100 / B200 / RTX-6000), x86_64 Linux, vLLM runtime.
- Context: 2 minutes of audio max; quality can degrade into unrecoverable garble after several turns.
- Artifacts: occasional runaway self-talk after turn end; occasional dropped words in user STT.
- Tool limits: 5 tools per session max; no parallel tool calls; user cannot interrupt during tool execution (to preserve
<TOOLCALL>/<TOOL_RESPONSE>causality). - Prompts: system prompts and tool responses must be ASCII-only and TTS-friendly (no emoji, no full-width symbols, no Unicode quotes). The Fast Conformer is English-stream and training data is predominantly English, which limits Chinese-language support.
- Best fit: Voice AI R&D teams with 80 GB GPU capacity (in-car voice, contact-center CX, game NPCs, retail kiosks, accessibility).
- Poor fit: Consumer products needing immediate production deployment, businesses requiring deep Chinese support, and teams needing a hosted API for fast integration.
- Official model card: https://huggingface.co/nvidia/NVIDIA-NemotronLabs-VoiceChat-11B
- Official code (NeMo Speech branch): https://github.com/NVIDIA-NeMo/Speech/tree/nemotron-labs-voicechat
- Official NGC container: https://catalog.ngc.nvidia.com/orgs/nim/nvidia/containers/nemotron-labs-voicechat
- OpenMDW 1.1 license: https://github.com/OpenMDW/OpenMDW/blob/main/1.1/LICENSE.OpenMDW-1.1
- MarkTechPost technical write-up: https://www.marktechpost.com/2026/08/09/nvidia-releases-nemotronlabs-voicechat-11b-an-open-full-duplex-speech-to-speech-model-with-450-ms-turn-taking-and-live-tool-calling
- Full-Duplex-Bench 1.0 paper: https://arxiv.org/abs/2503.04721
- AU Harness BFCL-v3: https://github.com/ServiceNow/AU-Harness
- VoiceBench paper: https://arxiv.org/abs/2410.17196
Training data totals ~550,000 hours of audio (real + synthetic), building on SALM-Duplex and Audio Flamingo 3 research.
Tool Calling Design
Tool calls use the format <TOOLCALL>[{"name":..., "arguments":...}]</TOOLCALL>. Business code returns results via <TOOL_RESPONSE>[...]</TOOL_RESPONSE>. Each tool can declare an on-hold utterance spoken at invocation, eliminating awkward silence while async calls resolve.
Benchmarks
Full-Duplex-Bench 1.0
AU Harness BFCL-v3 (speech)
Full-Duplex-Bench v3 tool calling
NVIDIA self-reports 2nd place among open-source full-duplex models on VoiceBench and Full-Duplex-Bench 1.0.
Deployment Requirements & Limits
Key Takeaways
1. It is a base model, not a product. 448 ms latency, 80 GB VRAM, and a research-only license position it as a baseline for voice-agent engineering teams and a research stimulus for competitors such as Moshi, Audio Flamingo 3, and SALM-Duplex. 2. On-hold utterances are an underrated design pattern. Filling the silence during async tool calls addresses a usability bottleneck distinct from raw latency. The same Agent-era paradigm shift appears in Cosmos 3's first-class action tokens and SeedRealtime's internalized "when to speak" modeling. 3. The H2 2026 full-duplex landscape is taking shape. Open-source bases provide tool calling and Chinese teams must navigate two portability concerns: OpenMDW's commercial boundary and the ASCII-only prompt constraint for Chinese interaction.