English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

NVIDIA NemotronLabs VoiceChat 11B: The First Open-Source Full-Duplex Speech Agent Base Model with Tool Calling

Forum topic · 小凯 · 2026-08-10

Summary

NVIDIA has released NemotronLabs VoiceChat 11B on Hugging Face, an open full-duplex speech-to-speech model positioned as a foundation for voice agent developers rather than end users. It replaces the traditional cascaded ASR + LLM + TTS stack with a single unified network: a Fast Conformer streaming encoder (16 kHz), a Nemotron Nano v2 9B LLM backbone, and NVIDIA's own TTS decoder, achieving 448 ms smooth turn-taking latency. It is the first open full-duplex model to support tool calling, emitting <TOOLCALL> blocks on a dedicated output channel, with preconfigured on-hold phrases that fill conversational gaps while awaiting API results. Benchmarks include Full-Duplex-Bench 1.0 turn-taking TOR 0.82 and VoiceBench rank 2 among open full-duplex models. Key limitations: OpenMDW-1.1 research-only license, 80 GB single-GPU requirement, 2-minute audio context, max 5 tools per session, ASCII-only system prompts, and English-centric training that limits Chinese use cases. The model suits engineering teams building voice agents with GPU resources, not production consumer products.

On August 9, NVIDIA published NemotronLabs VoiceChat 11B on Hugging Face. Its target audience is not end consumers—it is a base model for teams building voice agents. Among open-source full-duplex solutions, prior options have generally fallen into three categories: can listen but not speak, can speak but not listen, or can do both but cannot call tools. VoiceChat 11B fills the third gap: it can call external tools mid-conversation, emit <TOOLCALL> blocks over a bypass channel, and deliver preconfigured "on-hold" phrases while the agent waits seconds for tool results, so the conversation never goes silent.

How it reaches 448 ms latency

By removing the cascaded stack's ASR, LLM, and TTS components entirely, replaced with one unified network:

  • Fast Conformer encoder (from Nemotron-Speech-Streaming-En-0.6b, continuously encoding a 16 kHz audio stream)
  • Nemotron Nano v2 9B LLM backbone (consumes audio tokens, predicts text tokens)
  • NVIDIA's own TTS decoder and codec (outputs 22.05 kHz audio)
  • There is no token conversion or API handshake in between; measured end-to-end smooth turn-taking latency is 448 ms.

    Key engineering details

    First open full-duplex model with tool calling

    Tool calls are output on a dedicated channel in <TOOLCALL>[{"name":..., "arguments":...}]</TOOLCALL> format, with results returned via <TOOL_RESPONSE>[...]</TOOL_RESPONSE>. The key design is "on-hold phrases": each tool can have a preset line spoken at call time, so users do not think the call dropped during a 2-second API wait. This fills conversational gaps while awaiting async results—a real step forward in dialogue system usability.

    Architecture

    Four components: Fast Conformer encoder + Nemotron Nano v2 backbone + TTS decoder + an independent tool-call output channel. Training data is roughly 550,000 hours of audio (real + synthetic), building on SALM-Duplex and Audio Flamingo 3 research.

    Benchmarks

  • Full-Duplex-Bench 1.0: smooth turn-taking TOR 0.82 / 448 ms; user-interruption TOR 1.00 / 480 ms; pause-handling TOR (lower is better) 0.153 synthetic / 0.255 Candor
  • AU Harness BFCL-v3 speech: simple 58.5%, multiple 62.5%, parallel 42.5%, parallel-multiple 27.5%, irrelevance 89.6%, average 56.1%
  • Full-Duplex-Bench v3 tool calling: tool selection 82.5%, argument accuracy 44.2%, pass@1 33%
  • NVIDIA self-reports: rank 2 among open full-duplex models on VoiceBench and rank 2 on Full-Duplex-Bench 1.0 open category
  • Deployment requirements and limitations

  • License: OpenMDW-1.1, officially stated as "research purposes only"; commercial licensing must be negotiated separately. NVIDIA provides no hosted API—just an NGC container and HF weights.
  • Hardware: single 80 GB GPU (A100 / H100 / H200 / B100 / B200 / RTX-6000), x86_64 Linux, vLLM runtime—teams without GPU resources cannot touch it today.
  • Context: 2-minute audio limit; degrades into unrecoverable gibberish after several turns; occasional runaway self-talk after turn ends; occasional dropped words in user speech transcription.
  • Tool calling limits: hard cap of 5 tools per session; no parallel calls; user cannot interrupt during tool execution (to preserve <TOOLCALL> / <TOOL_RESPONSE> causal closure).
  • ASCII-only, TTS-friendly system prompts and tool responses: prompts must avoid emoji, full-width symbols, and Unicode quotes—a clear weakness for Chinese scenarios, since the Fast Conformer is an English streaming model trained predominantly on English data.
  • Three takeaways

    1. This is a base model, not a product. With 448 ms latency, 80 GB single-GPU requirements, and a research-only license, it is a reference and baseline for engineering teams building voice agents—not for consumers. It pushes the open-source full-duplex line from "demoable" to "evaluable," a direct stimulus for researchers working on Moshi, Audio Flamingo 3, and SALM-Duplex.

    2. On-hold phrases are an underrated design point. Traditional cascaded stacks can only "freeze and wait" during tool calls, breaking the conversation. VoiceChat 11B fills that gap with preset speech—it acknowledges that the usability bottleneck for voice agents is not latency itself, but the moment users think the other side hung up. This parallels a broader agent-era paradigm shift in speech and multimodal systems: beyond latency, agents need explicit rules for in-conversation behavior.

    3. The full-duplex landscape for late 2026 is taking shape. On the open-source side, NVIDIA has filled the tool-calling gap via NemotronLabs; commercially, GPT-4o Realtime, Gemini Live, and Realtime API push hosted offerings. Open-source base models with tool calling and commercial hosted reliability will coexist. For teams in China, NVIDIA has exposed two portable pain points: the boundaries of OpenMDW commercial licensing, and the ASCII-only system prompt limitation for Chinese interactions.

    Who it fits

  • Fits: voice AI teams with 80 GB GPU R&D capacity (in-car voice, customer service CX, game NPCs, retail kiosks, accessibility)
  • Not for: consumer products needing immediate production, businesses needing deep Chinese support, or fast integrators wanting a hosted API
  • Sources

  • Official model card (Hugging Face): https://huggingface.co/nvidia/NVIDIA-NemotronLabs-VoiceChat-11B
  • Official code repo (NeMo Speech branch): https://github.com/NVIDIA-NeMo/Speech/tree/nemotron-labs-voicechat
  • Official NGC container: https://catalog.ngc.nvidia.com/orgs/nim/nvidia/containers/nemotron-labs-voicechat
  • OpenMDW 1.1 license: https://github.com/OpenMDW/OpenMDW/blob/main/1.1/LICENSE.OpenMDW-1.1
  • MarkTechPost technical analysis: https://www.marktechpost.com/2026/08/09/nvidia-releases-nemotronlabs-voicechat-11b-an-open-full-duplex-speech-to-speech-model-with-450-ms-turn-taking-and-live-tool-calling
  • Full-Duplex-Bench 1.0 paper: https://arxiv.org/abs/2503.04721
  • AU Harness BFCL-v3: https://github.com/ServiceNow/AU-Harness
  • VoiceBench paper: https://arxiv.org/abs/2410.17196

Tags

#nvidia#voice-agents#full-duplex#speech-to-speech#open-source-models#tool-calling#nemotron#tts

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178630988