News · 2026-08-03

NVIDIA's Open Full-Duplex Voice Model Wants an 80GB GPU

NVIDIA released NemotronLabs VoiceChat, an 11-billion-parameter model that performs streaming speech understanding and speech generation in a single network, listens and speaks simultaneously, and calls tools without breaking the conversation. NVIDIA calls it "the first open full-duplex model to support tool calling while maintaining a natural conversation flow during tool execution." Its own requirements list also says you need a GPU with at least 80 GB of memory.

Key facts

• The model needs "NVIDIA GPU with at least 80 GB of memory," per the repository prerequisites.
• It is 11B parameters: a fast Conformer speech encoder, a Nemotron Nano v2 9B language backbone, and a text-to-speech decoder, plus a separate output channel for tool calls.
• Weights are under the OpenMDW 1.1 license; the surrounding NeMo code is Apache 2.0.
• Primary source: NVIDIA's nemotron-labs-voicechat branch and the model checkpoint.

How it works

Most voice assistants are three programs in a trench coat: speech recognition transcribes you, a language model reads the transcript and writes a reply, text to speech reads the reply aloud. Each handoff costs time, and the whole chain has to wait for a turn detector to declare that you have stopped talking.

NVIDIA's model collapses that into one network. Audio goes into a fast Conformer encoder, producing audio tokens. Those tokens feed a Nemotron Nano v2 9B backbone that predicts text tokens. The text tokens feed a decoder that predicts audio codes for the reply. A separate output channel emits tool-calling scripts. Because it is one model handling both streams, it can produce speech while still receiving speech, which is what full duplex means in practice, and which our lessons on automatic speech recognition and neural text to speech cover as the separate pieces it replaces.

The tool-calling design is genuinely thoughtful. For each tool you can define an "on-hold" message the agent speaks the moment it decides to make the call, so the silence while an API responds is filled by the assistant saying something rather than by dead air.

The research direction is documented. NVIDIA's SALM-Duplex paper describes exactly this approach: fuse the continuous user stream and the agent's codec output so a single model handles simultaneous streams, using a pretrained streaming encoder to avoid a full speech-pretraining stage.

The limitations section is the story

NVIDIA's documentation is unusually candid, and it deserves to be read before anyone declares local voice solved. In the repository's own words, the model "can degrade into non-recoverable gibberish after several turns." It is "trained with no more than 2-min long audio context windows," beyond which conversational context "may not be retained reliably." It uses a single fixed voice and does not support cloning. It "may not yet be able to systematically handle user backchannelling" -- the mm-hmms that make duplex worth having. Spoken output "may end early," sessions "can get stuck in a word/sentence loop," and after finishing a turn the model "may continue speaking or start new agent turns without further user input."

On tools, NVIDIA recommends a maximum of five per session, says the model cannot reliably call multiple tools at once, and states plainly: "The user cannot interrupt the agent during tool calling execution." That last one is the sharpest limitation, because it means full duplex and interruptible tool use are not yet the same achievement. The conversational floor is still seized the moment real work starts. Finally, it is "not suitable for noisy or highly reverberant environments," which describes most kitchens and every car.

Why it matters

Open weights change who can build a voice agent and what it costs to run one, and they do not change the physics. An 80 GB accelerator is a rental, not a laptop, and a full-duplex session reserves inference capacity for the entire length of a call rather than for the length of a request. Add media transport, echo and jitter handling, regional placement, and state handoffs, and you have most of the systems work that OpenAI describes doing for GPT-Live. The model is the part that just got commoditized. The always-on session that never drops a frame is not.

The honest caveat: there is a documentation discrepancy worth tracking. NVIDIA's early-access page describes a 12B evaluation model, while this branch and the checkpoint are 11B. They may be different builds of the same effort, and NVIDIA has not published a note reconciling them. Every performance claim here, including the latency targets, is NVIDIA measuring NVIDIA's own model; no independent evaluation exists yet.

Primary source, verified: read the paper → (arXiv 2505.15670)

Ground Truth. (2026, August 3). NVIDIA's Open Full-Duplex Voice Model Wants an 80GB GPU. Ground Truth. https://groundtruth.day/news/nvidias-open-full-duplex-voice-model-wants-an-80gb-gpu.html

@misc{groundtruth:nvidias-open-full-duplex-voice-model-wants-an-80gb-gpu,
  title  = {NVIDIA's Open Full-Duplex Voice Model Wants an 80GB GPU},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {aug},
  url    = {https://groundtruth.day/news/nvidias-open-full-duplex-voice-model-wants-an-80gb-gpu.html}
}

Topics: nvidia · voice-ai · full-duplex · open-weights · speech · tool-use