Learn · Intermediate

Full-Duplex Speech Models: Listening and Talking at the Same Time

A full-duplex speech model takes in audio and produces audio at the same time, in the same network, continuously. That single property removes the component every previous voice assistant depended on -- the turn detector that guesses when you have stopped speaking -- and it is why 2026's voice systems can hum agreement halfway through your sentence, notice you trailing off, and stop talking when you cut in without the whole exchange resetting.

The word comes from telephony. A half-duplex line, like a walkie-talkie, carries one direction at a time and needs a convention about who is transmitting. A full-duplex line carries both at once, which is why a phone call sounds like a conversation and a radio call sounds like a procedure.

What came before

The standard voice assistant is three models in a chain. Speech recognition turns your audio into text. A language model reads the text and writes a reply. Text to speech reads the reply out loud. Our lessons on automatic speech recognition and neural text to speech cover the ends of that chain.

The chain has a scheduling problem before it has a speed problem. Nothing downstream can start until something declares that you are done talking, so a small model called a turn detector sits in front and guesses, usually from silence duration and prosody. OpenAI described the bind precisely in its GPT-Live engineering write-up: "guess too soon, and the user gets cut off; guess too late, and the response feels sluggish. Only after the detector made its decision could the much larger LLM get to work."

You cannot tune your way out of that. Humans pause mid-sentence to think, and humans finish sentences with the same falling pitch they use when pausing. A detector that never cuts anyone off will always feel slow, and a detector that never feels slow will cut people off.

How full duplex works

The architectural move is to make the model's input a continuous stream rather than a completed utterance, and its output a continuous stream rather than a completed reply. Concretely:

Audio becomes tokens. A neural codec compresses waveforms into a sequence of discrete codes at a low rate -- a few dozen per second rather than sixteen thousand samples. Neil Zeghidour and colleagues' SoundStream established the design, and AudioLM from Google showed you could then run ordinary sequence modelling directly on those codes. This connects to vector quantization and discrete latent codes.

Two streams, one model. Rather than one sequence, the model handles the user's channel and its own channel together, predicting the next chunk of its own audio conditioned on everything heard so far, including whatever the user is saying right now. Tu Anh Nguyen and colleagues' Generative Spoken Dialogue Language Modeling modelled two-channel conversation with overlap and backchannels this way, and Ziyang Ma and colleagues' Language Model Can Listen While Speaking built listening into the generation loop so the model could stop itself.

Silence is a legal output. This is the part that makes the whole thing coherent. If the model emits audio tokens every timestep, and "quiet" is one of the things it can emit, then the decision to speak is not a separate control system. It is just what the model predicted. Turn-taking stops being infrastructure and becomes behaviour.

Moshi, from Kyutai, put the pieces together in a released system, running a text stream alongside the audio stream so the model keeps its language ability while operating on sound. NVIDIA's SALM-Duplex took the efficiency route, fusing the user's continuous input with the agent's codec output and reusing a pretrained streaming encoder to avoid a full speech-pretraining run from scratch.

The analogy worth holding is a jazz duo versus two people reading a script to each other. The script version needs a rule for who goes next. The duo does not, because both are listening and playing continuously, and the decision about when to come in is part of playing.

What gets harder

Full duplex does not make voice easy; it moves the difficulty.

Knowing when not to talk. A model that can speak at any moment will. Complaints about GPT-Live centred on backchannels and interruptions landing in the wrong places, which is a new failure mode that half-duplex systems structurally could not have.

Knowing whether you are being addressed. In a kitchen with three people, the hard question is not whether the model can listen while speaking. It is whether the speech it hears is aimed at it. Push-to-talk answers that with a button, and several builders have kept it for exactly that reason.

Everything becomes real-time. Once audio must flow continuously, a late frame is an audible artifact rather than a slow request, and the system inherits the whole apparatus of live media: jitter buffers, echo cancellation, regional placement, and state handoffs that stay inaudible. Our reporting on how OpenAI rebuilt voice as a systems problem is mostly about that, not about the model.

Cost changes shape. A duplex session occupies inference capacity for the length of a conversation rather than the length of a request, so capacity is measured in concurrent sessions. And the models are not small: NVIDIA's open 11-billion-parameter duplex model asks for a GPU with 80 GB of memory.

The takeaway

Full duplex is the shift from a voice interface that takes turns to one that holds a line open. It deletes the turn detector by making speaking-or-not a prediction the model makes continuously, and in exchange it inherits every hard problem of running a live phone call. The model was the easy part.

Ground Truth. (2026, August 3). Full-Duplex Speech Models: Listening and Talking at the Same Time. Ground Truth. https://groundtruth.day/learn/full-duplex-speech-models.html

@misc{groundtruth:full-duplex-speech-models,
  title  = {Full-Duplex Speech Models: Listening and Talking at the Same Time},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {aug},
  url    = {https://groundtruth.day/learn/full-duplex-speech-models.html}
}

Topics: speech · full-duplex · voice-ai · audio · fundamentals · architecture