Learn · Beginner

Neural text-to-speech: how a model turns writing into a voice

Neural text-to-speech turns written text into audio using learned models rather than recorded fragments, and it works in three stages: figure out which sounds the text corresponds to, decide how long each sound lasts, and generate the actual waveform. That last stage is the hard one. A second of speech is 24,000 numbers, and producing them so they sound like a human rather than a fax machine is where most of a speech model's size and difficulty lives.

The problem: text is short, audio is enormous

Take the sentence "the meeting is at three." That is 24 characters. Spoken at a normal pace it lasts about a second and a half, which at CD-adjacent quality is roughly 36,000 individual numbers describing air pressure over time. The model has to invent all of them, in an order where a single misplaced sample sounds like a click and a slightly wrong pitch contour sounds like a robot.

It also has to solve an alignment problem with no obvious answer. Nothing in the text says how long "meeting" should take, where to pause, or which word carries the emphasis that makes the sentence sound like an answer rather than a question. Human readers supply that automatically. A model has to predict it.

This is the mirror image of speech recognition, and the asymmetry matters: recognition maps a big messy signal to a small clean one, and can be graded against a transcript. Synthesis maps a small clean input to a big messy output, and there are thousands of correct answers.

Stage one: from letters to sounds

English spelling is not a reliable guide to pronunciation - "read" has two, "colonel" has none of the letters you would expect. So most systems first convert text into phonemes, the atomic sound units of a language, using a pronunciation dictionary plus rules for unknown words. This step is where numbers, dates, abbreviations and names get expanded: "Dr." becomes "doctor" or "drive" depending on context, "1998" becomes "nineteen ninety-eight."

This front end is often plain code rather than a neural network, which is why a tiny speech model can still handle text well - the linguistic knowledge lives outside the learned part. It is also where most embarrassing errors originate, and it is a very different job from the tokenization a language model does.

Stage two: duration and the acoustic model

Next the model predicts how many audio frames each phoneme gets, then produces a mel spectrogram - a compact picture of the sound, with time along one axis and frequency along the other, brightness showing energy. A spectrogram is roughly a thousand times smaller than the waveform it describes, which is exactly why the field uses it as an intermediate step: predict the small thing, then expand it.

The two families here are worth distinguishing. Early systems like Tacotron 2 generated the spectrogram one frame at a time, conditioning each on the last, which sounded good and failed spectacularly when it lost its place - repeating or skipping words. Later systems predict every duration up front and generate all frames in parallel, which is faster and far more stable. Glow-TTS showed you could find the text-to-audio alignment automatically using a normalizing flow rather than attention, and that made parallel generation practical.

Stage three: the vocoder, where the parameters hide

Converting a spectrogram back to a waveform is the vocoder's job, and it is genuinely hard because the spectrogram threw away phase information. DeepMind's WaveNet solved quality in 2016 by predicting audio one sample at a time - producing speech that fooled listeners, and taking minutes of computation per second of audio.

Making that fast took a different approach. HiFi-GAN in 2020 used a generative adversarial setup: one network produces waveforms, several discriminators judge them at different time scales, and the result runs hundreds of times faster than real time with quality people accept.

This is the part to watch when someone quotes a parameter count. Published numbers frequently cover only the acoustic model, with the vocoder as a separate download that may be larger. A "5 million parameter" system needing a 50 million parameter vocoder is a 55 million parameter system. That is precisely why a complete stack fitting in 9.4 million parameters is a real result rather than a marketing number.

The VITS architecture, published in 2021, collapsed the stages: a variational autoencoder with flows and adversarial training, learning text to waveform end to end, with the duration model built in. Most compact open TTS systems today are VITS descendants.

Cloning, and what comes next

Voice cloning works by encoding a short reference clip into a speaker vector capturing timbre and style, then conditioning generation on it. Trained across thousands of speakers, the model interpolates to a new voice from seconds of audio. Systems like VALL-E went further, treating speech as a sequence of discrete audio tokens and predicting them with a language model - which brought in-context learning to speech, along with an obvious set of impersonation risks.

The remaining hard problem is not intelligibility, which is solved. It is appropriate prosody: knowing that a sentence is sarcastic, that this comma is a breath and that one is a list, that the answer to a question should fall rather than rise. Getting that right requires understanding the text, not just pronouncing it.

Ground Truth. (2026, July 25). Neural text-to-speech: how a model turns writing into a voice. Ground Truth. https://groundtruth.day/learn/neural-text-to-speech.html

@misc{groundtruth:neural-text-to-speech,
  title  = {Neural text-to-speech: how a model turns writing into a voice},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/learn/neural-text-to-speech.html}
}

Topics: text-to-speech · speech · audio · fundamentals · generative-models