Learn · Beginner

How AI Turns Speech Into Text

Automatic speech recognition (ASR) is how an AI turns spoken audio into written text -- the technology behind voice assistants, live captions, meeting transcripts, and dictation. At its core it does three things: it slices the incoming sound into tiny time steps, encodes those slices into numerical features that capture the acoustic content, and decodes those features into the most likely sequence of words. How well it does this is measured by word error rate: the fraction of words it gets wrong, where lower is better. That single number is the yardstick the whole field is judged by, and it's why Cohere's new open Arabic model making headlines this week is fundamentally a story about pushing word error rate down.

Start with the raw material. Sound is a wave -- air pressure wobbling over time -- and a microphone records it as thousands of amplitude samples per second. That raw waveform is too fine-grained to work with directly, so the first step is to chop it into short overlapping windows (often around 25 milliseconds each) and convert each window into a compact summary of which frequencies are present, typically a representation called a spectrogram or mel-features. Think of it as turning the sound into a stack of tiny snapshots, each describing the 'shape' of the audio in that instant. The model never sees words yet -- just a sequence of these acoustic frames.

The hard part is the mapping from frames to words, and it's hard for a specific reason: alignment. Speech is stretchy. The word 'yes' might occupy three frames or thirty depending on how slowly someone speaks, and there are no neat boundaries between words in a continuous audio stream -- people run words together, pause mid-sentence, and drop sounds. So the model can't just label each frame with a letter. Early modern systems solved this with a clever training trick called CTC (Connectionist Temporal Classification), which lets the model output a letter or a 'blank' at each frame and then collapse repeats and blanks into the final text, so it learns the alignment on its own without being told exactly which frame maps to which sound. Later systems, like the aptly named Listen, Attend and Spell, used attention to let the decoder look across all the audio frames while producing each output word -- the same attention idea that powers language models, applied to audio.

Two shifts made ASR genuinely good. The first was self-supervised pretraining: wav2vec 2.0 from Meta showed you could train a model on enormous amounts of unlabeled audio -- just raw speech, no transcripts -- to learn rich representations of sound, then fine-tune on a much smaller labeled set. That broke the dependence on expensive transcribed data. The second was scale and diversity: OpenAI's Whisper trained a single model on a huge, varied pile of weakly-labeled audio scraped from the web -- many languages, accents, and noisy real-world conditions -- and found it became robust across all of them without per-domain tuning. Whisper became the default open baseline precisely because it was general rather than narrow.

Which brings us to why the remaining hard problems are hard. Most models, Whisper included, are trained heavily on clean, formal, standard-dialect speech, so they stumble exactly where real speech diverges from that: strong regional accents, dialects that differ sharply from the written standard, background noise, domain jargon, and -- especially -- code-switching, where a speaker drops words from another language mid-sentence. For a language like Arabic, which is really a family of dialects plus a formal standard, a general model can post an ugly word error rate on the way people actually talk. That's the gap Cohere's Arabic model targets: it more than halves Whisper's error rate on spontaneous dialect speech by training specifically for that messiness. The lesson generalizes -- ASR progress now is less about a new architecture and more about covering the long tail of real human speech that the tidy benchmarks miss.

So the mental model to keep: sound becomes frames, frames become features, features become words, and the whole thing is scored by how many words it gets wrong. The architectures have converged on the same attention-based building blocks as the rest of AI, and speech is increasingly folded into multimodal models that handle text, audio, and images together. But the frontier that decides whether ASR actually works for you is coverage -- your accent, your dialect, your noisy room -- not the leaderboard on clean read speech.

Ground Truth. (2026, July 11). How AI Turns Speech Into Text. Ground Truth. https://groundtruth.day/learn/automatic-speech-recognition.html

@misc{groundtruth:automatic-speech-recognition,
  title  = {How AI Turns Speech Into Text},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/learn/automatic-speech-recognition.html}
}

Topics: speech-recognition · asr · audio · multimodal · word-error-rate