Learn · Intermediate

Encoder, decoder, or both: the three ways to build a transformer

Every transformer is built from the same block, and the architecture family is decided by one thing: which positions each position is allowed to look at. An encoder lets every token see every other token in both directions, which makes it good at reading. A decoder masks the future so each token sees only what came before it, which makes it able to write. An encoder-decoder does both, reading one sequence completely and then writing another while referring back to it. GPT-style chat models are decoder-only, BERT-style search models are encoder-only, and translation-style models are typically encoder-decoder. Nothing else about the block changes.

The masking decision

Start from the transformer block: attention plus a feed-forward network, repeated. Attention lets each position build its representation as a weighted blend of other positions. The only question is which other positions are visible.

Allow all of them, in both directions, and you have an encoder. The word "bank" in "he sat on the river bank" can consult "river," which appears afterwards. This is called bidirectional attention, and it produces a richer representation of each token because the whole context informs it.

Now block every position from seeing anything to its right -- what practitioners call causal masking -- and you have a decoder. Each position knows only the past. This restriction looks like a handicap and is actually the enabling constraint: it means the model can be trained on every position of every sequence at once, predicting each next token, without ever being able to cheat by reading the answer. That single objective, described in our lesson on how AI picks its next word, is what scales.

The three families

Encoder-only models, of which BERT is the canonical example, read a complete input and emit a representation per token, plus usually one vector for the whole sequence. They do not generate. Because they cannot be trained by simply predicting the next word, they are trained by masking out random words and asking the model to fill them in -- an objective Google's Jacob Devlin and colleagues introduced in 2018 as masked language modelling. Their output is the input, understood. That makes them the workhorse of anything that needs a text-to-vector conversion: search, ranking, classification, and the embeddings that power retrieval-augmented generation.

Decoder-only models -- GPT, Claude, Llama, Qwen, and effectively every chat model you have used -- generate one token at a time, each conditioned on everything already written, including their own output. The prompt is not architecturally special; it is just tokens that happen to be there first. Everything the model produces gets fed back as input for the next step, which is why the KV cache exists and why generation is sequential.

Encoder-decoder models, such as the original 2017 transformer, T5, and BART, run both. The encoder reads the source completely and bidirectionally. The decoder generates the output token by token, and at each step attends both to what it has written and, through a separate cross-attention pathway, back to the encoder's full representation of the source. This is the natural shape for translation, summarisation, and speech transcription: a fixed input that should be understood as a whole, and a fresh output that should be written from scratch.

The analogy is a translator's workflow. An encoder-only model is the reader who finishes the whole document and can answer questions about it. A decoder-only model is the writer who composes forward, one word at a time, never revising. An encoder-decoder is the professional translator who reads the source completely, then writes the target while glancing back at the original for each phrase.

Why decoder-only took over

Encoder-decoder architectures were the default in 2018, and Google's T5 paper made the strongest case for them by reframing every task as text-to-text. Then decoder-only models absorbed the field, for reasons worth understanding.

The first is that next-token prediction is a universal objective. A question, its answer, a translation, a summary, and a classification label can all be written as a single stream of text, so one training procedure covers every task. Encoder-only models need a task-specific head bolted on and fine-tuned for each new job.

The second is in-context learning. Because a decoder-only model treats prompt and output as the same stream, examples placed in the prompt shape behaviour without any training at all. That property emerged from scale and turned out to be the thing that made these models general-purpose.

The third is plumbing. A decoder-only model has one stack, one attention pattern, and one cache. Encoder-decoder models have two stacks and cross-attention, which complicates every optimisation that matters at scale -- serving, caching, batching, and parallel training.

What survives

Encoders did not lose; they moved. Whenever you need a fixed vector for a piece of text rather than generated words, an encoder is still the right tool and is typically hundreds of times smaller than the generative model it serves. Every RAG pipeline runs an encoder over the corpus before a decoder ever sees a token, and most rerankers are encoders too. Encoder-decoder shapes also persist wherever the input is a genuinely different modality from the output -- speech recognition is the clearest surviving example.

The useful mental model: bidirectional attention is for understanding a thing that already exists, causal attention is for producing a thing that does not. Most real systems need both, and increasingly they get them from two different models rather than from two halves of one.

Ground Truth. (2026, August 13). Encoder, decoder, or both: the three ways to build a transformer. Ground Truth. https://groundtruth.day/learn/encoder-decoder-vs-decoder-only.html

@misc{groundtruth:encoder-decoder-vs-decoder-only,
  title  = {Encoder, decoder, or both: the three ways to build a transformer},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {aug},
  url    = {https://groundtruth.day/learn/encoder-decoder-vs-decoder-only.html}
}

Topics: transformers · architecture · embeddings · fundamentals · bert