Learn · Intermediate

Positional encoding: how transformers know word order

Positional encoding is how a transformer knows the order of the words it is reading. This is not automatic: the attention mechanism at the heart of a transformer treats its input as an unordered set, so without extra help it cannot tell 'dog bites man' from 'man bites dog.' Positional encoding is the mechanism that injects word order back in - and today's dominant version, rotary position embedding (RoPE), does it by rotating each token's vector by an angle that depends on where it sits in the sequence.

To understand why this is even a problem, you have to understand what attention actually does. In a transformer, every token looks at every other token and decides how much to pay attention to each, based purely on the content of their vectors. That's powerful, but it's also order-blind: attention computes the same thing whether a word is first or last, because it only compares vectors, not positions. Compare that to older architectures like recurrent networks, which read one word at a time and therefore know the order for free. Transformers gave up that built-in sense of sequence in exchange for speed and parallelism - so order has to be added back deliberately. See how this fits the full picture in the transformers lesson.

The original 2017 transformer paper, Attention Is All You Need, solved this with sinusoidal positional encoding. The idea: for each position in the sequence, generate a unique pattern of numbers using sine and cosine waves of different frequencies, and add that pattern to the token's embedding before it enters the model. Position 1 gets one wave pattern, position 2 another, and so on. Because the patterns are built from waves, the model can, in principle, learn to reason about relative distances between tokens. It works, but it has a weakness: it bakes position in as an absolute address added at the very start, which does not always generalize well when a model needs to handle sequences longer than any it saw in training.

The method that took over is rotary position embedding, or RoPE, introduced in the RoFormer paper in 2021 and now used in most large models. Instead of adding a position signal to the embedding, RoPE rotates the query and key vectors - the vectors attention uses to compare tokens - by an angle proportional to the token's position. A token at position 5 gets rotated five times as much as a token at position 1. The elegant consequence is that when two tokens are compared inside attention, the result depends only on the difference between their positions - their relative distance - not their absolute locations.

An analogy: think of each token's vector as the hand of a clock. RoPE spins each token's clock hand forward by an amount that matches its position in the sentence. When the model compares two tokens, what matters is the angle between their two clock hands - and that angle depends only on how far apart the words are. Two words five positions apart always have the same angular relationship, whether they sit at the start of the document or ten thousand tokens in. That relative-distance property is a big part of why RoPE-based models extend to long contexts more gracefully, and why techniques for stretching a model's context window often work by adjusting RoPE's rotation frequencies.

RoPE is not the only modern approach. ALiBi, from the Train Short, Test Long paper, skips position vectors entirely and instead adds a small, distance-based penalty to attention scores - the farther apart two tokens are, the more their attention is discounted - which also helps models trained on short sequences handle longer ones at test time. Different models make different choices, but the goal is always the same: give order-blind attention a reliable sense of where each token sits.

Why it matters: positional encoding is one of those quiet design choices that turns out to control a headline capability - how long a context a model can handle. The move from absolute sinusoidal encodings to relative, rotation-based ones like RoPE is a big reason models went from a couple thousand tokens of context to hundreds of thousands. The honest caveat is that no positional scheme fully solves length generalization: models still degrade when pushed far past their training length, and much of the engineering behind long-context models is about coaxing positional encodings - usually RoPE - to behave outside the range they were trained on. Ground Truth recently covered exactly this idea in its story on RoPE, words as rotations.

Ground Truth. (2026, July 6). Positional encoding: how transformers know word order. Ground Truth. https://groundtruth.day/learn/positional-encoding.html

@misc{groundtruth:positional-encoding,
  title  = {Positional encoding: how transformers know word order},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/learn/positional-encoding.html}
}

Topics: transformers · positional-encoding · rope · attention · architecture