Learn · Intermediate

Attention sinks: the token that soaks up attention and means nothing

An attention sink is a token that a language model pours a large share of its attention onto while extracting almost nothing from it. It is usually the very first token in the sequence, its content is irrelevant, and it exists because of a structural quirk: the softmax function that produces attention weights forces them to sum to one, so a model that has nothing worth attending to must still attend to something. The sink is where it puts the leftovers.

This sounds like a curiosity. It is actually one of the most operationally important facts about running long-context models, and it is why you cannot simply drop the oldest tokens from a conversation and carry on.

Why the model has no choice

Every attention head in a transformer works by scoring how relevant each previous token is to the current one, then converting those scores into weights with softmax. Softmax has one non-negotiable property: the weights it produces are all positive and always sum to exactly one. There is no option to output all zeros.

Now consider a head whose job is to detect something specific, say a matching open parenthesis. Most of the time in most sentences, there is no matching parenthesis to find. The head has nothing to contribute. But it cannot abstain. It must distribute a full unit of attention across the available tokens, and whatever it attends to gets mixed into the output.

The model's solution, discovered on its own during training, is to nominate a scapegoat. Pick a token whose value vector is close to harmless, and dump the unwanted attention there. The first token is ideal for this, because every position in the sequence can see it, it is present in every single training example, and in a chat model it is usually a template token with no semantic load. So attention heads across the network learn to point their idle attention at position zero.

The analogy that fits is a mandatory ballot. If a voting system requires everyone to vote and offers no abstention, you will get a large pile of votes for a joke candidate. Those votes are not endorsements. They are what abstention looks like when abstention is not on the form.

Why this matters in production

The consequence was documented in 2023 by Guangxuan Xiao and colleagues in Efficient Streaming Language Models with Attention Sinks, and it is counterintuitive enough to have surprised a lot of engineers. If you run a model over a conversation longer than its window and use the obvious fix, evicting the oldest tokens from the KV cache as new ones arrive, the model does not degrade gracefully. It falls off a cliff. Perplexity explodes and the output turns to noise.

The reason is that the oldest tokens include the sink. Remove it, and all the attention it had been absorbing has to go somewhere else, redistributed onto real tokens that were never meant to carry it. Every head that was quietly abstaining is now forcibly voting for something meaningful, and the residual stream fills with contributions nobody asked for.

The fix is almost comically simple and it works: keep the first four or so tokens pinned in the cache permanently, and slide the window across everything after them. Xiao's group called this StreamingLLM, and it let models handle millions of tokens of streaming input with stable perplexity, without any retraining. Four tokens of overhead.

Where the sink comes from

The functional story above explains why a sink is useful. It does not explain where the mechanism physically lives. That question connects to a second phenomenon: massive activations, values inside a model's hidden states that are thousands of times larger than everything around them, concentrated in a few dimensions of a few tokens. Mingjie Sun and colleagues documented these in Massive Activations in Large Language Models and showed they act like fixed bias terms rather than like content, and that removing them wrecks the model.

Recent work has tied the threads together. A Single Layer to Explain Them All identifies a specific layer, consistent across model families, where massive activations first emerge, shows that the normalization and feed-forward parameters in that layer jointly produce them, and finds that the affected token's representation then stays nearly unchanged as it travels deeper through residual connections. Loosening that rigidity, the authors report, "mitigates attention sinks by selectively weakening their influence" while improving instruction following and math reasoning.

That is the current best account: a particular layer manufactures an enormous, near-constant activation on one token, that token becomes an obvious place for attention heads to park, and the sink behavior everyone observes downstream is the visible consequence.

What to take away

Three things. First, when you see a model doing something structurally strange and doing it consistently across independently trained labs, the odds favor a mechanism rather than a bug. Sinks appear in essentially every large transformer because softmax appears in every large transformer.

Second, this is a case where a small amount of interpretability paid for itself immediately. Understanding why the first token mattered turned an unexplained quality collapse into a four-token fix.

Third, it constrains anything that manipulates context. Cache eviction, context compression, sliding windows and prompt trimming all have to respect the sink, which is why so many long-context systems have a hardcoded rule about keeping the first few tokens. If you build context machinery and skip that rule, you will find this lesson the hard way.

Related reading: mechanistic interpretability, sparse attention, and our report on the single layer that creates these activations.

Ground Truth. (2026, August 15). Attention sinks: the token that soaks up attention and means nothing. Ground Truth. https://groundtruth.day/learn/attention-sinks.html

@misc{groundtruth:attention-sinks,
  title  = {Attention sinks: the token that soaks up attention and means nothing},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {aug},
  url    = {https://groundtruth.day/learn/attention-sinks.html}
}

Topics: attention · transformers · interpretability · long-context · kv-cache · efficiency