attention
Baidu's OCR Model Reads Dozens of Pages in One Pass Without Its Memory Growing News
Unlimited OCR replaces the decoder's attention layers with a design that holds the key-value cache at a constant size, so transcription speed does not degrade as the document gets longer.
Linear Attention Lesson
Linear attention rewrites the transformer's attention math so its cost grows in a straight line with sequence length instead of exploding as the square, letting models handle very long inputs cheaply, at the price of a compressed memory of the past.
FlashAttention: making attention fast by respecting the memory hierarchy Lesson
FlashAttention is an exact attention algorithm that runs much faster and uses far less memory by tiling the computation to keep it in the GPU's fast on-chip memory - never writing the giant attention matrix to slow memory - which is why it became the default attention kernel in modern transformers.
A new attention method learns what to ignore, and reads 64x more text News
Tencent's Hunyuan team introduced HiLS, a sparse-attention method that learns end-to-end which parts of a long document to focus on, matching full attention while handling context 64 times longer than it was trained on.
Positional encoding: how transformers know word order Lesson
Positional encoding is how a transformer knows the order of words, since its attention mechanism sees the input as an unordered set. Modern models use rotary position embeddings (RoPE), which encode a token's position by rotating its vector by an angle proportional to where it sits in the sequence.
Sparse Attention Lesson
Sparse attention lets a transformer skip most of the pairwise comparisons between tokens, so instead of every token attending to every other token, each one attends to a chosen subset -- which is what makes million-token context windows affordable.
The KV cache: why AI gets slower and hungrier the longer it talks Lesson
The hidden notebook that lets a model avoid re-reading every previous word - and the single biggest reason long context is expensive.
DeepSeek's new open models give everyone a million-word memory by default News
DeepSeek previewed two free-to-download V4 models that can read a million tokens at once, no longer as a premium add-on but as the standard setting.
Transformers: the engine inside almost every modern AI Lesson
The neural-network design behind GPT, Claude, and nearly every modern AI model, and the one idea, attention, that made it work.
A small but elegant idea: putting 'experts' inside the attention layer News
Grouped Query Experts brings the mixture-of-experts trick into attention, activating only half a model's query heads per token while matching the full version -- at least at small scale.
A Classic Efficiency Trick Just Moved Into a New Part of the AI News
For years, the committee-of-specialists design that keeps big models fast lived in one layer of the network. A clean new result shows it works in the attention layer too, halving some of the work for free.