attention
Multi-head latent attention: compressing the memory that inference actually runs out of Lesson
Multi-head latent attention compresses the key-value cache that language models must hold in GPU memory during generation, projecting keys and values into a small shared latent vector instead of storing them per head. DeepSeek introduced it in DeepSeek-V2, reporting a cache reduction of more than 90% with quality matching full attention.
An open lab shipped six models at once, and released the checkpoints and data recipes too News
IFM released K2 Horizon as six Apache 2.0 models spanning 375 billion down to 0.9 billion parameters that share architecture, vocabulary and training methodology, publishing intermediate checkpoints, data-construction recipes, training code and logs alongside the final weights.
Causal masking and prefix invariance: how a model is stopped from reading ahead, and why the mask is no longer proof Lesson
Causal masking is the mechanism that stops a language model from seeing tokens it is supposed to predict, and prefix invariance is the property it is meant to guarantee. In modern hybrid architectures the mask no longer covers every place information can leak.
One layer creates the giant activations behind attention sinks News
Researchers identified a single layer, consistent across model families, where the outsized activations that produce attention sinks first appear, and showed that loosening that token's rigidity improves instruction following and math reasoning without retraining.
Attention sinks: the token that soaks up attention and means nothing Lesson
An attention sink is a token, almost always at the very start of a sequence, that absorbs a large share of a model's attention regardless of what it says, and it exists because softmax forces attention to add up to one whether or not anything deserves it.
Baidu's OCR Model Reads Dozens of Pages in One Pass Without Its Memory Growing News
Unlimited OCR replaces the decoder's attention layers with a design that holds the key-value cache at a constant size, so transcription speed does not degrade as the document gets longer.
Linear Attention Lesson
Linear attention rewrites the transformer's attention math so its cost grows in a straight line with sequence length instead of exploding as the square, letting models handle very long inputs cheaply, at the price of a compressed memory of the past.
FlashAttention: making attention fast by respecting the memory hierarchy Lesson
FlashAttention is an exact attention algorithm that runs much faster and uses far less memory by tiling the computation to keep it in the GPU's fast on-chip memory - never writing the giant attention matrix to slow memory - which is why it became the default attention kernel in modern transformers.
A new attention method learns what to ignore, and reads 64x more text News
Tencent's Hunyuan team introduced HiLS, a sparse-attention method that learns end-to-end which parts of a long document to focus on, matching full attention while handling context 64 times longer than it was trained on.
Positional encoding: how transformers know word order Lesson
Positional encoding is how a transformer knows the order of words, since its attention mechanism sees the input as an unordered set. Modern models use rotary position embeddings (RoPE), which encode a token's position by rotating its vector by an angle proportional to where it sits in the sequence.
Sparse Attention Lesson
Sparse attention lets a transformer skip most of the pairwise comparisons between tokens, so instead of every token attending to every other token, each one attends to a chosen subset -- which is what makes million-token context windows affordable.
The KV cache: why AI gets slower and hungrier the longer it talks Lesson
The hidden notebook that lets a model avoid re-reading every previous word - and the single biggest reason long context is expensive.
DeepSeek's new open models give everyone a million-word memory by default News
DeepSeek previewed two free-to-download V4 models that can read a million tokens at once, no longer as a premium add-on but as the standard setting.
Transformers: the engine inside almost every modern AI Lesson
The neural-network design behind GPT, Claude, and nearly every modern AI model, and the one idea, attention, that made it work.
A small but elegant idea: putting 'experts' inside the attention layer News
Grouped Query Experts brings the mixture-of-experts trick into attention, activating only half a model's query heads per token while matching the full version -- at least at small scale.
A Classic Efficiency Trick Just Moved Into a New Part of the AI News
For years, the committee-of-specialists design that keeps big models fast lived in one layer of the network. A clean new result shows it works in the attention layer too, halving some of the work for free.
K2 Horizon (IFM) Tool
Six Apache 2.0 models from 375B-A23B down to 0.9B, sharing architecture, vocabulary and training methodology, with 512K context on all but the smallest. The 36B-A4B is a 74.9 GB bf16 download or 48.4 GB in FP8, with a serving recipe validated on two H200 GPUs. Upstream llama.cpp support is still in progress.