News · 2026-07-08

A new attention method learns what to ignore, and reads 64x more text

Tencent's Hunyuan team introduced HiLS (Hierarchical Landmark Sparse attention), a method that lets a language model learn which parts of a long document to pay attention to, instead of using a fixed rule. The payoff: it matches the quality of full attention within its training range while extrapolating to context 64 times longer than it was trained on, keeping 90% retrieval accuracy. It was the highest-scoring non-world-model paper on Hugging Face today, and it targets one of the field's most expensive problems.

Key facts

• HiLS learns chunk selection end-to-end under the language-model loss, not by heuristic (paper).
• Matches full-attention quality in-domain; extrapolates 64x beyond training context at 90% retrieval accuracy.
• Existing full-attention models can be converted with lightweight continued pretraining — no full retrain.
• From Tencent Hunyuan; the top-scoring efficiency paper on Hugging Face for July 8.

The background: attention is the mechanism that lets a model weigh how much every word relates to every other word. Its cost grows with the square of the input length, so doubling the document quadruples the work — which is why long documents are slow and expensive. The standard fix is sparse attention: only look at some chunks of the past, not all of them. The catch has always been picking the right chunks. Most methods use a fixed heuristic — attend to nearby text and a few landmarks — and when the heuristic guesses wrong, quality drops.

How HiLS works: it makes chunk selection part of the learning problem. The model retrieves candidate chunks, scores them, and fuses the selected ones into its output — and because those retrieval scores flow through the forward pass, the training signal reaches them. The model is literally trained to "learn what to ignore" in whatever way minimizes prediction error. Think of it like a researcher who, instead of always reading the first and last pages of every source, learns from experience exactly which pages tend to hold the answer — and gets better at that skill the more they practice.

Why it matters: long-context efficiency is the bottleneck behind everything from analyzing a whole codebase to reasoning over a book, and it feeds straight into the cost pressures reshaping the industry. A method that keeps full-attention quality, extrapolates far past its training length, and can be retrofitted onto existing models without retraining is unusually practical — most efficiency tricks force a quality sacrifice or a from-scratch rebuild. HiLS claims to avoid both.

The honest caveat: the headline numbers come from the authors' own evaluation, and "64x extrapolation at 90% retrieval" is measured on retrieval-style probes, which don't capture every long-context task. Ninety percent accuracy also means one in ten target facts is missed at extreme lengths — fine for some uses, unacceptable for others. As with any single-lab result, the real test is independent replication and whether the gains hold on messy real-world documents rather than clean benchmarks.

Primary source, verified: read the paper → (arXiv 2607.02980)

Ground Truth. (2026, July 8). A new attention method learns what to ignore, and reads 64x more text. Ground Truth. https://groundtruth.day/news/hils-attention-learns-what-to-ignore.html

@misc{groundtruth:hils-attention-learns-what-to-ignore,
  title  = {A new attention method learns what to ignore, and reads 64x more text},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/news/hils-attention-learns-what-to-ignore.html}
}

Topics: attention · sparse-attention · long-context · efficiency · tencent · research