Learn · Intermediate

Selective context expansion: keeping a map of a document before reading it closely

Selective context expansion is a way for an AI system to look across a long document cheaply before spending expensive attention on the exact pages it needs. It preserves a coarse map of the whole document, then uses tools to request readable text or images from selected locations.

Key facts

• The method separates locating evidence from reading evidence accurately.
• It is especially useful when layout, tables, figures and multi-page reasoning matter.
• Apple describes the approach in LensVLM, released today as research-only weights and code.
• Expansion re-expresses existing document content; it is not web search or new retrieval.

Long context is valuable but expensive. Feeding every word, chart and page image of a large report to a model costs memory, time and money. Compressing the document solves part of that problem, but creates another: the answer may sit in the small print of one page, a figure caption, a form field or a diagram whose layout carries meaning. If compression makes that detail unreadable, an otherwise smart model cannot recover it.

Ordinary retrieval-augmented generation takes a different path. It turns a corpus into chunks, indexes them, and retrieves the few chunks that look most relevant to a question. That is excellent when evidence is concentrated and a stable corpus will be queried many times. But retrieval can fail before reasoning begins: it may miss the one page whose terminology differs from the question, discard a needed second page, or flatten a form into text that loses its structure.

Selective context expansion treats the document more like an atlas. First, the model receives downscaled page images or another compact global representation. It cannot read every sentence, but it can see headings, page density, table shapes, charts and visual landmarks. It then says, in effect, “open page 37 at full quality” or “give me the native text behind page 12.” The answer is produced only after it has gathered the detail it selected. LensVLM calls this operation Expand(k) and can return original text, OCR or a high-resolution image depending on the source material.

The analogy is researching in a library. RAG is a librarian handing you the three passages most likely to contain the answer. Selective expansion is being allowed to scan the table of contents and tiny page thumbnails of the whole book, then ask the librarian to enlarge exactly the pages you notice. The latter can be slower for a trivial fact, but safer for a question that requires evidence from two visually distinct places.

This is an agentic design because the model decides whether to call a tool, which location to request, and whether the first result justifies another request. The policy needs training and evaluation. A system can waste money by zooming into too many pages; it can hallucinate if it trusts a blurred overview; and it can become brittle if the document style changes. The useful metrics therefore include answer accuracy, page-selection accuracy, number of expansions, total token cost, latency and performance on documents that genuinely require multiple pieces of evidence.

LensVLM's authors report that one expansion earns most of the gain and a second helps multi-hop questions, but those results are author-reported. The general lesson extends beyond Apple. Document agents should not be forced to choose between “read everything at maximum fidelity” and “retrieve a few text chunks and hope.” A staged workflow—overview, select, inspect, answer—can spend compute where it changes the decision.

The caveat is that a visual map is not automatically trustworthy. OCR can misread tables, thumbnails can hide a footnote, and permissions still govern what the system should be allowed to expand. In high-stakes work, cite the selected page, show the retrieved evidence, and make the expansion trail auditable. Selective context expansion is not a substitute for retrieval or long context. It is a third control knob: retain the map, then pay for detail deliberately.

Ground Truth. (2026, September 24). Selective context expansion: keeping a map of a document before reading it closely. Ground Truth. https://groundtruth.day/learn/selective-context-expansion.html

@misc{groundtruth:selective-context-expansion,
  title  = {Selective context expansion: keeping a map of a document before reading it closely},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {sep},
  url    = {https://groundtruth.day/learn/selective-context-expansion.html}
}

Topics: long-context · document-ai · vision-language-models · retrieval · tool-use