News · 2026-11-06

Stanford researchers compress a memory readout into 16 vectors

Stanford researchers built Memorilla, a memory system that reads a persistent store of document embeddings and gives a language model a question-specific 16-vector readout. In one 512-document test, the authors report substantially lower query latency than sending the full text to the model, although the underlying store still grows with the documents.

Key facts

• The supervised setup inserts 16 learned vectors into a frozen decoder’s input.
• One 512-document comparison took 739 milliseconds versus 19,932 milliseconds.
• The system led five of eight benchmark primary metrics in the authors’ comparisons.
• Primary sources: paper and Stanford project.

The researchers’ paper describes a frozen encoder that converts each document into an embedding, followed by learned memory slots that read those embeddings with the question in mind. The resulting vectors are inserted into the decoder’s input. Unlike ordinary retrieval, which typically selects passages of text, this module synthesizes a compact representation before the decoder reads it.

Imagine a librarian keeping an index card for every document and composing a new briefing card for each question. The briefing may connect clues spread across many documents, but the original index cards have not disappeared. Nor are the 16 learned vectors readable sentences. The approach reduces repeated decoder-side input, not the entire storage requirement.

The 27-fold latency comparison used a single NVIDIA H100 with 80 GB of memory, one query at a time, a 32-token generation cap and 512 documents. Encoding the documents is a separate write-side cost; the authors report 2.59 seconds for that collection. The project’s repository says its inference example downloads about 25 GB of models and calls for one 80 GB GPU. These are the authors’ configuration requirements, not a universal minimum.

The authors call the memory readout “soft” tokens because they are vectors in the model’s input space rather than ordinary words. The authors compare their method on personalization and question-answering datasets. It does not win everywhere: on PubMedQA the paper reports 59% accuracy, below several baselines. A separate text-game experiment increased success while retaining the full game history, so it does not show that the eight memory vectors in that setup replaced the history.

The public review record is an AI-generated review of an earlier paper version, not independent replication. It flags an important comparison issue: Memorilla received training on benchmark training splits, while several baseline systems used standard configurations. That makes it difficult to isolate the memory mechanism from task-specific supervision. The method is a promising design for agent memory, but it is not proof of reliable lifelong personal memory.

The reported token reduction is also narrower than the popular shorthand. On NarrativeQA, the full-context setup averaged 67,385 document tokens, compared with 16 memory slots, yielding the paper's largest per-benchmark ratio of 4,212 to one. The comparison excludes the question and shared instructions. It is a comparison of what enters the decoder for that benchmark, not the amount of information stored by the whole system.

The full-context comparator had its own constraint. The serving configuration extended a 32,768-token context window with a scaling method, but the authors report that 10.8% of NarrativeQA examples still exceeded the extended window and had their earliest chunks dropped. This does not erase the reported timing result; it shows that even the comparison called full context could not include every long document in full.

In a separate text-game experiment, the system added completed turns to the memory store and updated the memory module with reinforcement learning. A frozen agent's average success over three attempts rose from 18.6% to 24.2% in held-out games. Crucially, the prompt retained the full textual game history and the learned memory supplied eight additional vectors. That result therefore tests augmentation, not replacement of the history.

The architecture depends on the memory readout being projected into the decoder's input space. The reported implementation uses particular Qwen encoder and decoder models; the paper does not establish that the same trained readout transfers unchanged to other model families. The authors also warn that stored embeddings may encode sensitive information. Persistent memory requires its own access controls, retention choices and consent, even when the final readout is compact.

The authors identify a further tradeoff: synthesizing across many documents can help combine dispersed clues, while exact names and fine details can be lost. Their proposed direction includes combining latent memory with retrieval of original passages.

Primary source, verified: read the paper → (arXiv 2610.21375)

Ground Truth. (2026, November 6). Stanford researchers compress a memory readout into 16 vectors. Ground Truth. https://groundtruth.day/news/stanford-memorilla-question-conditioned-memory.html

@misc{groundtruth:stanford-memorilla-question-conditioned-memory,
  title  = {Stanford researchers compress a memory readout into 16 vectors},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {nov},
  url    = {https://groundtruth.day/news/stanford-memorilla-question-conditioned-memory.html}
}

Topics: research · memory · agents