News · 2026-08-05

Models invent about 40 percent of what they claim to know about you

Tell a model three true things about yourself and it will hand back a fully furnished person. In a study posted August 5, researchers gave 12 language models exactly three first-person facts per user across 150 personas and six tasks, then had a validated judge sort every resulting claim by whether it was actually supported. Every single model invented or stereotyped between 35 and 49 percent of what it said, averaging 41.6 percent across more than 143,000 judged claims. Only about a quarter to a third of what a personalized answer says about you is something you told it.

Key facts

• Mean over-inference of 41.6 percent across 12 models from 7 families, ranging from 35 to 49 percent, over 143,000-plus judged claims.
• Posted August 5, 2026 as arXiv:2608.04570 by Yushi Sun, Yanjie Zhang, and Rui Sheng, with part of the work done during an internship at Tencent LIGHTSPEED.
• The benchmark, MirageBench, uses 150 personas given exactly 3 facts each, run through 6 tasks from a dating bio to identifying stress sources.
• The judge that classified claims was validated against 400 human-labeled examples, with agreement between 0.86 and 0.90.

The paper's own example is the clearest way in. A user states three things: works as a software engineer, goes rock climbing, once had an incident involving a cat. What comes back is a person who lives in a "modern minimalist apartment," "prefers nature trips over city tours," is "probably single," and "enjoys indie rock music." None of that was said. It is assembled from occupational and hobby stereotypes and then delivered with exactly the same confidence as the three real facts. Asked to describe someone's home, models reach for a stock template: warm lighting, comfortable seating, a few plants. That is not inference. It is furniture.

The four-way taxonomy the authors use is what makes this measurable rather than anecdotal. A claim is grounded if it restates what the user said, reasonable if it is one common-sense step beyond, a stereotype if it substitutes a demographic or occupational prior for the individual, and fabricated if there is no evidence at all. Over-inference is the last two added together. It varies a lot by task, from 27 to 59 percent, and the authors name three drivers: verbosity traps, where longer answers invent more in order to sound complete; pretraining priors filling gaps; and genre expectations, since a dating bio demands invented color that a stress summary does not.

The finding that should change how teams pick models

The second experiment is the better one. After each model produced its personalized answer, the authors asked it, in a separate turn, to audit its own output and classify each claim under the same four categories. The model plays judge on itself.

The result is what the authors name the Self-Monitoring Inversion. Across the 12 models, the ones reporting the least fabrication were the ones the independent judge caught fabricating the most, a negative correlation of about -0.60. The cleanest pair: one small open model rated its own answers as 13 percent over-inferred while the outside judge scored it worst of all 12 at 48.7 percent. Another model self-flagged 58.2 percent of its own claims while the judge measured it at 43.1 percent, squarely mid-pack.

The explanation is not that introspection is broken. It is a calibration split. Some models are strict self-auditors, quick to call their own inference a stretch, and that same self-skepticism visibly restrains how much they invent while generating. Others are lenient, waving nearly everything through as reasonable, and that permissiveness shows up as license to fabricate. Within a single model, self-audit still tracks something real: a claim it flags as shaky genuinely is more likely to be shaky. Across models, comparing self-reported confidence measures how strict each internal judge is, not how honest the output is.

The practical consequence is direct. A team choosing a model for a personalization feature by trusting each candidate's own reports of carefulness would rank them close to backwards. This is a specific, testable failure of the calibration and confidence signals people routinely lean on, and a close cousin of sycophancy in that the model is optimizing for how a good answer sounds.

It compounds

A smaller pilot ran eight-round conversations with models instructed to maintain a running memory profile. Nine of twelve grew their inferred-attribute count near-linearly, and the fastest accumulators went from three stated facts to more than 120 assumed attributes by round eight, while revising almost nothing, with revision rates between 0.4 and 5 percent against 70 to 82 percent for the most conservative models. The authors call it silent memory pollution: invented attributes are not corrected, they are compounded, so a long-running assistant's model of you drifts further from reality the longer you talk to it. That connects directly to how agent memory systems store and reuse what they infer, an area where nobody has been testing the failure modes.

The honest caveat

The authors flag the inversion correlation as exploratory. It rests on 12 model families and the bootstrap confidence interval technically contains zero, so treat it as a named, well-instrumented phenomenon rather than an established statistical law. The memory-pollution pilot is smaller still, just two personas, with a prompt that explicitly told models to retain information, so the right signal there is the cross-model contrast, where some accumulate a hundred times faster than others, rather than the absolute counts. MirageBench is promised as a full release but is not yet posted.

Primary source, verified: read the paper → (arXiv 2608.04570)

Ground Truth. (2026, August 5). Models invent about 40 percent of what they claim to know about you. Ground Truth. https://groundtruth.day/news/models-invent-about-40-percent-of-what-they-claim-to-know-about-you.html

@misc{groundtruth:models-invent-about-40-percent-of-what-they-claim-to-know-about-you,
  title  = {Models invent about 40 percent of what they claim to know about you},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {aug},
  url    = {https://groundtruth.day/news/models-invent-about-40-percent-of-what-they-claim-to-know-about-you.html}
}

Topics: research · personalization · hallucination · evaluation · agent-memory · privacy