Learn · Intermediate

Simulating People with Language Models

Simulating people with language models means using a model as a stand-in for a human respondent, conditioning it on information about a specific person or population and then asking it the questions you would have asked them. It works better than most people expect, well enough that companies now sell simulated market research to enterprise clients, and it fails in specific, measurable ways that are worth understanding before you trust any output.

The idea has a clear starting point. In 2022, Argyle and colleagues published Out of One, Many, showing that conditioning a language model on demographic backstories produced response distributions that correlated surprisingly well with real survey data from those groups. They called the property algorithmic fidelity. The model was not simulating one person; it was reproducing something like the distribution of a population, because that population's writing was in its training data.

Three generations of the idea

Populated prototypes. Park and colleagues' Social Simulacra applied this to design. If you are building an online community, you cannot test the moderation rules until you have members, and you cannot get members until the rules work. Social Simulacra generated a plausible population of users producing plausible posts, including the antisocial ones, so a designer could see how a space would fail before shipping it.

Generative agents. The 2023 paper Generative Agents built a small sandbox town of twenty-five characters and gave each one an architecture worth knowing, because it became the template for agent memory generally. A memory stream logs everything the character experiences in natural language. A retrieval step surfaces memories by recency, importance, and relevance to the current moment. A reflection step periodically reads recent memories and writes higher-level conclusions, turning "Klaus was at the library on Tuesday, Wednesday, and Thursday" into "Klaus is deeply engaged in his research." Planning then converts those conclusions into a daily schedule. Run that loop and the characters coordinate, spreading news of a party through the town without anyone scripting it.

Grounded individuals. The most rigorous work is the 2024 paper LLM Agents Grounded in Self-Reports, and its design is what makes the numbers interpretable. Researchers recruited 1,052 Americans stratified to approximate national distributions on age, gender, race, region, education, and party, then conducted roughly two-hour voice interviews with each person. Those transcripts conditioned an agent per participant. Two weeks later the same people returned for held-out survey items, a personality inventory, five behavioral economics games, and five replicated experiments.

The headline is that interview-grounded agents reached 86% on held-out survey items, but the denominator is the clever part. The comparison is not raw accuracy against ground truth. It is normalized against each participant's own two-week consistency, how well that person matched their own earlier answers. Humans score well below perfect on that test. Measured against that honest bar, agents built from a two-hour interview reached 86%, agents from interviews alone 83%, and agents from survey data alone 82%.

Why the interview beats the demographics

That gap, 86 against 82, is small in absolute terms and large in what it implies. Demographic conditioning tells the model which stereotype to load. An interview transcript tells it what this particular person actually thinks, including the parts that cut against their demographic profile. This is the mechanism behind the field's central failure mode: flattening. A model conditioned on "45-year-old rural conservative" produces the most legible version of that category, and real populations contain far more internal variance than the most legible version admits. Simulated populations come out more homogeneous, more stereotyped, and more agreeable than real ones, which is exactly the direction that makes a market-research result comforting and wrong.

The analogy is a wind tunnel. Enormously useful for narrowing a design space, cheap enough to run hundreds of times, and never a substitute for flying the aircraft. The tunnel models the air it was built to model. It does not model the gust nobody anticipated.

The commercial turn

This is no longer only academic. Simile, founded by the lead author of the generative-agents line, sells simulation to large organizations for testing launches, pricing, and campaigns, and its public materials describe validating against real humans weekly across thousands of evaluations, with a confidence label attached to each result. That last detail is the right instinct: a simulation that reports how much to trust it is a different product from one that just answers.

What to hold onto

Simulation is strongest for breadth before depth, screening many options cheaply so real human effort goes to the survivors. It is weakest wherever the answer depends on the tails of a distribution, on genuine novelty, or on a minority view the training data under-represents. And it inherits every bias in the underlying model, including sycophancy, which is a serious problem when the thing you are measuring is whether people like your idea. Related reading on this site: multi-agent systems and AI persuasion.

Ground Truth. (2026, August 21). Simulating People with Language Models. Ground Truth. https://groundtruth.day/learn/simulating-people-with-language-models.html

@misc{groundtruth:simulating-people-with-language-models,
  title  = {Simulating People with Language Models},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {aug},
  url    = {https://groundtruth.day/learn/simulating-people-with-language-models.html}
}

Topics: simulation · agents · social-science · evaluation · applied-ai