Synthetic Survey Populations Match Averages but Hide Three Key Distortions

Imagine you are an urban planner who wants to know how residents feel about building a new data center. Rather than spending months on polling, you use an LLM to generate questionnaire answers from a thousand "synthetic residents." On average, 55% support expansion—almost identical to the 53% from the real survey. Relieved, you start making decisions based on this result.

But you may be stepping into a deep pit.

Getting the mean right does not mean the distribution is right, the relationships are right, or the response to change is right. This is the core problem revealed in *Artificial Societies Benchmark: A Validation Framework for Synthetic Research* by Chidichimo et al. from the University of Edinburgh.

The Promise and Trap of Synthetic Populations

Using LLMs to simulate human survey responses has become a popular method in social science research over the past two years: cheap, fast, and repeatable. But the authors pose a sharp question: a synthetic survey can reproduce average answers while being severely distorted in three dimensions:

1. How people differ (internal validity): if 10% of the real population is extremely opposed, the synthetic population may flatten that 10% into "mildly dissatisfied" 2. How answers correlate (construct validity): in reality "supporting the data center" and "caring about energy consumption" may be negatively correlated—the synthetic population might invert this relationship 3. How responses shift under changed conditions (external validity): if telling residents the data center creates 500 jobs raises real support by 15%, the synthetic population might rise only 3% or as much as 30%

An 11-Test Health Check

The authors built a benchmark of 11 tests covering three types of validity:

Internal validity (IV) — does the distribution shape of synthetic answers match humans? Tests include:

• Marginal distribution matching (per-question response distributions)
• Trial-to-trial consistency (humans change answers to the same question; a model being too consistent is itself a problem)
• Conditional independence (whether relationships hold after controlling for other variables)

Construct validity (CV) — do relationships between variables match? Tests include:

• Factor structure of personality scales (which items cluster together)
• Cross-source consistency (consistency patterns between different human data sources)
• Trait-behavior mapping (does extraversion actually predict social behavior?)

External validity (EV) — do synthetic responses to changed conditions match? Tests include:

• Direction and magnitude of treatment effects
• Cross-condition prediction
• Behavioral prediction (can questionnaire answers predict actions like signing petitions?)

Health Check Results for Nine Models

The authors tested 9 language models against 20 human data sources. The results are sobering:

Models are too consistent. When humans answer the same question repeatedly, 15-30% change their answers. Most models change less than 5%. Chasing consistency actually deviates from human nature.

Compressed scales. Humans use both 1 and 5 on 1-5 scales; models tend to cluster in 2-4. Extreme opinions are systematically erased.

Altered trait relationships. In human data, "extraversion" and "risk-taking" are positively correlated (r≈0.4), but in synthetic populations generated by some models, this correlation turns negative.

Richer personas are not necessarily better. Giving models more respondent background (age, occupation, education) improved some models' predictions and worsened others—because extra information can push the model more confidently in the wrong direction.

Good at one dimension ≠ good at another. GPT-4o performed best on internal validity, but trailed Claude 3.5 on external validity. There is no all-around champion.

Why This Matters

Synthetic populations are being used for serious decisions: policy simulation, market research, pre-testing social experiments. If researchers only check "is the average right," it is like declaring a patient healthy after taking only their temperature.

The authors offer a precise analogy: it is like a thermometer that tells you the average temperature is 22°C, but not that some rooms are 5°C and others 40°C, not the relationship between temperature and humidity, and not what happens when you turn on the heater.

The Scorecard: Not a Ranking, a Prescription

The framework's output is not a single overall leaderboard but a "scorecard"—reporting each model's performance on each test separately. Researchers choose models based on their specific needs:

• If you only need to estimate marginal distributions, model A may suffice
• If you need to analyze variable relationships, you must use model B and supplement with human data
• If you need to predict intervention effects, no model is reliable enough—you must run real experiments

This "prescription-style" output is far more useful than a single ranking. It acknowledges that "there is no best model, only the model best suited to your question."

My Take

This paper does something few have done: instead of proposing a better synthetic population method, it proposes a framework for testing whether synthetic populations are trustworthy. Like in food safety—before inventing tastier food, ensuring food is testable and traceable matters more.

Among the 11 tests, the one that struck me most is the "trial-to-trial consistency" test. Human inconsistency is not noise—it is a feature. The same person supporting something today and opposing it tomorrow may be due to a bad mood or a news story they just saw. A model being too consistent precisely shows it has failed to simulate the contextual sensitivity of human cognition.

Another finding worth remembering: richer personas are not necessarily better. This is counterintuitive—give the model more information and it should do better. But extra information can make the model more confidently wrong. This echoes the concept of a "rationalization shell": the model is not using information to make better judgments, but to construct more plausible narratives.

Code is open source: https://github.com/artificial-societies/artificial-societies-benchmark

---

Paper: Chidichimo, E., Jung, M. J., Wallis, F. P. S., & He, J. K. (2026). Artificial Societies Benchmark: A Validation Framework for Synthetic Research. arXiv:2609.30030 Link: https://arxiv.org/abs/2609.30030 Code: https://github.com/artificial-societies/artificial-societies-benchmark