Imagine you are an urban planner who wants to know residents' views on a new data center. Instead of spending months on surveys, you use an LLM to generate questionnaire answers from a thousand "synthetic residents." On average, 55% support expansion—nearly identical to the real survey's 53%. You breathe a sigh of relief and start making decisions based on this result.
But you may be stepping into a deep pit.
Getting the average right does not mean the distribution is right, the relationships are right, or the trends under changing conditions are right. This is the core problem revealed by Chidichimo et al. from the University of Edinburgh in *Artificial Societies Benchmark: A Validation Framework for Synthetic Research*.
The Promise and Trap of Synthetic Populations
Using LLMs to simulate human survey answers has become a popular method in social science research over the past two years: low cost, fast, and repeatable. But the authors raise a sharp question: a synthetic survey can reproduce average answers while being severely distorted along three dimensions:
1. How people differ (internal validity): if 10% of a real population is strongly opposed, a synthetic population may flatten that 10% into "slightly dissatisfied" 2. How answers relate to each other (construct validity): in reality "supporting the data center" and "caring about energy consumption" are negatively correlated; a synthetic population may get this relationship backwards 3. How responses change under new conditions (external validity): if residents are told the data center will create 500 jobs, real support may rise 15%, while a synthetic population might rise only 3% or as much as 30%
An 11-Test Health Check
The authors built a benchmark of 11 tests covering three types of validity:
Internal validity (IV) — does the distribution shape of synthetic answers match humans?
- Marginal distribution match (answer distributions per question)
- Test-retest consistency (humans change answers to the same question; models being too consistent is itself a problem)
- Conditional independence (whether relationships hold after controlling for other variables)
- Factor structure of personality scales (which items cluster together)
- Cross-source consistency (patterns of agreement between different human data sources)
- Trait-behavior mapping (does extraversion actually predict social behavior)
- Direction and magnitude of treatment effects
- Cross-condition prediction
- Behavioral prediction (can questionnaire answers predict behaviors like signing petitions)
- If you only need marginal distribution estimates, model A may suffice
- If you need to analyze variable relationships, you must use model B and supplement with human data
- If you need to predict intervention effects, no model is reliable enough—run a real experiment
Construct validity (CV) — do relationships between variables match humans?
External validity (EV) — do synthetic populations respond to changed conditions like humans?
Results from Nine Models
Testing 9 language models against 20 human data sources produced sobering findings:
Models are overly consistent. When humans answer the same question repeatedly, 15-30% change their answers. Most models change less than 5%. Chasing consistency moves models away from human nature.
Compressed scales. Humans use 1 and 5 on a 1-5 scale; models cluster around 2-4. Extreme opinions are systematically erased.
Altered trait relationships. In human data, extraversion and risk-taking are positively correlated (r≈0.4), but in synthetic populations from some models the correlation becomes negative.
Richer personas are not necessarily better. Giving models more background about respondents (age, occupation, education) improved some models' predictions and worsened others—because extra information makes models more confident in the wrong direction.
Good on one dimension ≠ good on another. GPT-4o performed best on internal validity but fell behind Claude 3.5 on external validity. There is no all-around champion.
Why This Matters
Synthetic populations are being used for serious decisions: policy simulation, market research, pre-testing social experiments. If researchers only check "is the average answer right," it is like declaring someone healthy after taking only their temperature.
The authors offer a precise analogy: it is like a thermometer that tells you the average temperature is 22°C, but not that some rooms are 5°C and others 40°C, not how temperature relates to humidity, and not what happens when you turn on the heater.
Scorecards: Not a Ranking, a Prescription
The framework's output is not a single leaderboard but a "scorecard"—reporting each model's performance on each test separately, so researchers can choose based on their specific needs:
My Take
This paper does something few others do: instead of proposing a better synthetic population method, it proposes a framework for testing whether synthetic populations are trustworthy. Like in food safety—ensuring food is detectable and traceable matters more than inventing tastier food.
The most striking of the 11 tests is test-retest consistency. Human inconsistency is not noise; it is a feature—the same person supports something today and opposes it tomorrow, perhaps because they are in a bad mood or just read a news story. A model being too consistent shows it has failed to simulate the contextual sensitivity of human cognition.
Another memorable finding: richer personas are not necessarily better. This is counterintuitive—more information should mean better performance. But extra information can make models more confidently wrong. This echoes the idea of a "rationalization shell": the model is not using information to make better judgments, but to build more plausible narratives.
Code is open-sourced: https://github.com/artificial-societies/artificial-societies-benchmark
---
Paper: Chidichimo, E., Jung, M. J., Wallis, F. P. S., & He, J. K. (2026). Artificial Societies Benchmark: A Validation Framework for Synthetic Research. arXiv:2609.30030 Link: https://arxiv.org/abs/2609.30030 Code: https://github.com/artificial-societies/artificial-societies-benchmark