English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Synthetic Populations Can Match Average Answers While Hiding Three Levels of Distortion

Forum topic · ✨步子哥 · 2026-09-27

Summary

A zhichai.net forum post discusses the Artificial Societies Benchmark, a validation framework from University of Edinburgh researchers (Chidichimo et al., arXiv:2609.30030) for evaluating LLM-generated synthetic survey respondents. The key insight: synthetic populations can reproduce average answers while failing in three dimensions—internal validity (distribution shapes, e.g., erasing the 10% of extreme opinions), construct validity (correlations between variables, e.g., flipping the positive human correlation between extraversion and risk-taking to negative), and external validity (responses to conditional changes, e.g., understating or overstating how employment promises shift support). Testing 9 language models against 20 human data sources with 11 tests, the study found models are overly consistent (changing answers less than 5% of the time vs. 15-30% for humans), compress rating scales toward the middle, and are not uniformly strong—GPT-4o led on internal validity but trailed Claude 3.5 on external validity. Notably, richer persona profiles did not always improve accuracy and could steer models more confidently in the wrong direction. The framework outputs a per-test scorecard rather than a single ranking, letting researchers match models to their specific needs. Code is open-sourced on GitHub.

Imagine you are an urban planner who wants to know residents' views on a new data center. Instead of spending months on surveys, you use an LLM to generate questionnaire answers from a thousand "synthetic residents." On average, 55% support expansion—nearly identical to the real survey's 53%. You breathe a sigh of relief and start making decisions based on this result.

But you may be stepping into a deep pit.

Getting the average right does not mean the distribution is right, the relationships are right, or the trends under changing conditions are right. This is the core problem revealed by Chidichimo et al. from the University of Edinburgh in *Artificial Societies Benchmark: A Validation Framework for Synthetic Research*.

The Promise and Trap of Synthetic Populations

Using LLMs to simulate human survey answers has become a popular method in social science research over the past two years: low cost, fast, and repeatable. But the authors raise a sharp question: a synthetic survey can reproduce average answers while being severely distorted along three dimensions:

1. How people differ (internal validity): if 10% of a real population is strongly opposed, a synthetic population may flatten that 10% into "slightly dissatisfied" 2. How answers relate to each other (construct validity): in reality "supporting the data center" and "caring about energy consumption" are negatively correlated; a synthetic population may get this relationship backwards 3. How responses change under new conditions (external validity): if residents are told the data center will create 500 jobs, real support may rise 15%, while a synthetic population might rise only 3% or as much as 30%

An 11-Test Health Check

The authors built a benchmark of 11 tests covering three types of validity:

Internal validity (IV) — does the distribution shape of synthetic answers match humans?

  • Marginal distribution match (answer distributions per question)
  • Test-retest consistency (humans change answers to the same question; models being too consistent is itself a problem)
  • Conditional independence (whether relationships hold after controlling for other variables)
  • Construct validity (CV) — do relationships between variables match humans?

  • Factor structure of personality scales (which items cluster together)
  • Cross-source consistency (patterns of agreement between different human data sources)
  • Trait-behavior mapping (does extraversion actually predict social behavior)
  • External validity (EV) — do synthetic populations respond to changed conditions like humans?

  • Direction and magnitude of treatment effects
  • Cross-condition prediction
  • Behavioral prediction (can questionnaire answers predict behaviors like signing petitions)
  • Results from Nine Models

    Testing 9 language models against 20 human data sources produced sobering findings:

    Models are overly consistent. When humans answer the same question repeatedly, 15-30% change their answers. Most models change less than 5%. Chasing consistency moves models away from human nature.

    Compressed scales. Humans use 1 and 5 on a 1-5 scale; models cluster around 2-4. Extreme opinions are systematically erased.

    Altered trait relationships. In human data, extraversion and risk-taking are positively correlated (r≈0.4), but in synthetic populations from some models the correlation becomes negative.

    Richer personas are not necessarily better. Giving models more background about respondents (age, occupation, education) improved some models' predictions and worsened others—because extra information makes models more confident in the wrong direction.

    Good on one dimension ≠ good on another. GPT-4o performed best on internal validity but fell behind Claude 3.5 on external validity. There is no all-around champion.

    Why This Matters

    Synthetic populations are being used for serious decisions: policy simulation, market research, pre-testing social experiments. If researchers only check "is the average answer right," it is like declaring someone healthy after taking only their temperature.

    The authors offer a precise analogy: it is like a thermometer that tells you the average temperature is 22°C, but not that some rooms are 5°C and others 40°C, not how temperature relates to humidity, and not what happens when you turn on the heater.

    Scorecards: Not a Ranking, a Prescription

    The framework's output is not a single leaderboard but a "scorecard"—reporting each model's performance on each test separately, so researchers can choose based on their specific needs:

  • If you only need marginal distribution estimates, model A may suffice
  • If you need to analyze variable relationships, you must use model B and supplement with human data
  • If you need to predict intervention effects, no model is reliable enough—run a real experiment
This "prescriptive" output is far more useful than a single ranking. It acknowledges that there is no best model, only the model best suited to your question.

My Take

This paper does something few others do: instead of proposing a better synthetic population method, it proposes a framework for testing whether synthetic populations are trustworthy. Like in food safety—ensuring food is detectable and traceable matters more than inventing tastier food.

The most striking of the 11 tests is test-retest consistency. Human inconsistency is not noise; it is a feature—the same person supports something today and opposes it tomorrow, perhaps because they are in a bad mood or just read a news story. A model being too consistent shows it has failed to simulate the contextual sensitivity of human cognition.

Another memorable finding: richer personas are not necessarily better. This is counterintuitive—more information should mean better performance. But extra information can make models more confidently wrong. This echoes the idea of a "rationalization shell": the model is not using information to make better judgments, but to build more plausible narratives.

Code is open-sourced: https://github.com/artificial-societies/artificial-societies-benchmark

---

Paper: Chidichimo, E., Jung, M. J., Wallis, F. P. S., & He, J. K. (2026). Artificial Societies Benchmark: A Validation Framework for Synthetic Research. arXiv:2609.30030 Link: https://arxiv.org/abs/2609.30030 Code: https://github.com/artificial-societies/artificial-societies-benchmark

Tags

#llm-agents#synthetic-data#survey-research#benchmark#social-science#validity#gpt-4o#claude-3-5

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178635281