85-Model Scaling Study: Does LLM Social Simulation Improve with Scale?

Forum topic · ✨步子哥 · 2026-07-03

Summary

A Stanford study examined whether scaling laws apply to LLM-based social simulation, where models must match population response distributions rather than produce single correct answers. The team pretrained 85 transformer models (10^18 to 10^20 FLOPs) using a fixed Qwen3 architecture on DCLM web text, then evaluated opinion simulation (WVS), behavioral simulation (Psych-101), and longitudinal wellbeing prediction. Results show compute scaling significantly improves simulation fidelity across all three domains, suggesting today's imperfect simulations will improve with next-generation models. However, scaling amplifies data inequality: populations better represented in English web corpora improve fastest, while minority groups lag. Human cognitive biases like risk aversion do not scale with model size, and roughly one-third of behavioral tasks show poor scaling-law fits (r² < 0.3). Notably, knowledge-heavy benchmarks (e.g., MMLU) predict simulation fidelity while coding and formal reasoning do not, indicating world knowledge—not reasoning—drives social simulation. Paper: https://arxiv.org/abs/2607.02464

Will Scaling Improve Social Simulation with LLMs?

> Paper: Will Scaling Improve Social Simulation with LLMs? > Authors: Caleb Ziems, William Held, Su Doga Karaca, David Grusky, Tatsunori Hashimoto, Diyi Yang > Institutions: Stanford University, Open Athena > Link: https://arxiv.org/abs/2607.02464 > Code: https://github.com/SALT-NLP/social-scaling

The Question

Social scientists increasingly use LLMs for "social simulation"—having models role-play different populations to answer surveys, make decisions, and simulate long-term behavior change. But simulations today are imprecise. The open question: if we scale models up, will they become accurate?

Unlike traditional NLP tasks with single correct answers, social simulation requires matching a distribution—e.g., 30% of a population choosing A, 50% B, 20% C. Whether distribution matching improves with scale had never been systematically tested—until now.

Experimental Design

The Stanford team pretrained 85 transformer models spanning 10^18 to 10^20 FLOPs, using a fixed Qwen3 architecture and the DCLM web corpus—same architecture, data, and training strategy, with compute as the only variable. They also evaluated 35 larger open-source models (up to 70B) to validate extrapolation of the scaling law.

Three test domains:

1. Opinion simulation (WVS): simulating survey responses across countries via the World Values Survey 2. Behavioral simulation (Psych-101): simulating human decisions in psychology experiments 3. Longitudinal prediction: tracking the same population's wellbeing over time

Main Finding: Scaling Works

Across all three domains, compute scaling significantly improved simulation fidelity from 10^18 to 10^20 FLOPs. If LLM social simulation feels inaccurate today, next-generation models will likely do better—no paradigm shift required.

Opinion simulation: scaling gains were strongest for populations well represented in English web corpora (e.g., mainstream Western groups)
Behavioral simulation: most tasks improved, especially decisions involving independent, stable reward distributions
Longitudinal prediction: scaled, but slowest

Three Important Caveats

1. Underrepresented populations get left behind. The more a group is represented in English web text, the more scaling improves its simulation. Scaling is not neutral—it amplifies existing data inequality, much as the internet widened the digital divide between English and low-resource languages.

2. Cognitive biases don't scale. Tasks involving human cognitive biases showed no improvement. Risk aversion, for example, did not become better calibrated even as fine-tuned models grew from 0.5B to 8B parameters. Correlated reward learning (transferring heuristics across related tasks) also failed to scale. The authors suggest these capabilities depend on an implicit understanding of human cognitive structure, which next-token prediction does not directly optimize.

3. One-third of tasks fit scaling laws poorly. In behavioral simulation, ~1/3 of tasks had low scaling-law fits (r² < 0.3), sharing features like individual variability (e.g., clinical populations), non-stationary rewards, and skewed population distributions.

What Predicts Simulation Fidelity?

Which general benchmarks predict social simulation ability?

Positive: knowledge-heavy benchmarks (MMLU) and knowledge-based reasoning → high fidelity
Negative/unrelated: coding and multi-step formal reasoning → little help

The paper's first principal component (PC-1) correlates linearly with simulation fidelity, implying social simulation is fundamentally about world knowledge, not reasoning ability. A model that knows history, sociology, and culture outperforms one that solves math problems.

Limitations

1. Only three subdomains tested; open-ended generation and agentic systems were not covered 2. Single architecture (Qwen3); other architectures may scale differently 3. PC-1's causal role is unclear 4. Distribution matching ≠ behavioral realism—models may match distributions without human-like internal processes

Implications for AI Research

Scaling isn't万能—but closer than expected. Identifying blind spots matters more than celebrating wins.
Pretraining data composition shapes scaling direction. Targeted data balancing may be needed, not just scale.
General ability ≠ social simulation ability. Future "social simulation models" may diverge from "general reasoning models"—breadth of world knowledge versus depth of logical reasoning.

Conclusion

The paper's honesty is its standout quality: it maps precisely where scaling works, where it fails, and why. The good news for social scientists: LLM social simulation will keep improving without a paradigm breakthrough. The bad news: improvement will concentrate on already well-simulated populations while hard-to-simulate groups lag. Scaling won't solve fairness automatically—it makes the strong stronger.

---

Paper: https://arxiv.org/abs/2607.02464 Code: https://github.com/SALT-NLP/social-scaling