Scaling Experiments Across 85 LLMs: Does Bigger Really Improve Social Simulation?

> Paper: Will Scaling Improve Social Simulation with LLMs? > Authors: Caleb Ziems, William Held, Su Doga Karaca, David Grusky, Tatsunori Hashimoto, Diyi Yang > Institutions: Stanford University, Open Athena > Paper: https://arxiv.org/abs/2607.02464 > Code: https://github.com/SALT-NLP/social-scaling

A question that excites and worries sociologists

Over the past two years, more and more sociologists have started using LLMs for "social simulation" — having LLMs role-play different populations, answer surveys, make decisions, and even simulate long-term behavioral change. This is orders of magnitude faster and cheaper than traditional agent-based modeling.

But everyone shares an unresolved question: today's LLM simulations aren't accurate enough — but if we make models bigger, will they get better?

The answer has been entirely unknown, because social simulation differs fundamentally from traditional NLP tasks:

• Traditional tasks (QA, translation) have definite answers, and "bigger is better" is nearly a law.
• Social simulation requires a model to match a distribution — if 30% of a population picks A, 50% B, 20% C, the model must reproduce that distribution, not produce a single "correct answer."

Whether distribution matching improves with scale had never been systematically tested — until this paper.

Experiment scale: 85 models, from 10^18 to 10^20 FLOPs

The Stanford team did something remarkably rigorous: using a fixed Qwen3 architecture on the DCLM web-text corpus, they pretrained 85 transformers spanning 10^18 to 10^20 FLOPs.

These 85 models form a clean scaling curve — same architecture, same data, same training strategy, with compute as the only variable. They also evaluated 35 larger open models (up to 70B) to validate extrapolation of the scaling law.

Three test domains:

1. Opinion simulation (WVS): World Values Survey — simulating survey responses across national populations. 2. Behavioral simulation (Psych-101): simulating human decisions in psychology experiments and cognitive tasks. 3. Longitudinal prediction: tracking the same population's well-being over time.

Surprising finding: scaling does work

The results are counterintuitive — at least to the authors.

In all three domains, compute scaling works significantly. As models go from 10^18 to 10^20 FLOPs, simulation fidelity keeps rising. This means: if LLM social simulation feels inaccurate today, it will most likely improve with the next generation of models.

More specifically:

• Opinion simulation: scaling helps most for populations well-represented in English web text (e.g., mainstream Western groups).
• Behavioral simulation: most tasks improve with scale, especially decisions involving independent, stationary reward distributions.
• Longitudinal prediction: also scales, but slowest.

This reassures social scientists — the current scaling paradigm is not a dead end; social simulation doesn't need a paradigm shift, just continued scaling.

But there are three "buts"

But 1: Under-represented groups get left behind

The paper finds a harsh pattern: the better represented a population is in English web text, the more scaling improves it.

This means opinions of white, middle-class Western populations will be simulated ever more accurately, while minority, non-English, and marginalized groups improve much more slowly. Scaling is not neutral — it amplifies existing data inequality.

Analogy: it's like the internet itself — search engines democratized information access, but English content grew far faster than content in smaller languages, and the digital divide widened.

But 2: Cognitive biases don't scale

In behavioral simulation, one class of tasks fails to improve with scale at all — human cognitive biases.

For example, risk aversion: humans systematically over-avoid risk in gambles, one of the most classic findings in cognitive psychology. But bigger LLMs do not become more human-like in risk aversion — even going from 0.5B to 8B, calibration of fine-tuned models barely improves.

Similarly, correlated-reward learning: when tasks are related, humans transfer heuristics across them, but LLMs don't scale on such tasks either.

Why? The paper's analysis: these abilities depend on "implicit understanding of human cognitive structure," which is not what the pretraining objective (next-token prediction) directly optimizes. Models learn the statistical regularities of language, but not the cognitive structure of human decision-making.

But 3: One-third of tasks fit scaling laws poorly

In behavioral simulation, about 1/3 of tasks show poor compute-scaling fits (r² < 0.3) — for these, there is almost no predictable relationship between model size and simulation fidelity.

These non-scaling tasks share features:

• Strong individual differences (e.g., decision patterns of psychiatric patient groups)
• Non-stationary rewards (gambles whose reward distributions shift over time)
• Skewed population distributions (certain groups too rare in training data)

A deeper finding: what capability predicts simulation fidelity?

The paper runs a fascinating analysis: which general benchmarks predict social simulation ability?

The answer is unexpected:

• Positively correlated: knowledge-heavy benchmarks (MMLU, etc.) and knowledge-intensive reasoning → high simulation fidelity
• Negatively correlated / irrelevant: coding ability and multi-step formal reasoning → almost no help

The paper defines a PC-1 (first principal component) that correlates linearly with simulation fidelity. This implies: the core of social simulation is "world knowledge," not "reasoning ability." A model versed in history, sociology, and culture simulates human society better than one that excels at math problems.

Direct implication for future training: to improve social simulation, increasing humanities/social-science data may be more effective than adding more code and math data.

An honest assessment

Notable limitations:

1. Limited dimensions tested: three subfields can't cover all social simulation scenarios; open-ended generation and agentic systems were not tested. 2. Single Qwen3 architecture: other architectures may scale differently. 3. PC-1 causality unclear: knowledge and fidelity correlate, but whether causally or via a common factor is unknown. 4. Definition of "fidelity": distribution matching is not behavioral authenticity — a model may match a distribution while its internal "process" differs entirely from humans.

Implications for AI research

Three takeaways:

1. Scaling isn't everything, but it's closer to everything than expected

On traditional NLP tasks scaling always helps; in social simulation it mostly helps but has blind spots. Identifying the blind spots matters more than celebrating success — non-scaling tasks are exactly where targeted research is needed.

2. Pretraining data composition determines the direction of scaling

If a population or cognitive pattern is under-represented in the data, scaling amplifies the imbalance. Future social simulation models may need targeted data balancing, not just more scale.

3. General capability ≠ social simulation capability

Coding and math reasoning barely help social simulation. This suggests future "social-simulation-specialized models" may diverge from "general reasoning models" — one pursuing breadth of world knowledge, the other depth of logical reasoning.

Closing

What impresses most about this paper is its honesty. It doesn't simply declare "scaling works" or "scaling fails," but carefully maps the boundaries of scaling — where it works, where it fails, and why.

For sociologists, it's one piece of good news and one piece of bad news. Good news: LLM social simulation will keep improving without a paradigm shift. Bad news: it will improve fastest on populations that were already easy to simulate, and slowly on those that were hard.

Scaling won't automatically solve fairness. It will just make the strong stronger.

---

Paper: https://arxiv.org/abs/2607.02464 Code: https://github.com/SALT-NLP/social-scaling