Wisdom of LLM Crowds: Aggregating Predictions from 15 Language Models and the Training Data Contamination Trap

Forum topic · 小凯 · 2026-07-22

Summary

This post analyzes the paper 'Wisdom of LLM Crowds' (Douven, arXiv:2607.18269), which tests whether the classic 'wisdom of crowds' effect holds when the crowd consists of large language models. The study asked 15 LLMs—including frontier cloud models and smaller local models—to give probability estimates on 254 binary prediction-market questions, then compared aggregation methods: simple averaging, median, weighted averaging, a neural network (MLP) aggregator, and logistic regression. Key findings: learned aggregation beats both individual models and naive averaging; a simple logistic regression nearly matches the MLP, suggesting the benefit comes mostly from a linear combination; symbolic regression reveals the learned strategy largely reduces to weighting 'model disagreement'—consensus signals reliability. Most strikingly, the paper documents severe training data contamination: on contaminated questions, the capability gap between frontier and small models was 35.8%, but only 8.9% on questions occurring after all training cutoffs, implying much of the apparent superiority of large models reflects memorized answers rather than genuine forecasting. Model rankings also shifted on clean data. Even aggregated LLM predictions remain substantially less accurate than real prediction markets. Practical takeaways include preferring simple learned aggregation, monitoring model disagreement, and using only post-cutoff events for evaluation.

Wisdom of LLM Crowds: Aggregating Predictions from 15 Language Models

This post is a detailed Chinese-forum analysis of the paper "Wisdom of LLM Crowds: Aggregation and Contamination in Language Model Ensembles" (Douven, arXiv:2607.18269), exploring whether the classic "wisdom of crowds" phenomenon—famously demonstrated by Francis Galton's 1906 ox-weighing experiment, where the median of 787 crowd guesses was within 1% of the true weight—extends to large language models.

Key points

Experimental setup: 15 LLMs (frontier cloud models such as GPT-4-class systems, plus smaller local, open-source, and closed-source models from OpenAI, Anthropic, Google, etc.) were asked for probability estimates (0–1) on 254 binary prediction-market questions (e.g., election outcomes, product launches, economic indicators).
Aggregation methods compared:
Simple averaging and median — decent but not optimal for LLMs.
Performance-weighted averaging — reasonable but risks overfitting to historical accuracy.
MLP neural network aggregator — outperformed every individual model and all traditional aggregation methods.
Logistic regression — performed *nearly as well* as the neural network, implying the gains come mainly from learning a simple linear combination rather than complex nonlinear interactions.
Symbolic regression insight: Applying symbolic regression to the trained network revealed that the learned strategy largely reduces to model disagreement: when models agree, trust the consensus; when they diverge, be cautious.

The contamination finding: 35.8% vs 8.9%

The paper's most striking result concerns training data contamination:

On the "contaminated" set (events before some models' training cutoffs): the gap between frontier cloud models and smaller local models was 35.8%.
On the "clean" set (events after all cutoffs): the same gap shrank to 8.9%.

In other words, much of the apparent superiority of large models may reflect "remembering" answers embedded in training data rather than genuine forecasting ability. Model rankings also became unstable on clean data—suggesting benchmark rankings are unreliable without controlling for contamination. Even so, aggregated LLM predictions remained substantially less accurate than real prediction markets, where participants have economic incentives, real-time information, and genuine expertise.

Why LLM crowds differ from human crowds

1. Independence is questionable — LLMs share similar training data *and* the same Transformer architecture; 15 models may be more homogeneous than 15 people. 2. Ability distribution is long-tailed — unlike roughly normal human ability distributions, frontier vs. small models can differ by tens of points, so the best individual may rival the crowd average. 3. Contamination is a systemic bias, analogous to insider trading distorting a market price. 4. Quality and diversity may matter more than sheer count.

Practical implications

Don't rely on a single model; aggregation still helps even atop frontier models.
Simple learned weighting (logistic regression) is often sufficient—no deep aggregator needed.
Track model disagreement as a reliability signal.
Evaluations should use only events after all models' training cutoffs; many current benchmarks are likely contaminated.
For AI governance: regulators should assess *aggregate* risk when multiple AI systems feed one decision, and transparency about single-vs-ensemble deployment matters.

Conclusion

Yes, LLM crowds exhibit wisdom—but only with the right aggregation method, contamination controls, and awareness of how LLM crowds differ from human ones. As the post puts it: the core of crowd wisdom is not "more heads are better," but that "diversity cancels bias and independence preserves signal."

References

Douven, I. (2026). Wisdom of LLM Crowds: Aggregation and Contamination in Language Model Ensembles. arXiv:2607.18269.
Galton, F. (1907). Vox Populi. Nature, 75(1949), 450-451.
Surowiecki, J. (2004). The Wisdom of Crowds. Doubleday.