Wisdom of LLM Crowds: Aggregating 15 LLM Predictions and the Training Data Contamination Trap

Forum topic · 小凯 · 2026-07-22

Summary

A detailed Chinese-language review of the paper 'Wisdom of LLM Crowds: Aggregation and Contamination in Language Model Ensembles' (Douven, arXiv:2607.18269) explains whether classic crowd wisdom holds when humans are replaced by large language models. The study queried 15 LLMs on 254 binary prediction-market questions and compared aggregation methods: simple averaging, median, weighted averaging, an MLP neural aggregator, and logistic regression. The neural aggregator beat every individual model and traditional method, but a simple logistic regression performed nearly as well, showing the benefit comes mainly from a learned linear combination. Symbolic regression revealed that model disagreement was the key learned signal: consensus warrants confidence, disagreement warrants caution. The most striking finding concerns training data contamination: the gap between frontier cloud models and smaller local models shrank from 35.8% on potentially contaminated questions to 8.9% on clean, post-cutoff questions, with model rankings also shifting. Even aggregated LLM predictions remained substantially less accurate than real prediction markets. Practical takeaways: aggregate multiple models, favor simple learned weighting, monitor disagreement, and control contamination in benchmarks.

Wisdom of LLM Crowds: When Machines Learn 'Two Heads Are Better Than One'

This post is a Chinese forum author's deep-dive commentary on the paper "Wisdom of LLM Crowds: Aggregation and Contamination in Language Model Ensembles" (Douven, arXiv:2607.18269). Below is a structured English summary of the key findings.

Key points

The core question

Classical crowd wisdom (Galton's 1907 ox-weight experiment, where the median of 787 guesses was within 1% of the true weight) relies on independent, roughly equally capable judges. Do these conditions hold when the crowd is 15 different LLMs?

The experiment

The author posed 254 binary prediction-market questions (e.g., election outcomes, product launches, economic indicators) to 15 LLMs, spanning frontier cloud models (GPT-4 series, Claude, Gemini) and smaller local/open models.
Each model gave a probability estimate (0–1); various aggregation methods were then compared.

Aggregation method showdown

Simple average and median: reasonable baselines, but not the best for LLMs.
Weighted average: intuitive but risks overfitting to historical performance.
MLP neural aggregator: beat every individual model and all traditional aggregation methods.
Logistic regression: performed almost as well as the neural network — the benefit of aggregation comes mainly from a learned *linear combination*, not complex nonlinear interactions.
Symbolic regression on the neural network revealed the key learned signal is model disagreement: trust consensus when models agree, be cautious when they diverge.

Contamination: the paper's most striking finding

On questions whose events occurred *before* a model's training cutoff, models may effectively "remember" the answer rather than predict.
The capability gap between frontier cloud models and smaller local models was 35.8% on potentially contaminated data, but only 8.9% on clean (post-cutoff) data.
Model rankings shifted on clean data — leaderboard positions may be unreliable without contamination controls.
Even so, aggregated LLM predictions remained substantially less accurate than real prediction markets, which benefit from real economic incentives, live information, and market dynamics like hedging and arbitrage.

Why LLM crowds differ from human crowds

Independence is an illusion: LLMs share similar training data and the same Transformer architecture, making the "crowd" more homogeneous than 15 humans.
Capability is long-tailed, not normally distributed: the best individual model may matter far more than in human crowds.
Quality over quantity: a diverse set of 5 models may beat 15 homogeneous ones.

Practical implications

Aggregate multiple models rather than relying on one, even a frontier model.
Simple learned weighting (logistic regression) is often enough; watch disagreement as an uncertainty signal.
Benchmarks should use only events after all models' training cutoffs — many leaderboards may be contaminated.
Governance questions arise: who is responsible when an aggregated prediction from 15 models leads to a bad decision?

Conclusion

"diversity cancels bias, independence preserves signal."

References

Douven, I. (2026). Wisdom of LLM Crowds: Aggregation and Contamination in Language Model Ensembles. arXiv:2607.18269.
Galton, F. (1907). Vox Populi. Nature, 75(1949), 450-451.
Surowiecki, J. (2004). The Wisdom of Crowds. Doubleday.