The Illusion of Numbers: Why 30 AI Agents Voting Can't Beat a Single Shared Bias

> A Feynman-style walkthrough of Fortuna & Bertalanič, *Multi-agent Scaling Across Disjunctive and Compensatory Tasks* (arXiv:2609.31563)

Key points

• The premise: The multi-agent LLM community largely assumes that enlarging a team (N agents) improves performance, echoing the classic "wisdom of crowds" and the Condorcet Jury Theorem. This paper tests that assumption systematically.

The theoretical framework

• The authors import Steiner's 1972 taxonomy of group tasks into LLM research:
• Disjunctive (Y = max): team succeeds if *any* member is right — math reasoning, multiple-choice QA, code generation. Mechanisms: voting, verifier reranking, debate.
• Compensatory (Y = average): members give continuous estimates; errors cancel statistically — forecasting, Fermi problems (Galton's ox).
• Plus additive, conjunctive, and discretionary types (mapped but not experimentally tested).
• Disjunctive math: pass@N = 1 − (1 − p)^N grows, but voting under correlated samples converges (Proposition 2.1) to π, the fraction of items where the correct answer is the model's modal answer. Gains are bounded by |π − p̄| ≤ 2p̄(1−p̄)(1−ρ), where ρ is the intra-class correlation of correctness (≈ difficulty variance). When ρ → 1, voting gains vanish.
• Compensatory math: MSE(N) = b² + σ²/N_eff → b² + σ²ρ. Shared item-level bias bₖ and effective sample size capped at 1/ρ mean averaging removes noise but never the bias.

The experiments

• 13 open-weight models (3B–20B: smollm3-3b, qwen2.5 series, qwen3 series, mistral-7b, llama3.1-8b, phi4-14b, gpt-oss-20b, etc.); team sizes N ∈ {1,2,3,5,7,10,15,20,25,30}; 3 rounds of communication; temperature 0.4; ~6.8 × 10⁷ total generations; answer-first prompting.
• Disjunctive results: pass@30 shows real potential (e.g., GSM8K 34.3% → 54.4%, MATH-500 29.7% → 47.1%), but voting from N=2 to N=30 gains only +0.3 to +1.3 points. A resampling model predicts measured voting accuracy with mean error 0.48 points (r = 0.999) across 650 configurations. Intra-class ρ ranges 0.53–0.97 (mean 0.78).
• Reasoning-first models are different: gpt-oss-20b and r1-distill-qwen-14b (ρ = 0.66) gained +7.2 and +2.8 points from scaling, versus ±1.0 for others — explaining why self-consistency works with chain-of-thought.
• Temperature doesn't help voting: raising temperature to 1.0 boosts pass@30 but leaves N=30 vote accuracy unchanged — the added diversity lands on wrong answers; the modal answer never moves.
• Multi-round discussion: corrections nearly double GSM8K accuracy (34.1% → 61.9% at N=5), but the gain is *independent of team size* (+26.5 at N=2 vs. +26.6 at N=30). The benefit comes from being forced to re-reason, not from peer wisdom. Round-3 accuracy even declines at large N as shared wrong intermediate results become "error attractors."
• Compensatory results (RealFP Fermi problems): geometric-mean accuracy rises only from 31.6% (N=1) to 33.4% (N=30). Item-level bias accounts for β = 0.87 of squared error; error reduction from scaling is only ~6%. Five-agent effective capacity is just 1.04–1.32 people. On 31.5% of items, *all* models err in the same direction (cross-model bias correlation r = 0.63).
• Data honesty: 144 of 529 RealFP items (27.2%) had implausible reference labels; cleaning them raised absolute numbers but left all patterns unchanged.

Heterogeneity and interventions

• Mixed-family teams: a best-five heterogeneous pool raised accuracy +22.1 points over the member mean, but lost to its own strongest member in all 10 splits (Nitzan–Paroush: equal-weight voting is suboptimal with unequal abilities).
• Compensatory tasks are the exception: a five-model 7B–8B mix cut error 25.8% and beat the strongest single member (1.45 vs. 1.80 log error) — bias diversity is real information.
• Interventions mostly failed: different estimation templates, log-space calculation, units, constant tables — none reliably helped; logcalc *hurt*. Only structured scaffolds gave a marginal gain. RL-tuned agentic coding models showed almost no transfer to estimation. Program-of-Thought prompting helped individuals (~50% order-of-magnitude accuracy) but doesn't change team aggregation.

Takeaways

• Scaling payoff = task potential × aggregation mechanism's realization rate. Both factors must be nonzero.
• Disjunctive tasks: use reasoning-before-answering or replace voting with verifiers/reward models to recover the pass@N vs. voting gap.
• Compensatory tasks: you need *differently wrong* models (cross-family teams), not more sampling of the same one.
• Two kinds of "diversity" matter in opposite ways: answer-distribution diversity helps voting; bias-structure diversity helps averaging. Wasted money comes from confusing them.
• The paper's honest limitations: answer-first prompting, no reasoning-only control, one noisy benchmark for Fermi tasks, models ≤20B, and idealized conditional independence (real conformity effects can only make things worse).

> Galton's 800 fairgoers were accurate because each weighed the ox with independent, unbiased experience. Thirty agents sharing one set of weights and one corpus are thirty echoes — voting, they always elect themselves.

--- Reference: Fortuna C., Bertalanič B. "Multi-agent Scaling Across Disjunctive and Compensatory Tasks." arXiv:2609.31563. https://arxiv.org/abs/2609.31563