Beyond Consensus: Why the Reasoning Traces Discarded by Majority Voting Are Worth More Than the Consensus Itself

Imagine a fifth-grade classroom. The teacher asks: "What is 7×8?" Of 30 students, 20 say 54, and 10 say 56. Majority vote: 54. Correct answer: 56.

If you only count hands, you get the wrong answer. But if you read the scratch paper of the 10 minority students, you find they wrote out the correct vertical multiplication. The right answer is not in the votes. It is in the scratch paper that voting throws away.

In May 2026, a team from Bioscope AI turned this intuition into rigorous empirical findings in a paper they frame as the Aggregation Paradox: when multiple AI agents solve the same problem, standard majority voting compresses all reasoning into a single answer letter — and discards it. If you instead preserve the reasoning traces and let another LLM read them, it can reconstruct the correct answer from the minority's scratch work. Even when all agents unanimously agree on a wrong answer, reading the traces can flip the result.

| Item | Detail | |------|--------| | Paper | Beyond Consensus: Trace-Level Synthesis in Mixture of Agents | | Authors | Shreyas Fadnavis, Praitayini Kanakaraj, Felix Wyss | | Institution | Bioscope AI | | arXiv ID | 2605.29116 | | Submitted | May 27, 2026 | | Category | cs.AI | | Core finding | Majority voting systematically discards correct intermediate steps in minority reasoning traces; using only semantically faithful input perturbations of a single model, trace-level synthesis beats majority voting over heterogeneous model pools on all five benchmarks. The unit of aggregation should be reasoning traces, not answers. |

1. 🗳️ The Voting Ceiling

Self-Consistency — sample the same question N times and take the modal answer — is standard practice for LLM reasoning. But it has a hard flaw.

The paper first shows: perturbation diversity does not raise the voting ceiling.

Intuitively, perturbing the prompt (rewording, restyling, changing perspective) should produce reasoning paths whose errors are independent, so voting should reduce error — the Condorcet Jury Theorem logic.

The paper tested this directly by measuring mean pairwise error correlation \(\bar{\rho}\) under perturbation diversity vs. i.i.d. sampling. The two were nearly identical: 0.633 vs 0.603 on GPQA-Diamond, with fully overlapping 95% confidence intervals. Error correlation did not drop under perturbation. Perturbations create different reasoning paths, but their errors are not independent — they share a fixed set of tendencies at a deeper level.

So improving voting accuracy through more perturbation hits a fundamental limit. Voting only sees answer letters, and error patterns across perturbed paths are highly correlated at the letter level. But this doesn't mean perturbation is useless — it just doesn't work through voting.

2. 📜 The Goldmine in the Scratch Paper

The decisive experiment: generate 5 reasoning traces from the same perturbations, then either (a) majority-vote the answers, or (b) have an LLM aggregator read all 5 full reasoning traces.

On GPQA-Diamond (198 PhD-level science questions), reading full traces produced 12 beneficial flips vs. only 5 harmful flips over majority voting — a net gain of +3.5 percentage points, a benefit ratio of 2.4×.

On BBH (296 structured reasoning questions), the benefit ratio was 3.2× — 16 beneficial vs. 5 harmful flips, net +3.7 points (p=0.016).

More strikingly, a substantial share of these gains occurred when all agents unanimously agreed on a wrong answer. The standard practice is to skip aggregation when consensus is unanimous to save a call. That logic is wrong. Unanimity ≠ correctness. Reading the traces can flip back even unanimous errors.

Why? Different traces contain different correct intermediate steps. Trace 1 gets step A right and step B wrong; trace 2 gets B right and A wrong. Voting only sees the final letter — both errors count as "answer C," the majority. But an aggregator reading full traces can extract the correct step A from trace 1 and the correct step B from trace 2, assembling a fully correct solution.

The paper calls this trace-level complementarity. The gain comes from different reasoning paths naturally containing different correct fragments — fragments that voting washes away.

3. 🔧 SC-MoA: A Three-Stage Algorithm

The findings are engineered into a deployable algorithm — SC-MoA (Self-Consistent Mixture of Agents):

Step 1: Perturbation diversity. The perturbation content doesn't matter: handcrafted personas, SPUQ semantic-preserving perturbations, and GPT-generated strategies are statistically indistinguishable on GPQA-Diamond (72.7%, 72.7%, 70.2%, McNemar p>0.25). As long as perturbations preserve semantics, any method works — so SC-MoA uses the cheapest option: one LLM call producing N faithful rewrites, verified for keyword integrity.

Step 2: Anchored refinement. After N agents generate traces, cluster by answer and select the majority answer \(a^*\). The key move: lock the majority, revise only the minority, giving minority agents the majority's full traces as reference. The paper proves this transforms unconstrained debate (which can degrade — shown to be a martingale by Choi et al. 2025) into a submartingale: post-refinement voting accuracy cannot fall below the pre-refinement level. Experiments confirm — across 867 questions, refinement never reduced consensus.

Step 3: Aggregate unconditionally, never gate. Standard practice skips aggregation above a consensus threshold. SC-MoA always runs the aggregator, because even 100% consensus can hide correct answers in the traces. On the LCB-Hard coding benchmark: 69% of questions reached unanimity, yet always-aggregating beat consensus gating by 5.1 points — because pass-based clustering is unfaithful (48.3% of "unanimous" clusters concealed disagreement on hidden tests).

4. 📊 Sweeping Five Benchmarks

On gpt-oss-120b, SC-MoA achieved the top point estimate on all five benchmarks:

| Benchmark | Self-Consistency | MoA | GoA | SC-MoA | Gain | |------|---------|-----|-----|--------|------| | BBH | 80.4 | 69.9 | 82.8 | 86.5 | +3.7 | | MMLU-ML | 90.2 | 85.7 | 89.3 | 92.0 | +1.8 | | GPQA | 70.7 | 67.7 | 72.7 | 73.2 | +0.5 | | AIME | 85.6 | 87.8 | 77.8 | 91.1 | +1.1 | | LCB-Hard | 57.3 | 57.3 | 24.6 | 62.6 | +5.3 |

Key comparisons:

• Single model + perturbation beats heterogeneous model pools. Five persona-perturbations of gpt-oss-120b outperformed mixing multiple different models. Model diversity is not required.
• Minimal configuration (N=4, k=1, ~5 calls) beats 10-call self-consistency on GPQA (74.7% vs 70.7%). Half the compute, higher accuracy.
• Refinement itself contributes ≤1.8 points — aggregation is the active ingredient.

5. 🔬 Tracing the Active Ingredient

Ablations pinpoint where the gains come from:

• Replacing full traces with answers only reverts accuracy to the voting baseline — the aggregator extracts information from reasoning content, not answer letters.
• Showing the aggregator majority traces only reverts to baseline. Minority traces only restores the full gain (73.2% vs 72.2% for full traces). The minority's reasoning content is the real goldmine.
• Shuffling reasoning steps within traces leaves accuracy unchanged. The aggregator treats traces as a bag of evidence, not an ordered chain — it only cares whether correct fragments exist.
• An information-ladder ablation with no traces at all drops to 55.6% (-14.1 points, p<10⁻⁴). Zero reasoning = zero gain.

In one sentence: minority reasoning traces are the only fuel that lets the aggregator exceed the voting ceiling.

6. 🎯 Free Calibration

A bonus: since SC-MoA naturally produces a consensus signal across traces, this signal serves as an answer-confidence proxy with no extra calibration training. On AIME, answering only the most-consensual 54% of questions raised accuracy from 91.1% to 100% (AUROC=0.85). On GPQA, the calibration error (ECE) between consensus and actual correctness was just 0.064 — near-perfect calibration.

This enables a deployment win-win: high consensus → auto-accept; low consensus → flag for human review. No extra compute required.

7. 📝 Honest Blind Spots

What's clear:

• The aggregation paradox claim has clean causal structure — same proposals, only the aggregation method varies.
• The "perturbation content doesn't matter" finding was tested across three methods (handcrafted, SPUQ, automatic) — all McNemar tests non-significant.
• The submartingale guarantee for anchored refinement was empirically validated — zero degradation across 867 questions.
• The minority-trace conclusion is supported by multiple ablations (answers-only, majority-only, minority-only, step shuffling).

What's unclear:

• All experiments ran on gpt-oss-120b. Cross-model validation (Fig. 6a shows 6 model families) is not fully detailed in the main text; effect sizes on other families lack systematic reporting.
• The "perturbation content doesn't matter" conclusion is restricted to semantically faithful perturbations. Adversarial, cross-lingual, and format-level perturbations were not tested, and the paper explicitly warns against extrapolation.
• What does the unfaithfulness of pass-based clustering on code tasks mean? The LCB-Hard "always aggregate" choice sidesteps but does not solve the problem that 48.3% of consensus clusters hid disagreement on hidden tests.
• Do beneficial flips converge to zero on easier questions? Gains were larger on the hardest benchmark (GPQA-Diamond), fitting the intuition, but the difficulty-benefit function was not fully mapped.
• How deployment realities (prompt drift, latency, model versioning) affect perturbation diversity and trace complementarity in production systems is unknown.

8. ⚖️ The Unit of Aggregation

This paper accomplishes something small enough to be overlooked and big enough to matter: it moves the focus of LLM aggregation research from answers to reasoning.

Since Self-Consistency (2023), the field's default assumption has been that the answer letter is the useful aggregation signal and reasoning is disposable scaffolding. The paper proves this assumption wrong with rigorous data. Reasoning traces are not scaffolding — they are raw material that can be decomposed, recombined, and integrated across chains. Minority traces contain correct fragments missing from majority reasoning, and voting naturally throws those fragments away.

SC-MoA is the operational recipe: cheapest perturbations for diversity, mathematically guaranteed anchored refinement to protect the majority, then always perform trace-level synthesis and never gate on consensus.

From an industry perspective, the message is radical pragmatism: one model plus perturbation beats mixing many models; half the calls beat more calls. No new models, no new data, no expert prompt design — just a cheap rewriter and one extra aggregation call.

From a scientific perspective, it asks a deeper question: if so many correct answers live in the scratch paper voting throws away, what else have our evaluation methodologies been discarding?

> | Item | Detail | > |------|--------| > | Paper | Beyond Consensus: Trace-Level Synthesis in Mixture of Agents | > | Authors | Shreyas Fadnavis, Praitayini Kanakaraj, Felix Wyss (Bioscope AI) | > | arXiv ID | 2605.29116 | > | Category | cs.AI | > | Core contributions | (1) The aggregation paradox: trace-level synthesis beats majority voting even under unanimity, with asymmetric (beneficial-dominant) flips; (2) Perturbation diversity does not reduce error correlation — the voting ceiling comes from data structure, not insufficient perturbation; (3) SC-MoA: perturbation diversity + anchored refinement (submartingale guarantee) + unconditional aggregation; (4) Within the semantically faithful family, perturbation methods are statistically indistinguishable; (5) Ablations isolate minority traces as the sole active ingredient; (6) A single perturbed model beats heterogeneous model pools | > | Key limitations | All core experiments on one model (gpt-oss-120b); incomplete cross-model validation; the perturbation finding restricted to semantically faithful families; unfaithful code clustering unresolved; difficulty-benefit function unmapped; production robustness unknown |

References: 1. Fadnavis, Kanakaraj & Wyss, "Beyond Consensus: Trace-Level Synthesis in Mixture of Agents", arXiv:2605.29116, 2026. 2. Wang et al., "Self-Consistency Improves Chain of Thought Reasoning in Language Models", ICLR 2023. 3. Wang et al., "Mixture-of-Agents Enhances Large Language Model Capabilities", arXiv:2406.04692, 2025. 4. Choi et al., "Debate or Vote: Which Yields Better Decisions in Multi-Agent Large Language Models?", NeurIPS 2025 (Spotlight). 5. Gao et al., "SPUQ: Perturbation-Based Uncertainty Quantification for Large Language Models", EACL 2024.