English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Why 67-Model Voting Can't Beat a Single Strongest Model: The Overlooked Co-Failure Ceiling

Forum topic · ✨步子哥 · 2026-06-26

Summary

A paper by Josef Chen (KAIKAKU), 'When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Models', argues that the LLM-orchestration field optimizes the wrong metric. Instead of pairwise error correlation (rho), the binding constraint is beta: the probability that all models fail simultaneously on the same query, since no selection strategy can exceed accuracy 1 - beta. Across 67 frontier models from 21 vendors, the paper shows rho mathematically cannot identify beta, and single-factor copula models calibrated on rho systematically underestimate beta by about 2.5x on open math tasks — a structure analogous to the 2008 CDO/Gaussian-copula crisis. Four routers (TF-IDF, gradient boosting, multiclass predictor, GPT-5-mini LLM-router) captured roughly zero of the oracle gain, with the LLM-router always picking the single strongest model. A 'free' pre-deployment certificate based on the all-wrong count (released as beta_certificate.py) bounds any strategy's maximum gain before any engineering investment. Experiments also show answer format drives co-failure: on GPQA-Diamond, beta is near zero in multiple-choice form but 0.127 in open-response form.

Why 67-Model Voting Can't Beat a Single Strongest Model: The Overlooked Co-Failure Ceiling

The Scenario: An AI Architect's Dilemma in 2026

Imagine you are an AI architect at a company. By 2026 the model market resembles a stock market: GPT-5.5, Claude Opus 4.8, Gemini 3.1 Pro, Grok-4.3, DeepSeek V4, Qwen3.7-Max, Kimi K2.7... 67 frontier models from 21 vendors, priced from $30/M tokens down to $0.1/M tokens, with new generations devaluing old ones every few months.

Your boss asks: "Can we combine these models to be more accurate than any single one?"

The literature says the key metric is pairwise error correlation ρ — how correlated models' errors are. Low ρ means models fail in different places, so combining them should help, just like diversification in portfolio theory.

It sounds reasonable. So you train a router to pick the best model per query. You try majority voting, cascades, Mixture-of-Agents... and find that routers capture almost no gain; an LLM-as-router (GPT-5-mini) picks the single strongest model on 100% of queries.

You suspect the router is too weak. So you swap in gradient boosting, multiclass predictors, even an LLM that reads all models' strength descriptions. Four routers, none beats the single strongest model.

What's wrong?

The Core Finding: ρ Is the Wrong Metric — β Is the Right One

Josef Chen (KAIKAKU) answers in the paper *"When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Models"*: the field has been looking at the wrong number.

The key insight fits in one inequality:

> The accuracy ceiling of any selection strategy (routing, voting, cascading) = 1 − β

where β = the probability that all models fail simultaneously on the same query.

If every model is wrong, any strategy that selects one of their answers must also be wrong. Yet the field optimized ρ while nobody measured β.

The paper proves something sharper: ρ mathematically cannot identify β. There exist error distributions with identical marginal error rates and identical pairwise correlations ρ but completely different β. ρ is "blind" to β.

The $0 Certificate: Know Before You Deploy

The paper's most practical contribution is a free certificate. Before training any router, before paying for labels, before any engineering investment, you only need to do one thing:

Count how many of n queries every model gets wrong; call it K.

Then compute a Clopper-Pearson confidence lower bound for β. Plug into 1 − β_lo − a_sb (a_sb = the single best model's accuracy) and you get a confidence upper bound: the maximum gain any selection strategy can achieve over the single strongest model.

If that number is below your orchestration overhead (router training cost, extra latency, maintenance), don't bother — no strategy can pay off.

It's a $0 test: no router training, no pairwise ρ, no complex analysis. Just a labeled test set and the models' answers, released as beta_certificate.py.

Why ρ "Underestimates" β — The Ghost of 2008 CDOs

Could you infer β from ρ, say with a single-factor Gaussian copula? The paper says: no — and it systematically underestimates.

At the 67-model market scale, even a correctly calibrated tetrachoric single-factor model predicts β about 2.5x lower than observed on open math questions (90% CI 1.7–3.4, k=17). The model says β = 0.023; reality is β = 0.052. The underestimate worsens as the pool grows.

If this sounds familiar, it's the mathematical structure of the 2008 subprime crisis. Gaussian copula models priced CDOs using pairwise default correlations to infer joint default probability, and systematically underestimated tail risk because assets crash together in extremes. The paper explicitly cites the analogy: "body-vs-tail base-correlation smile of Gaussian-copula portfolio-credit (CDO) models". The difference: the CDO miscalibration wasn't visible before the crisis; this paper points out the AI-orchestration version before it bites.

Format Determines Everything: Same Questions, β from 0 to 0.127

The paper's most elegant experiment is a content-controlled one. Using the same GPQA-Diamond questions, tested twice:

  • Multiple-choice version: β ≈ 0 (almost no all-model failures)
  • Open-response version (options removed): β = 0.127 (≈8% of questions all models fail)
  • Same questions, same models, only the answer format changed. Mean accuracy dropped from 0.66 to 0.51; the best model fell from 0.91 to 0.77.

    Co-failure is determined not by subject content but by the "openness" of the answer format. Multiple choice gives models a strong prior — guessing yields 1/4 accuracy even without knowledge — suppressing β to near zero. Remove the options and true gaps are exposed — and exposed collectively.

    Practical implication: an orchestration strategy validated on multiple-choice benchmarks may fail in real open-ended production settings, where β is much higher and the ceiling much lower.

    Two Kinds of Ceilings That ρ Cannot Distinguish

    Ceiling-bound

  • Setting: open math, code generation, open science questions
  • Feature: β > 0; all models fail together on some queries
  • Consequence: the 1 − β ceiling is real and insurmountable
  • Data: MATH-500 β=0.052; code competitions β=0.079; open GPQA β=0.127
  • Realizability-bound

  • Setting: multiple-choice science questions (GPQA-Diamond MCQ)
  • Feature: β ≈ 0, high ceiling, large oracle gain G=0.154
  • Consequence: the ceiling isn't the problem — deployable routers can't capture the gain
  • Data: oracle reaches 1.000 on GPQA MCQ, but learned routers achieve near-zero gain
  • Both imply the same practice: without strong query-level routing signals, model combinations rarely beat the single strongest model — but for entirely different reasons, and ρ can't tell you which regime you're in.

    Four Routers, Total Failure

    On a 15-model pool, the paper tested four routers:

    | Router | Fraction of G captured | |--------|------------------------| | TF-IDF + logistic regression | 9% (CI [-0.67, 0.50], not significant) | | Gradient boosting (word+char features) | -9% (negative) | | Multiclass best-model predictor | -127% (worse the more it routes) | | LLM-as-router (GPT-5-mini) | 0% (picks the single strongest 100%) |

    Why? The paper's explanation: "prompt carries little signal about which model will be the one that is right when the frontier disagrees" — when frontier models disagree, the query text itself barely signals which model will be right.

    This directly contradicts optimistic results like "More Agents Is All You Need": with quality mismatch, majority voting gains are negative (-0.10 on hard sets, -0.02 on saturated sets). Mixing unequal models lets weak models vote down strong ones.

    But With Quality Matching, Diversity Does Help

    The paper doesn't reject diversity outright. With quality-matched models (similar accuracy), low ρ does yield gains:

  • Self-MoA (multiple samples from one strongest model, ρ_intra=0.80) vs. heterogeneous ensembles (6 accuracy-matched models, ρ_inter=0.42)
  • At k=3 (fair comparison), heterogeneous fusion beats Self-MoA by +0.027
  • 100% of 60 resamples positive (direction robust, magnitude partition-dependent)
  • This validates the diversification theorem: with quality matched, lower pairwise correlation buys larger diversifiable gains. But the premise — quality matching — rarely holds in real settings. So the practical conclusion remains: naive diversity is a liability.

    Engineering Takeaways

    1. Run the $0 certificate before deployment. Count the all-wrong rate K/n and run beta_certificate.py. If the certified max gain is below orchestration overhead, abandon the project. 2. Monitor β, not ρ. Two model pools can share the same ρ with wildly different β. β only requires a labeled test set. 3. Watch format, not subject. Whether your benchmark is multiple-choice or open-response matters more than "math vs. science vs. code". MCQ's β≈0 makes orchestration look effective; it won't transfer to open settings.

    Deeper Reflection: AI's "Shared Blind Spots"

    β measures something philosophical: the blind spot shared by all models. On open math questions, 67 models from 21 vendors share failure on 5.2% of questions. These aren't just "hard" — they are blind spots jointly produced by shared training data, architectures, and alignment approaches.

    The CDO isomorphism is more than mathematical. The 2008 crisis happened because every rating agency, bank, and investor shared the same assumption (home prices can't fall nationwide), so all underestimated tail risk. Models' shared blind spots likewise stem from shared assumptions — possibly common data contamination, common alignment preferences, common refusal patterns from safety training.

    The paper implies a warning: as training becomes more homogeneous (everyone uses RLHF, similar data recipes), β may rise. More models doesn't mean more diversity — if failure modes converge, adding models just adds homogeneous redundancy.

    The paper's closing line is worth remembering:

    > "On open-ended tasks the best models increasingly fail alike, so the lever is failure-mode dispersion and market churn, not peak capability or model count."

    On open-ended tasks, the best models increasingly fail together. The real lever is not "a stronger model" or "more models" but "models with different failure modes" — and waiting for market churn to bring new heterogeneity.

    This matches portfolio theory: diversification gains come not from asset count but from low correlation — and when all assets crash together, diversification fails, because β, not ρ, is decisive.

    ---

    Paper: When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Models

    Author: Josef Chen (KAIKAKU)

    Core tool: beta_certificate.py — a $0 pre-deployment certificate; input the all-wrong count to get the max-gain upper bound for any combination strategy

    Scale: 67 frontier models, 21 vendors, ≈$270 reported experiment cost

    Key numbers:

  • MATH-500: β=0.052, ρ underestimates β by 2.5×
  • Code competitions: β=0.079, ρ underestimates β by 3.1×
  • GPQA MCQ: β≈0 (realizability-bound)
  • GPQA open-response: β=0.127 (format-flip regime)
  • Learned routers' captured gain: ≈0%
  • LLM-as-router: picks the single strongest model 100% of the time

Tags

#ai-orchestration#model-ensembling#co-failure-ceiling#llm-routing#portfolio-theory#cdo-crisis-analogy#benchmark-evaluation#beta-certificate

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178208157