Why 67-Model Ensembles Lose to the Single Best LLM: The Co-Failure Ceiling
The Scenario: An AI Architect's Dilemma in 2026
Suppose you are the AI architect at a company in 2026. The model market has become stock-market-like: GPT-5.5, Claude Opus 4.8, Gemini 3.1 Pro, Grok-4.3, DeepSeek V4, Qwen3.7-Max, Kimi K2.7, and more—67 frontier models from 21 vendors, priced from $30 per million tokens down to $0.10, with every new generation making the previous one obsolete.
Your boss asks: can we combine these models to be more accurate than any single one?
You search the literature and find that everyone uses one metric to decide: pairwise error correlation rho. Low rho means models fail on different queries, so combining them should help. It is diversification from portfolio theory.
You train a router, try majority voting, cascades, and Mixture-of-Agents. Result: the router captures almost no gain. An LLM-as-router (GPT-5-mini) picks the single strongest model on 100% of queries. You swap in gradient-boosted trees, multi-class predictors, and an LLM that reads each model's description first. Four routers, none beats the strongest single model.
Where is the flaw?
Core Finding: Rho Is the Wrong Metric; Beta Is the Right One
Josef Chen (KAIKAKU), in "When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Models," gives the answer. The entire field has been looking at the wrong number.
The central insight is captured by an inequality:
> The accuracy ceiling of any selection strategy (routing, cascading, voting) equals 1 minus beta
where beta = the probability that every model fails on the same query.
The inequality is obvious: if all models are wrong, any strategy that picks one of their answers is wrong too. But this obvious fact has been ignored. The paper sharpens it further: rho is mathematically incapable of identifying beta. Two error distributions can share identical marginal error rates and identical pairwise rho while having completely different beta values. Rho is blind to beta.
The $0 Certificate: Know Before You Deploy
The most practical contribution is a "free certificate." Before training any router, before labeling data, before any engineering investment, you do only one thing:
Count how many queries K out of n every model got wrong.
Apply the Clopper-Pearson formula to get a lower confidence bound on beta. Subtract the strongest single model's accuracy a_sb from 1 minus this bound. You now have a confident upper bound on the maximum gain any selection strategy can deliver over the strongest single model.
If that number is below your orchestration overhead (router training, extra latency, maintenance), do not bother. No approach can pay for itself.
This is a $0 test. It requires only an already-labeled test set and each model's answers. The tool is open-sourced as beta_certificate.py: input the all-wrong count K/n and the strongest single model's accuracy, output a Clopper-Pearson-certified upper bound on maximum gain.
Why Rho Under-Predicts Beta: The Ghost of 2008 CDOs
You might wonder whether beta can be back-calculated from rho, for example via a single-factor Gaussian copula. The paper says no, and shows the under-prediction is systematic.
At a 67-model scale, even a correctly calibrated tetrachoric single-factor model under-predicts beta by about 2.5x on open math (90% CI 1.7–3.4, k=17). The model tells you beta=0.023 when beta=0.052. Under-prediction worsens as the pool grows.
If this sounds familiar, it is because it is the mathematical structure of the 2008 subprime crisis.
Gaussian copula models priced CDOs by inferring "all assets default together" probability from pairwise default correlation. In normal times the estimates looked fine. In tail events, assets crashed together—tail correlation far exceeded pairwise correlation. The models systematically under-priced tail risk.
The paper explicitly cites this analogy: the "body-vs-tail base-correlation smile of Gaussian-copula portfolio-credit (CDO) models." Same math, same trap, relocated from subprime mortgages to language models.
Format Decides Everything: Beta Goes From 0 to 0.127 on the Same Questions
The cleanest experiment in the paper is content-controlled. The authors took GPQA-Diamond (graduate-level science) and ran two tests:
- Multiple-choice version: beta near 0 (almost no queries where every model is wrong)
- Open-ended version (options removed): beta = 0.127 (about 8% of queries all models fail)
- Setting: open math, code generation, open-ended science
- Trait: beta > 0, real co-failure on some queries
- Consequence: the 1 minus beta ceiling is real; no strategy can break it
- Data: MATH-500 beta=0.052, code contest beta=0.079, open GPQA beta=0.127
- Setting: multiple-choice science (GPQA-Diamond MCQ)
- Trait: beta near 0, ceiling is high (1 minus 0 = 1), oracle gain G = 0.154 is large
- Consequence: the ceiling is not the problem; learned routers fail to realize the gain
- Data: GPQA MCQ oracle reaches 1.000; learned routers achieve near-zero gain
- Self-MoA (multiple samples from the strongest model, intra-rho=0.80) vs heterogeneous fusion (6 accuracy-matched models, inter-rho=0.42)
- At a fair k=3 (3 independent samples per side), heterogeneous fusion beats Self-MoA by +0.027
- Across 60 resamples, 100% of deltas are positive (direction robust, magnitude sensitive to partition)
- MATH-500: beta=0.052, rho under-predicts beta by 2.5x
- Code contests: beta=0.079, rho under-predicts beta by 3.1x
- GPQA MCQ: beta near 0 (realizability-bound)
- GPQA open-ended: beta=0.127 (format-flip regime)
- Learned routers capture approximately 0% of G
- LLM-as-router: 100% pick the strongest single model
Same questions, same models, only the answer format changed. Average accuracy fell from 0.66 to 0.51; the strongest single model fell from 0.91 to 0.77.
Co-failure is not driven by subject matter; it is driven by answer-format openness. Multiple choice gives models a strong prior (25% chance by guessing), driving beta toward zero. Remove the options and models must generate answers themselves. Genuine blind spots surface—and they surface as all-models-wrong.
Practical consequence: a routing strategy that validates on multiple-choice benchmarks will not transfer to real open-ended production settings.
Two Ceilings: Two Regimes Rho Cannot Distinguish
The paper identifies two distinct failure regimes that rho cannot separate.
Regime 1: Ceiling-bound
Regime 2: Realizability-bound
The practical message is the same in both regimes: without strong per-query routing signal, combining models rarely beats the strongest single model. But the reason is entirely different in each regime—mathematical ceiling in one, engineering realizability in the other. Rho tells you neither.
The 67-Model Experiment: Four Routers, Total Defeat
The paper tests four routers on a 15-model pool:
| Router | Fraction of G captured | |--------|------------------------| | TF-IDF + logistic regression | 9% (CI [-0.67, 0.50], not significant) | | Gradient-boosted trees (word + char features) | -9% (negative) | | Best-model multi-class predictor | -127% (routing makes it worse) | | LLM-as-router (GPT-5-mini) | 0% (100% pick strongest single) |
Four routers, simple to complex, none significantly beats the strongest single model.
The explanation is sharp: "prompt carries little signal about which model will be the one that is right when the frontier disagrees." When frontier models disagree, the query text itself contains almost no signal about which model will be right.
This directly contradicts the optimism of "More Agents Is All You Need"–style work. The paper shows that with quality mismatch, majority voting gains are negative: -0.10 on hard questions, -0.02 on saturated ones. Mixing unequal models lets the weaker majority outvote the stronger minority.
But Diversity Helps When Quality Matches
The paper does not entirely reject diversity. Under quality-matched conditions (similar accuracy), low rho does buy gain:
This validates the core prediction of diversification: under quality matching, lower pairwise correlation does buy greater diversifiable gain.
But the precondition is quality matching. In real production, the models you actually have are not quality-matched, so the main practical conclusion still holds: naive diversity is a liability.
Engineering Takeaways: Three Practical Recommendations
1. Run the $0 Certificate Before Deployment
Before any router training, count all-wrong K/n and runbeta_certificate.py. If the certificate's maximum gain is below your orchestration cost, stop.2. Monitor Beta, Not rho
Rho is misleading. Two model pools can have identical rho and completely different beta. Measure beta directly: it needs only a labeled test set, no pairwise computation.
3. Format Matters More Than Subject
Is your benchmark multiple-choice or open-ended? That question matters more than "math vs science vs code." Beta near 0 in MCQ will make your orchestration look effective; transfer to open-ended settings and the ceiling collapses.
A Deeper Reflection: AI's Shared Blind Spots
The deeper question is what beta measures: the shared blind spot across all models. On open math, 67 models from 21 vendors all fail on 5.2% of questions. These are not "hard" questions in some abstract sense; they are questions where every model's training data, architecture, and alignment recipe together produce a common blind spot.
The structural parallel to 2008 goes beyond math. CDOs collapsed because every rating agency, every bank, every investor shared the same assumption (house prices do not fall nationally), so they all under-priced tail risk. Model co-failure may reflect shared assumptions—shared data contamination, shared alignment preferences, shared safety training producing shared refusal patterns.
The paper warns: as model training grows more homogeneous (everyone uses RLHF, similar data recipes), beta may rise. More models does not mean more diversity if their failure modes converge; it means more redundant homogeneity.
The closing line is worth remembering:
> "On open-ended tasks the best models increasingly fail alike, so the lever is failure-mode dispersion and market churn, not peak capability or model count."
On open-ended tasks, the best models increasingly fail together. The real lever is not "stronger models" or "more models" but finding models with different failure modes—and waiting for market churn to deliver new heterogeneity.
This matches portfolio theory exactly: diversification gains do not come from the number of assets, but from low inter-asset correlation. When all assets crash together, diversification fails—because beta, not rho, is decisive.
---
Author: Josef Chen (KAIKAKU)
Core tool: beta_certificate.py — a $0 pre-deployment certificate; input the all-wrong count, output a Clopper-Pearson upper bound on any combination strategy's maximum gain
Experimental scale: 67 frontier models, 21 vendors, approximately $270 reported experimental cost
Key data: