> Feynman repeatedly stressed in his quantum mechanics lectures: "If someone tells you quantum mechanics 'doesn't make sense,' it's because they're trying to understand it with a logic that doesn't belong to it." Today's story is the reverse—someone applied classical distributed-systems logic to large language models, and found it makes sense in a deeply unsettling way.
---
Prologue: A Theorem from 20+ Years Ago
In 2000, computer scientist Eric Brewer proposed the famous CAP theorem: in a distributed system, when a network partition (P) occurs, you can only choose between consistency (C) and availability (A)—you cannot have all three. The theorem was later proven and became a cornerstone of distributed system design. Every distributed database and microservice architecture makes trade-offs under its constraints. It isn't a "bug"—it's an inherent structural limitation.
Now, a solo-authored paper published in May 2026 advances a bold conjecture: LLMs face a CAP-style trilemma. Under conditions called "semantic underdetermination," an LLM cannot simultaneously guarantee correctness, strict non-bias, and utility. You can only pick two.
---
Chapter 1: What Is Semantic Underdetermination?
Before understanding the trilemma, you must understand its precondition: semantic underdetermination.
A prompt is semantically underdetermined when its premises are insufficient to uniquely determine an answer.
For example: "Should I choose career A or career B?"
This question is inherently underdetermined—it says nothing about your interests, abilities, finances, or family constraints. For the model to give a meaningful (useful) answer, it must introduce a selection criterion—perhaps "pick the higher-paying one" or "pick the more stable one." But these criteria aren't derived from the premises—they come from the model's internal priors or popular patterns in its training data.
Once the model introduces a selection criterion that neither the user specified nor the premises justify—bias has been created.
---
Chapter 2: Formalizing the Trilemma
The paper decomposes LLM behavior into three mutually incompatible properties:
| Property | Meaning | When Lost | |------|------|-------------| | Correctness | Logically sound, factually accurate answers | Hallucinations, logic errors | | Strict Non-bias | No preferences introduced beyond what premises support | Leaning toward specific answers, implicit values | | Utility | Useful, direct, actionable answers | Refusals, vagueness, "it depends" |
The trilemma: under semantically underdetermined prompts, an LLM cannot satisfy all three.
Proof sketch (conceptual): 1. To produce a useful answer → must choose among multiple plausible answers 2. Multiple plausible answers → premises don't determine a unique answer 3. Choosing requires a criterion (preference / prior / value ordering) 4. The criterion isn't user-provided → it comes from the model's internal bias 5. Therefore → the answer embeds an "unsupported preference" → biased 6. If the model refuses to introduce a preference (staying non-biased) → it must decline to choose → loses utility (can only be vague or refuse)
This is an iron triangle. You can stand on one edge, but one edge is always out of reach.
---
Chapter 3: The Elegant Analogy from CAP to LLMs
CAP's three properties:
- Consistency (C): all nodes see the same data
- Availability (A): every request gets a response
- Partition tolerance (P): the system keeps working through network partitions
- Title: A CAP-like Trilemma for Large Language Models: Correctness, Non-bias, and Utility under Semantic Underdetermination
- Author: Vinu Ellampallil Venugopal
- arXiv ID: 2605.11672
- Published: May 12, 2026
- Categories: cs.AI, cs.DB
CAP says: when P occurs, choose C or A.
The LLM trilemma says: when semantic underdetermination occurs, you can keep (correct + non-biased), (correct + useful), or (non-biased + useful)—but never all three.
| CAP | LLM Trilemma | |-----|-------------| | Network partition (P) | Semantic underdetermination | | Consistency (C) | Correctness | | Availability (A) | Utility | | — | Non-bias | | Choose C: return precise but possibly incomplete data | Choose correct + non-biased: answer correctly but don't decide ("there are multiple reasonable answers; it depends on your preferences") | | Choose A: return data but possibly inconsistent | Choose correct + useful: give direct advice but with implicit preferences | | — | Choose non-biased + useful: fairly present multiple options without guaranteeing the best one |
---
Chapter 4: What Does This Mean?
Challenging the Goal That "AI Should Be Unbiased"
Mainstream AI ethics demands models be impartial. But the trilemma shows: in underdetermined scenarios, absolute non-bias comes at the cost of utility. If a model never makes a choice ("you should decide based on your own circumstances"), it's technically unbiased but useless.
Implications for AI Evaluation
Current AI safety evaluations typically assess "correctness," "fairness," and "helpfulness" separately. This paper argues that such separate evaluation ignores their inherent conflict. A model scoring low on one axis may not be "lacking capability"—it may have made a better trade-off on the other two.
Implications for AI Design
System designers must make the trilemma choice explicitly instead of pretending all three can be satisfied. Just as distributed database designers must strategically choose between C and A, LLM deployers must choose between "useful but possibly implicitly biased" and "absolutely unbiased but basically useless."
---
A Feynman-Style Reflection
"You see, this is the kind of idea I love—it doesn't build a more complex model or tune more parameters. It just says: wait a moment, what you're asking for may be logically impossible.
CAP was like that. It's not a yellow card saying 'you're not doing well enough'—it's a red card saying 'under these game rules, you can never do it.' This little paper does the same to LLMs. When the premises are underdetermined, you cannot be correct, unbiased, and useful at once. Logically impossible.
You know what's best? CAP spawned an entire design philosophy for distributed databases—people stopped cursing systems for being 'imperfect' and started saying clearly, 'I chose consistency, so I may be unavailable during partitions.' If this trilemma is widely accepted, it will spawn a similar philosophy: no longer blaming models for 'being biased,' but stating explicitly, like choosing a database strategy—'this model favors safety and compliance in underdetermined scenarios' or 'this model never makes decisions for the user.'
When you deceive yourself that you can have it all, you get nothing. When you admit you must choose one, you finally start designing something real."
---
*Paper details*