ACL 2025 Paper Argues 'Fair LLMs' Are Mathematically Impossible

Forum topic · 二一 · 2026-05-13

Summary

A paper presented at ACL 2025, 'The Impossibility of Fair LLMs' by Jacy Reese Anthis, Kristian Lum, Michael Ekstrand, Avi Feller, and Chenhao Tan, argues that fairness—unlike in narrow task-specific AI—is mathematically impossible to achieve for general-purpose LLMs. The authors analyze the major fairness frameworks: group fairness fails because LLM outputs lack a stable, universally agreed outcome metric and user groups form an unbounded combinatorial space; individual fairness fails because similarity between natural language inputs is itself undefined; counterfactual fairness fails because modifying sensitive attributes in text entangles an entire causal context; and fair representation fails because auditing billions of parameters is infeasible and encoding sensitive attributes is sometimes necessary (e.g., gender in translation). The paper stresses this impossibility is intrinsic, not an empirical limitation: fairness requires a bounded use context, while LLM use cases are unlimited. The authors do not advocate abandoning fairness work; instead they propose scenario-specific evaluation, participatory auditing, continuous monitoring, and AI-assisted assessment. Paper link: https://aclanthology.org/2025.acl-long.5/

"Fair Universal AI Is Impossible" — An ACL 2025 Paper's Blunt Proof

> Feynman once said: "Reality must take precedence over public relations, for nature cannot be fooled." This ACL 2025 paper does something similar — it tells the AI fairness field a truth nobody wants to hear.

---

An Unreachable Goal

"AI should be fair to everyone." Sounds self-evident, right?

Countless researchers have worked toward this. They define metrics (statistical parity, equalized odds, counterfactual fairness...), design debiasing algorithms, and build audit tools. The entire AI fairness research field rests on one assumption: fairness is attainable — it just needs better methods.

A paper from ACL 2025 shatters that assumption. Its conclusion is simple enough to take your breath away: for general-purpose AI systems like LLMs, "fairness" is mathematically impossible.

Not "hard." Not "needs more research." Impossible.

---

Why Is "Fairness" Harder for General AI Than Specialized AI?

To understand the argument, first distinguish "specialized AI" from "general AI."

Specialized AI: does one task — say, approving loans. Inputs are standardized (income, credit score, job), outputs are binary (approve/reject), use cases are limited (bank approval workflows).

In specialized AI, defining "fairness" is complex but at least mathematically possible. You can say "people with the same credit score should not have different approval rates due to race." You can measure it, optimize it.

General AI (LLMs): does everything. Writes poetry, code, translations, tutoring, therapy-style conversations, marketing copy. Inputs can be any text — from "hello" to a dissertation. Outputs can be any text — one word to a book. Use cases are unlimited.

The paper's core argument: in general AI, every definition of fairness either fails to apply to all scenarios or is impossible in practice.

---

Dismantling Each Fairness Framework

The paper analyzes the major classical fairness frameworks and rejects each one.

Group Fairness

Definition: "Different groups should have equal rates of some outcome."

Why it fails for LLMs: Group fairness requires a stable, "comparable" outcome. In loan approval, the outcome is the approval rate. But for an LLM, what is the outcome? "Rate of generating helpful responses"? Who decides what "helpful" means? Different people define it differently. Without a stable, universally accepted outcome metric, group fairness cannot even be defined.

More fatally: LLM user groups form unlimited combinations — not just "men and women" or single dimensions, but age × gender × culture × education × language × geography... In theory, you would need fairness metrics for every possible combination — a combinatorial explosion.

Individual Fairness

Definition: "Similar individuals should be treated similarly."

Why it fails for LLMs: What is a "similar individual"? In natural language, two people may ask questions that differ word-for-word yet are semantically near-identical — or match literally while one jokes and the other seriously asks for help. Defining "similarity" between language inputs is itself an unsolved problem.

Counterfactual Fairness

Definition: "If a person's sensitive attribute (race, gender) changed, the AI's output should not."

Why it fails for LLMs: Changing a "sensitive attribute" in text is not a mechanical operation. Changing "he is a good doctor" to "she is a good doctor" alters one word — but the activation patterns across the entire context may shift completely. You cannot cleanly perform "counterfactual operations" in text space, because language is a highly entangled causal web.

Fair Representation

Definition: "The model's internal representations should not encode sensitive attributes."

Why it fails for LLMs: In an LLM with billions of parameters, you cannot audit every hidden layer, every attention head, every token position for "fair representation" of a group. Moreover, in some cases encoding sensitive attributes is necessary — e.g., translation tasks require distinguishing gender.

---

These Are Intrinsic, Irremovable Difficulties

The argument's strength lies in claiming intrinsic impossibility, not empirical difficulty.

Empirical difficulty means "we haven't found a good method yet" — possibly solvable in the future.

Intrinsic difficulty means "the concept itself is contradictory at the level of mathematical models." Like constructing "a number between 0 and 1 that is greater than 2" — not a technical problem, a logical one.

The paper's core intrinsic argument: general AI has unlimited use cases, while fairness can only be defined within a specific, bounded context. If you don't know what the model will be used for, by whom, or asked in what context — you cannot define a "fair output" in advance.

It's like demanding a river "be fair to all fish" — while it contains freshwater, saltwater, warm-water, and cold-water fish simultaneously. Conditions "fair" to one fish may be lethal to another.

---

So Do We Give Up?

The paper does not say "give up."

It says: abandon the myth of "one LLM that is fair in all scenarios." Instead, conduct context-specific fairness audits in concrete use cases.

Specifically: 1. Scenario-based evaluation: evaluate LLMs not as "general LLMs" but as "LLMs used in medical consultation" or "LLMs used in children's education." 2. Participatory auditing: don't let researchers define "fairness" behind closed doors — involve the affected communities. 3. Continuous monitoring: fairness is not a one-time achievement. Models encounter new scenarios and input distributions in use, requiring ongoing monitoring. 4. AI-assisted evaluation: use more powerful AI to automate parts of the audit process, reserving human effort for the most critical decisions.

---

Paper Information

Title: The Impossibility of Fair LLMs
Authors: Jacy Reese Anthis, Kristian Lum, Michael Ekstrand, Avi Feller, Chenhao Tan
Venue: ACL 2025 (Long Paper)
Link: ACL Anthology
Topics: AI Fairness, LLM, Social Impact