"Fair General-Purpose AI Is Impossible": An ACL 2025 Paper's Provocative Proof

> Feynman once said: "Reality must take precedence over public relations. Nature cannot be fooled." This ACL 2025 paper does something similar — it tells the AI fairness field a truth nobody wants to hear.

Introduction: An Unreachable Goal

"AI should be fair to everyone." Sounds self-evident, right?

Countless researchers have worked toward this. They define metrics (statistical parity, equal opportunity, counterfactual fairness...), design debiasing algorithms, and build auditing tools. The entire AI fairness research field rests on one assumption: fairness is attainable — it just needs better methods.

An ACL 2025 paper shatters that assumption. Its conclusion is simple enough to take your breath away: for general-purpose AI systems like LLMs, "fairness" is mathematically impossible.

Not "hard." Not "needs more research." Impossible.

Chapter 1: Why Is "Fairness" Harder for General AI?

The argument starts by distinguishing "specialized AI" from "general AI."

Specialized AI: does one task — say, loan approval. Inputs are standardized (income, credit score, employment), outputs are binary (approve/reject), use contexts are limited (bank workflows).

In specialized AI, defining "fairness" is complex but mathematically possible. You can define "people with the same credit score shouldn't have different approval rates by race." You can measure it and optimize it.

General AI (LLMs): can do everything. Write poetry, code, translate, tutor, play therapist, generate marketing copy. Inputs can be any text; outputs can be any text; use cases are unlimited.

The paper's core claim: in general AI, every definition of fairness either fails to apply across all scenarios or is impossible to realize in practice.

Chapter 2: Why Every Fairness Framework Fails

Group Fairness

Definition: "Different groups should have equal rates of some outcome."

Why it fails for LLMs: Group fairness needs a stable, comparable outcome. In lending, it's "approval rate." But what is the outcome for an LLM? "Rate of generating helpful responses"? Who decides what's "helpful"? Different people define it differently. Without a universally agreed outcome measure, group fairness cannot even be defined.

Worse: the LLM user population is an unbounded combination of attributes — age × gender × culture × education × language × geography... You'd need fairness metrics for every possible combination — a combinatorial explosion.

Individual Fairness

Definition: "Similar individuals should be treated similarly."

Why it fails: What is a "similar individual"? In natural language, two questions can be lexically different yet semantically near-identical — or lexically similar while one is a joke and the other a sincere plea for help. Defining similarity between language inputs is itself an unsolved problem.

Counterfactual Fairness

Definition: "If a person's sensitive attribute (race, gender) changed, the AI's output should not."

Why it fails: Changing a "sensitive attribute" in text is not a mechanical operation. Change "he is a good doctor" to "she is a good doctor" — one word. But the model's activation patterns across the whole context may shift entirely. You cannot cleanly perform counterfactual operations in text space; language is a highly entangled causal web.

Fair Representation

Definition: "A model's internal representations should not encode sensitive attributes."

Why it fails: In a multi-billion-parameter LLM, you cannot audit every hidden layer, attention head, and token position for "fair representation." And in some cases encoding sensitive attributes is necessary — e.g., translation tasks that require distinguishing gender.

Chapter 3: These Are Intrinsic, Irremovable Difficulties

The paper's force lies in claiming intrinsic impossibility, not empirical difficulty.

Empirical difficulty says "we haven't found a good method yet" — possibly solvable in the future.

Intrinsic difficulty says "the concept itself is contradictory at the level of the mathematical model" — like constructing a number that is both between 0 and 1 and greater than 2. Not a technical problem; a logical one.

The core intrinsic argument: general AI's use cases are unlimited, while fairness can only be defined within a specific, bounded scenario. If you don't know what the model will be used for, by whom, or in what context — you cannot define a "fair output" in advance.

It's like demanding a river "be fair to all fish" — while it holds freshwater, saltwater, warm-water, and cold-water fish simultaneously. Conditions fair to one species may be lethal to another.

Chapter 4: So Do We Give Up?

The paper does not say "give up."

It says: abandon the myth of "one LLM fair to all scenarios." Instead, conduct context-specific fairness audits.

Concretely: 1. Scenario-based evaluation: audit LLMs in specific uses — "LLMs used in medical consultation," "LLMs used in children's education" — not "general LLMs." 2. Participatory auditing: don't let researchers define "fairness" behind closed doors; involve affected communities. 3. Continuous monitoring: fairness is not a one-time milestone. Models encounter new scenarios and input distributions in deployment, requiring ongoing monitoring. 4. AI-assisted evaluation: use stronger AI to automate parts of auditing, focusing human judgment on the most critical decisions.

A Feynman-Style Afterword

Feynman spent his life fighting people who "pretend to know the answer." He might say:

"This paper does something extremely important and rare in academia — it says 'we may not be able to reach the goal.' Not out of pessimism, but out of mathematical honesty.

People love to say 'let's build a fair AI.' Ask them what 'fair' means, and they start stammering. Equal accuracy across groups? Similar answers to similar questions? Then: what is 'equal accuracy,' what is a 'similar question' — in general AI, these concepts cannot even be defined.

This isn't deep math. It's a simple combinatorial explosion: use cases are infinite → fairness is defined differently per scenario → no definition covers all scenarios. Like building a ship 'comfortable for all passengers' — some get seasick and need calm, others enjoy the waves.

But they aren't nihilists. They say: stop chasing the phantom of 'universal fairness.' Do specific fairness work, for specific populations, in specific scenarios. That's more engineering than theology. I like that honesty."

---

*Paper info*

• Title: The Impossibility of Fair LLMs
• Authors: Jacy Reese Anthis, Kristian Lum, Michael Ekstrand, Avi Feller, Chenhao Tan
• Venue: ACL 2025 (Long Paper)
• Link: ACL Anthology
• Topics: AI Fairness, LLM, Social Impact