> Feynman once said: "If you think you understand something, you should be able to explain it clearly." Translated to AI fairness: if a model is truly "fair" to everyone, it should make decisions using the same reasoning logic for everyone. This paper shows that many "fair" models are nothing of the sort.
---
Introduction: Two applicants, same outcome — different reasons
Imagine you are a bank loan officer. Two applicants walk in — a white man and a Black man — with nearly identical income, occupation, and education.
Current AI fairness methods (statistical parity, equalized odds, etc.) require you to give them the same outcome — if their risk is the same, both are approved or both rejected. This is called "Outcome Fairness."
But the paper asks a deeper question: even with the same outcome, did your AI model decide using the same "reasons"?
The researchers identified a hidden phenomenon they call "Procedural Bias":
- For the white applicant, the model says: "Approved — because of a high credit score."
- For the Black applicant, the model says: "Approved — but mainly because of stable income; credit score is secondary."
- All baseline models flagged as "outcome-fair" exhibited significant hidden procedural bias
- These models assigned systematically different weights to credit history, education level, and employment stability across groups
- Looking at outcomes alone — completely invisible
- Procedural bias was substantially reduced
- Predictive accuracy paid only a moderate price (small losses in most scenarios)
- This shows outcome fairness and procedural fairness are not a zero-sum game — you don't need to sacrifice much accuracy to gain procedural fairness
- Title: Do Fair Models Reason Fairly? Counterfactual Explanation Consistency for Procedural Fairness in Credit Decisions
- Authors: Gideon Popoola, John Sheppard
- arXiv ID: 2605.12701
- Categories: cs.LG, cs.AI, cs.CE, cs.CY
- Test data: Synthetic + German Credit + Adult Income + HMDA
Same result: "approved." All standard fairness metrics pass. But the model's reasoning process differs. For one group it emphasizes credit score; for the other, income. Even in outcome-fair models, the meaning of certain features varies by race.
---
Chapter 2: Regime B — "Same exit, different path"
The paper introduces a key taxonomy:
| | Same outcome | Same outcome | Different outcome | |:--|:--|:--|:--| | | Same reasoning | Different reasoning | Any reasoning | | Label | Regime A (truly fair) | Regime B (hidden bias) | Standard bias (detectable) |
Regime B is the biggest blind spot in current AI auditing. It's like two students scoring identically on an exam but using completely different solution methods — you can't claim "same score means same work." When a model applies different feature weights to different groups (even while achieving the same outcomes), it remains fundamentally unfair — biases from the training data have seeped into the model's reasoning structure, merely being "balanced out" at the output level.
---
Chapter 3: Detecting procedural bias
The researchers propose a framework called CEC (Counterfactual Explanation Consistency):
1. Generate a counterfactual counterpart for each individual — "what decision would this person receive if they were a different race?" 2. Compare the feature attributions of the original and counterfactual decisions — which features did the model rely on in each case? 3. If feature attributions differ significantly between counterfactual pairs → procedural bias exists
Testing on four datasets (synthetic data, German Credit, Adult Income, HMDA mortgage lending) revealed:
---
Chapter 4: The cost of fixing it — and a training loss
The paper doesn't just find the problem; it proposes a fix. They add a procedural fairness regularization term to the training loss — requiring the model to produce fair outcomes *and* maintain consistent feature attributions across groups.
Experimental results:
---
Chapter 5: Why this matters enormously
Current AI regulatory frameworks (including the EU AI Act and US algorithmic accountability initiatives) focus mainly on outcome fairness. This paper's findings mean:
1. Passing outcome audits ≠ real fairness. A model can perform perfectly on all outcome metrics while still using different reasoning logic for different groups. 2. New regulatory metrics are needed: not just "same people get the same outcomes," but "same people get the same outcomes for the same reasons." 3. Explainability is not optional: without examining the model's reasoning process (feature attributions), Regime B cannot be detected.
---
A Feynman-style closing thought
"It's like taking an exam where you and a classmate submit identical answers — but the teacher checks your scratch paper and finds: you used method A, your classmate used method G. And this difference only systematically appears under certain 'identity' conditions.
Shouldn't the teacher ask: why did two students reach the same answer via different methods? The answer is simple — your scratch papers carry traces of different 'training data.'
This paper did something remarkable: it proved that audits that only look at outcomes and never at process are self-deception. A model can hand you a perfect 'non-discrimination' — and then lie to you at the explanation level."
---
*Paper information*