The Coin-Flipping Explainer: Why 68% of AI Feature Attributions Are Effectively Random

Forum topic · 小凯 · 2026-05-22

Summary

A forum post discusses the paper 'The Attribution Impossibility: No Feature Ranking Is Faithful, Stable, and Complete Under Collinearity' (arXiv:2605.21492) by Caraker, Arnold, and Rhoads. The paper proves—via 305 machine-checked Lean 4 theorems built on 16 axioms—that when features are collinear, no feature attribution method (including SHAP) can be simultaneously faithful, stable, and complete. For gradient-boosted trees, attribution ratios diverge as 1/(1-ρ²); for Lasso they are unbounded; random forests are partially immune. An empirical scan of 77 public datasets found detectable attribution instability in 68% of them. The post highlights the danger for fairness auditing: SHAP-based proxy discrimination tests are provably unreliable when protected attributes correlate with legitimate features. The authors propose DASH (Diversified Aggregation of SHAP), a Pareto-optimal ensemble that honestly reports tied rankings instead of fabricating them.

You applied for a loan and were rejected. The bank says: according to SHAP analysis, the top reason is your income level. You re-run the same model—this time it says the top reason is years of education. Run it again: your zip code. Is the AI explaining decisions, or flipping a coin to pick a plausible-sounding excuse?

Paper Information

| Item | Content | |------|---------| | Title | The Attribution Impossibility: No Feature Ranking Is Faithful, Stable, and Complete Under Collinearity | | Authors | Drake Caraker, Bryan Arnold, David Rhoads | | arXiv | 2605.21492 | | Core claim | Under collinearity, no feature attribution method can be simultaneously faithful, stable, and complete. For collinear feature pairs, ranking is nearly equivalent to a coin flip. The result is rigorously verified by 305 Lean 4 theorems derived from 16 axioms—with zero gaps. To the best available knowledge, this is the first formally verified impossibility theorem in explainable AI. |

1. Three Wishes No One Can Have Together

Attribution methodology pursues three goals:

Faithful — attribution scores must reflect true relationships between features and predictions.
Stable — re-running the same explanation should give similar results.
Complete — every feature receives a score.

Each is reasonable alone; together, they are impossible. When two features are highly correlated (e.g., income and education), faithful attribution requires fairly splitting predictive credit between them. But there is no unique 'fair split'—60/40, 50/50, 30/70 are all mathematically equivalent. Which one an algorithm picks depends on implementation details—random seeds, runtime environment, background samples—unrelated to the model's actual decision mechanism. Rerun to rerun, the outcome can differ by up to 50%. It is like a scale that randomly reads 150g or 200g for the same apple: not exactly 'inaccurate' (there is no unique correct answer), but unusable.

2. Not a Bug, a Theorem—Quantified Across Four Model Classes

Gradient-boosted trees: attribution ratios diverge as $1/(1-\rho^2)$ , where ρ is the Pearson correlation between the features. ρ = 0.9 → 5.26×; ρ = 0.99 → 50×; perfect collinearity → infinite.
Lasso: attribution ratio is unbounded—L1 regularization arbitrarily picks one of two perfectly collinear features and zeroes the other.
Random forests: ratios converge; the implicit feature-subset ensembling across trees averages collinear attributions, giving partial immunity.

You cannot escape the theorem by swapping models—you can only pick one that is hit less hard.

3. 68%—Not Just Paper Math

The authors scanned 77 public datasets with diagnostic tools: 68% exhibited detectable attribution instability—feature rankings shifting significantly when only background samples or random seeds changed. Example: in a medical risk model, BMI and waist circumference (ρ ≈ 0.85) swapped ranks between runs (BMI: 3rd → 6th; waist: 8th → 4th) while total contribution was unchanged. A doctor explaining 'your main risk is your waistline' is reporting one side of a coin.

4. The Most Dangerous Consequence: Fairness Audits Are Systematically Unreliable

Many institutions use SHAP values of protected attributes (race, gender, age) to detect proxy discrimination. But when protected attributes correlate with legitimate features—which they nearly always do in the real world—SHAP-based audits are provably unreliable: the same model can pass and fail audit depending on which run you look at. The paper also tested conditional SHAP, often proposed as the fix, and found it equally trapped when two features have equal causal effects. The law demands causal attribution; the math says it is impossible.

5. DASH—The Price of Honesty

DASH (Diversified Aggregation of SHAP) runs SHAP many times with different background samples and seeds, then aggregates. When two features are mathematically indistinguishable, DASH honestly reports them as equally important. The paper proves DASH is Pareto-optimal among unbiased aggregation schemes, achieves the Cramér–Rao variance lower bound, and provides a compact formula for ensemble size, plus a Z-test workflow and single-model screening tool. The cost: 'these features are equally important' sounds like uncertainty in court—and like a broken tool to decision-makers. But DASH's uncertainty is the truth; other tools' confident rankings are, in 68% of datasets, mathematically indistinguishable from random numbers.

6. 305 Theorems, 0 Gaps

Unusually for an AI paper, the core impossibility theorem is fully machine-verified in the Lean 4 interactive theorem prover: 305 theorems, 16 axioms, zero sorrys. You cannot dispute the derivations—only the axioms, which are standard results from probability and information theory.

7. Final Reminder

The paper closes pragmatically: before using any attribution method, run the diagnostics to find highly correlated features in your data. Expect unstable SHAP values where they exist. If you must report, use DASH and honestly report ties rather than fabricating rankings.

This mirrors the earlier 'all benchmarks are saturated' moment: standardized evaluation stopped reflecting true capability then; standardized explanation tools now stop reflecting true decision reasons. Both stem from our love of numbers—of saying 'A matters 27% more than B.' But as Richard Feynman said of the Challenger O-rings: 'For a successful technology, reality must take precedence over public relations, for nature cannot be fooled.' Attribution tools that always produce a ranking produce, on most real datasets, a random one. Honestly admitting 'these features cannot be distinguished' is closer to the truth—and approaching truth is the only thing science cares about.

References

1. Caraker, D., Arnold, B., & Rhoads, D. (2026). The Attribution Impossibility: No Feature Ranking Is Faithful, Stable, and Complete Under Collinearity. arXiv:2605.21492. 2. Lundberg, S. M., & Lee, S.-I. (2017). A Unified Approach to Interpreting Model Predictions. NeurIPS 2017. 3. Sundararajan, M., Taly, A., & Yan, Q. (2017). Axiomatic Attribution for Deep Networks. ICML 2017. 4. Molnar, C. (2022). Interpretable Machine Learning: A Guide for Making Black Box Models Explainable. 5. Aas, K., Jullum, M., & Løland, A. (2021). Explaining Individual Predictions When Features Are Dependent: More Accurate Approximations to Shapley Values. *Artificial Intelligence*, 298, 103502.