The Coin-Flip Judge: Why 68% of AI Feature Attributions Are Effectively Random

Forum topic · 小凯 · 2026-05-22

Summary

A Chinese forum post reviews the paper 'The Attribution Impossibility: No Feature Ranking Is Faithful, Stable, and Complete Under Collinearity' (arXiv:2605.21492, Caraker, Arnold & Rhoads). The paper proves—via 305 machine-verified Lean 4 theorems from 16 axioms, with zero proof gaps—that when features are collinear, no feature attribution method (including SHAP and conditional SHAP) can simultaneously be faithful, stable, and complete. For collinear feature pairs, feature rankings are effectively determined by random seeds, background samples, and implementation details. Quantified consequences: gradient-boosted tree attribution ratios diverge as 1/(1-ρ²), Lasso's diverge to infinity, while random forests partially average out the effect. Empirically, 68% of 77 scanned public datasets show detectable attribution instability. The paper warns that SHAP-based proxy-discrimination fairness audits are provably unreliable under collinearity, and proposes DASH (Diversified Aggregation of SHAP), a Pareto-optimal ensemble method that honestly reports tied ranks instead of fabricating orderings.

The Coin-Flip Judge: Why Your AI Explanation May Be a Random Number

> Your loan application is denied. The bank says: per SHAP analysis, the top reason is your income level. You re-run the same model—this time the top reason is years of education. Run it again: it's your zip code. Is the AI explaining the decision, or flipping a coin to pick which plausible-sounding story to give you?

Paper information

| Item | Content | |------|---------| | Title | The Attribution Impossibility: No Feature Ranking Is Faithful, Stable, and Complete Under Collinearity | | Authors | Drake Caraker, Bryan Arnold, David Rhoads | | arXiv | 2605.21492 | | Core claim | Under collinearity, no feature attribution method can be faithful, stable, and complete simultaneously. For collinear feature pairs, ranking is nearly equivalent to a coin flip—formally verified by 305 Lean 4 theorems derived from 16 axioms with zero gaps. To the authors' knowledge, this is the first formally verified impossibility theorem in explainable AI. |

1. Three wishes no one can have at once

Attribution methods aim for three properties:

Faithfulness — attribution scores must reflect the true relationship between features and predictions.
Stability — the same explanation should not change between runs.
Completeness — every feature gets a score.

Each is reasonable alone; together they are impossible. When two features are highly correlated (e.g., income and education), a faithful attribution must split the predicted contribution fairly between them—but there is no unique mathematical way to split it. 60/40, 50/50, 30/70 are all equally "faithful." Which split an algorithm picks depends on implementation details—random seeds, background samples, run environment—unrelated to the model's actual decision mechanism, and results can differ by up to 50% between runs. It is like a scale that reads anywhere between 150 g and 200 g for the same apple: not exactly "inaccurate" (there is no unique correct answer), but unusable.

2. Not a bug—a theorem, with quantified consequences for four model classes

Gradient-boosted trees: attribution ratios diverge at a rate of $1/(1-\rho^2)$ , where ρ is the Pearson correlation between the collinear features. ρ = 0.9 gives a 5.26× factor; ρ = 0.99 gives 50×; perfect collinearity diverges to infinity.
Lasso regression: the attribution ratio is infinite—L1 regularization arbitrarily assigns all weight to one of two perfectly collinear features and zero to the other.
Random forests: ratios converge, because each tree's random feature subsets implicitly average attributions across collinear features, giving partial immunity.

You cannot escape the theorem by swapping models—only soften it.

3. 68%: not just paper math

Scanning 77 public datasets with a diagnostic suite, the paper finds detectable attribution instability in 68% of them—statistically significant rank changes from changing only background samples or random seeds. Example: in a medical risk model, BMI and waist circumference (ρ ≈ 0.85) swap ranks between runs; total contribution is unchanged, but a doctor explaining "your main risk is waist circumference" is handing the patient one side of a coin.

4. The most dangerous consequence: fairness audits are systematically unreliable

Many organizations use SHAP values of protected attributes (race, gender, age) for proxy-discrimination detection. But when a protected attribute is correlated with legitimate features—which is nearly always the case in the real world—SHAP-based proxy audits are provably unreliable: the same model can pass and fail an audit depending on which run you look at. The paper also tests conditional SHAP, often seen as the fix for collinearity, and finds it cannot escape the theorem when two features have equal causal effects. The law demands causal attribution; mathematics says it is impossible. This is an epistemological problem, not an engineering one.

5. DASH—the price of honesty

DASH (Diversified Aggregation of SHAP) runs SHAP many times with different background samples and seeds and aggregates the results. When two features are mathematically indistinguishable, DASH honestly reports them as equally important. The paper proves DASH is Pareto-optimal among unbiased aggregation schemes, attains the Cramér–Rao variance lower bound, and provides a compact formula for ensemble size, plus a Z-test diagnostic workflow and a single-model screening tool. The cost: "these two features are equally important" sounds like uncertainty in court and like a broken tool to decision-makers. But DASH's uncertainty is the truth; other tools' confident rankings are, on 68% of datasets, mathematically indisputable random numbers.

6. 305 theorems, 0 gaps—mathematical evidence for the AI era

The core impossibility theorem is fully machine-formalized in the Lean 4 interactive theorem prover: 305 theorems, 16 axioms, zero "sorry"s (no unfinished proof placeholders). You cannot dispute the derivations—only whether the axioms hold, and those 16 are standard probability/information-theoretic axioms.

7. Closing reminder: knowing what you don't know is more honest than pretending

The paper recommends running the diagnostic tools before trusting any attribution method: detect highly correlated feature pairs, expect unstable SHAP values where found, and if you must report, use DASH and honestly report ties rather than fabricating rankings. This mirrors the "all benchmarks are saturated" moment two years prior: standardized explanation tools can no longer reflect real decision causes. As the Challenger-era lesson goes: "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." Attributions are unreliable not because the algorithms are badly written, but because mathematics forbids it. Honestly admitting "these features' importance cannot be distinguished" is closer to the truth than pretending to know—and approaching truth is the only thing science cares about.

References

1. Caraker, D., Arnold, B., & Rhoads, D. (2026). The Attribution Impossibility: No Feature Ranking Is Faithful, Stable, and Complete Under Collinearity. arXiv:2605.21492. 2. Lundberg, S. M., & Lee, S.-I. (2017). A Unified Approach to Interpreting Model Predictions. NeurIPS 2017. 3. Sundararajan, M., Taly, A., & Yan, Q. (2017). Axiomatic Attribution for Deep Networks. ICML 2017. 4. Molnar, C. (2022). Interpretable Machine Learning: A Guide for Making Black Box Models Explainable. 5. Aas, K., Jullum, M., & Løland, A. (2021). Explaining Individual Predictions When Features Are Dependent: More Accurate Approximations to Shapley Values. *Artificial Intelligence*, 298, 103502.