— A rigorous "leakage-graded audit" methodology paper covering 10 models
> arXiv: 2609.11838 | Raad Bin Tareaf, Murad Al-Rajab, Samia Loucif, Samer Ellaham, Cedric Schmitz | 2026
---
A Suspicious Number
If you skim recent cardiovascular disease screening papers, you'll notice a recurring figure: AUROC ≈ 0.89. Machine learning models built on national health survey data almost all reach this level. It looks like ML has "solved" heart disease screening.
But the paper's authors raise a pointed question: Is that 0.89 what the model learned, or what leaked out of the data?
"Target leakage" refers to training features that are directly tied to the outcome variable—for example, a "has previously been diagnosed with heart disease" feature essentially stuffs the answer into the input. The model doesn't need to "learn"; it only needs to "look it up."
---
Experimental Design: Five Tiers of Leakage Risk
The authors did something very systematic: they stratified features from 442,067 respondents into five tiers by leakage risk, from high to low.
Then they tested 10 different models at each tier: linear models, tree ensembles, neural networks, glass-box models (Glass-Box EBM), and tabular foundation models. Every model was fully audited at every tier—not just AUROC, but also calibration, fairness, conformal coverage, explanation faithfulness, and inference cost.
Finally, the trained models (including their thresholds) were applied unchanged to data from 430,755 respondents in 2023, to test transferability.
---
Core Finding: Remove Two Features and Every Model Drops
Removing two "post-diagnosis" features dropped every model's AUROC by 0.049–0.051. This means roughly 5 percentage points of that earlier 0.89 came from pure leakage.
More striking: after removing the leaky features, all 10 models clustered in a narrow band just 0.0045 wide. The differences between linear models, neural networks, glass-box models, and tabular foundation models nearly vanished.
This directly answers the question posed in the paper's title: "Is the reported performance due to the model class or to leakage?" The answer is leakage.
---
Glass-Box Models: Faster, Better, and Transparent
Among all models, the glass-box model (Explainable Boosting Machine, EBM) was the most pleasant surprise:
- Non-inferiority: within a pre-specified 0.005 margin, EBM was not worse than any other model, including the most expensive tabular foundation model
- Speed: 104× faster than the strongest tabular foundation model
- Interpretability: EBM's shape functions can be directly audited, meaning fairness fixes and uncertainty calibration are transparent
- Scalar illusion: AUROC is a scalar, but the causal structure behind "0.89" can be completely different—it may come from learning or from leakage. Managing a vector problem with a scalar is the root of systematic blind spots.
- Judgment-gate decoupling: what a model "learned" and what it "exploited from leakage" are separate. After removing leakage, the model still "learned something," but the earlier 0.89 judgment never asked "how much of this accuracy comes from learning?"
- Omission blindness: conventional evaluation only checks "what is the model's accuracy" (presence), not "where does the accuracy come from" (absence). This paper does exactly the latter—making "accuracy provenance" the audit target.
Transparency comes at no extra cost. This may be the paper's most important practical takeaway.
---
Fairness: Women Are Being Missed
The paper reveals a serious fairness problem: at a uniform threshold, the model detects myocardial infarction in 75.4% of women versus 89.0% of men—a 13.6 percentage-point gap.
But the authors don't stop at reporting the problem. By editing EBM's shape functions, they shrank the gender gap to 1.0 percentage point. This kind of repair is possible precisely because glass-box models are auditable—you can see which feature contributes what over which value ranges, then adjust surgically.
Black-box models can't do this. You don't know why it misses women, so all you can do is retrain and pray.
---
Conformal Prediction: A Coverage Blind Spot
The paper also tested Marginal Conformal Prediction and found uneven coverage across groups: 0.86 for men, 0.82 for adults over 60. Uncertainty estimates are more reliable for men and older adults, less reliable for women and younger people.
The authors fixed coverage for all groups with Mondrian calibration. Again, this fix relies on model auditability.
---
Transferability: Frozen Models Are Temporally Stable
Applying the 2022-trained models (with thresholds) unchanged to 2023 data shifted AUROC by less than 0.002. The models are temporally stable.
This result shows that once leaky features are removed, the remaining signal is real and transferable. The model isn't memorizing the 2022 data distribution—it's learning genuine cardiovascular risk factors.
---
What This Means
The paper's conclusion can be condensed into one sentence:
> "The reported performance space is a property of the feature set, not the learner. Evaluation practice, not model capacity, is the binding constraint."
In plain language: you think you're comparing models, but you're actually comparing feature sets. As long as you use the same leaky features, any model hits 0.89; remove them and every model falls to 0.84. Model class simply isn't the key variable.
---
Connection to the "Evaluation Blind Spot Law"
This paper is one of the cleanest illustrations of the "evaluation blind spot law" I've seen recently:
Practical Implications
1. Audit your features. Before celebrating 0.89, ask yourself: will these features be available at inference time? Have any "post-diagnosis" features slipped in? 2. Glass-box models are underrated. EBM wins across speed, accuracy, interpretability, and fairness repair. Transparency isn't just an "explainability" bonus—it's the prerequisite for "repairability." 3. Model class isn't the key variable. Stop agonizing over "XGBoost vs. neural networks." On tabular data, feature engineering and leakage auditing matter 100× more than model choice. 4. Fairness requires auditability. You can't fix a model you can't understand. Fairness repair on black-box models is essentially patching a black box; better to use an auditable model directly.
---
My Take
This paper does something few papers do: it doesn't propose a new model—it audits the evaluation practices of an entire field. And its audit isn't a "you're all doing it wrong" critique; it's a systematic analysis of five leakage tiers, ten models, and six audit dimensions.
This kind of "meta-evaluation" work is more valuable than proposing a new model. Because if you don't even know whether "0.89 is real or fake," all your architectural optimizations are built on sand.
Before chasing higher accuracy, confirm your accuracy is real.
---
Paper link: https://arxiv.org/abs/2609.11838
Open-source code: not yet released (data from the CDC BRFSS 2022–2023 public dataset)
Related concepts: Target Leakage, Glass-Box ML, Conformal Prediction, Fairness Auditing