*English translation of a zhichai.net forum post reviewing a leakage-audit methodology paper.*
A tiered leakage audit of 10 models — a rigorous methodological study
> arXiv: 2609.11838 | Raad Bin Tareaf, Murad Al-Rajab, Samia Loucif, Samer Ellaham, Cedric Schmitz | 2026
---
A Suspicious Number
If you browse recent cardiovascular disease screening papers, one number keeps recurring: AUROC ≈ 0.89. Machine learning models built on national health survey data almost all reach this level. It looks like ML has "solved" heart disease screening.
But the authors pose a sharp question: is this 0.89 something the model learned, or something the data leaked?
"Target leakage" means training features contain information directly related to the label — e.g., a "has been diagnosed with heart disease" feature essentially puts the answer into the input. The model doesn't need to learn; it only needs to look up.
---
Experimental Design: Five Leakage-Risk Tiers
The authors did something very systematic: they stratified features for 442,067 respondents into five tiers by leakage risk, from high to low.
Then they tested 10 different models at each tier: linear models, tree ensembles, neural networks, glass-box models (Glass-Box EBM), and tabular foundation models. Every model was fully audited at every tier — not just AUROC, but also calibration, fairness, conformal coverage, explanation faithfulness, and inference cost.
Finally, the trained models (along with their thresholds) were applied unchanged to data from 430,755 respondents in 2023 to test transferability.
---
Core Finding: Remove Two Features and Every Model Drops
After removing two "post-diagnosis" features, every model's AUROC fell by 0.049–0.051. That means roughly 5 percentage points of the earlier 0.89 was pure leakage.
More striking: after removing the leaky features, all 10 models clustered in a narrow band just 0.0045 wide. Linear models, neural networks, glass-box models, tabular foundation models — their differences essentially vanished.
This directly answers the question in the paper's title: "Is the reported accuracy a credit to the model class or to leakage?" The answer is leakage.
---
Glass-Box Models: Fast, Accurate, and Transparent
Among all models, the Explainable Boosting Machine (EBM) was the pleasant surprise:
- Non-inferiority: within a pre-set 0.005 margin, EBM was no worse than any other model, including the most expensive tabular foundation model
- Speed: 104x faster than the strongest tabular foundation model
- Interpretability: EBM's shape functions can be directly audited, meaning fairness fixes and uncertainty calibration are transparent
- Scalar illusion: AUROC is a scalar, but the causal structure behind "0.89" can be entirely different — it could be the result of learning or of leakage. Managing a vector problem with a scalar is the root of systematic blind spots.
- Judgment-gate decoupling: what a model "learned" and what it "exploited via leakage" are separate. After removing leakage, the model still "learned something," but the earlier 0.89 judgment never asked how much of it came from learning.
- Omission blindness: conventional evaluation only checks "how accurate is the model" (presence), not "where does the accuracy come from" (absence). This paper does exactly the latter — auditing the *sources* of accuracy.
Transparency comes at no extra cost. This may be the paper's most important practical conclusion.
---
Fairness: Women Are Under-Detected
The paper reveals a serious fairness issue: at a uniform threshold, the model's myocardial infarction detection rate is 75.4% for women versus 89.0% for men — a 13.6 percentage point gap.
But the authors didn't stop at reporting the problem. By editing the EBM's shape functions, they shrank the sex gap to 1.0 percentage point. This kind of fix is possible precisely because the glass-box model's structure is auditable — you can see which feature contributes what in which value range, and adjust accordingly.
Black-box models can't do this. You don't know why they under-detect women, so all you can do is retrain and pray.
---
Conformal Prediction: Blind Spots in Coverage
The paper also tested Marginal Conformal Prediction and found uneven coverage across groups: 0.86 for men, 0.82 for adults over 60. This means uncertainty estimates are more reliable for men and older adults, less reliable for women and younger people.
The authors fixed coverage for all groups using Mondrian calibration. Again, this fix depends on model auditability.
---
Transferability: Frozen Models Are Temporally Stable
Applying models trained in 2022 (with thresholds) unchanged to 2023 data changed AUROC by less than 0.002. The models are temporally stable.
This result shows that once leaky features are removed, the remaining signal is real and transferable. The models aren't memorizing the 2022 data distribution — they're learning genuine cardiovascular risk factors.
---
What This Means
The paper's conclusion distills to one sentence:
> "The reported accuracy space is a property of the feature set, not of the learner. Evaluation practice, not model capacity, is the binding constraint."
In plain language: you think you're comparing models, but you're actually comparing feature sets. With the same leaky features, any model hits 0.89; remove them, and every model drops to ~0.84. Model class simply isn't the key variable.
---
Connection to the "Evaluation Blind-Spot Law"
This paper is one of the cleanest illustrations I've seen of the "evaluation blind-spot law":
Practical Takeaways
1. Audit your features. Before celebrating 0.89, ask: are these features available at inference time? Have any "post-diagnosis" features crept in? 2. Glass-box models are undervalued. EBM wins on speed, accuracy, interpretability, and fairness repair. Transparency isn't just an "explainability" bonus — it's a precondition for repairability. 3. Model class is not the key variable. Stop agonizing over XGBoost vs. neural networks. On tabular data, feature engineering and leakage auditing matter 100x more than model choice. 4. Fairness requires auditability. You can't repair a model you can't understand. Fairness fixes on black boxes are essentially patching a black box; auditable models are better.
---
My Take
This paper does something few papers do: it doesn't propose a new model — it audits the evaluation practices of an entire field. And its audit isn't a "you're all doing it wrong" critique, but a systematic analysis across five leakage tiers, ten models, and six audit dimensions.
This kind of "meta-evaluation" work is more valuable than proposing a new model. If you don't even know whether "0.89 is real or fake," every optimization you make to model architecture is built on sand.
Before chasing higher accuracy, confirm your accuracy is real.
---
Paper link: https://arxiv.org/abs/2609.11838
Open-source code: not yet released (data from the public CDC BRFSS 2022–2023 datasets)
Related concepts: Target Leakage, Glass-Box ML, Conformal Prediction, Fairness Auditing