When Prediction Error Is Not Enough: Evaluating Nuisance-Function Estimators in Causal Inference

Forum topic · 小凯 · 2026-09-03

Summary

This paper examines whether prediction error, widely used to evaluate nuisance-function estimators in causal inference, reliably reflects causal estimator performance. Using Monte Carlo simulations in a partially linear model, the author compares ordinary least squares (OLS), generalized additive models (GAMs), XGBoost, and Double Machine Learning with XGBoost (DML-XGBoost), measuring nuisance-function prediction error, bias, RMSE, and 95% confidence interval coverage, plus a joint-error measure based on the absolute cross-product of exposure and outcome nuisance estimation errors. Results show XGBoost achieved the lowest RMSE among non-oracle methods, while DML-XGBoost generally provided better confidence interval coverage. Prediction error did not consistently track causal bias across methods and settings, and methods with the best point-estimate performance did not necessarily have the best interval coverage. The joint-error measure was only weakly correlated with causal bias and offered no useful standalone assessment. The paper concludes that prediction error is useful for evaluating nuisance estimation but should not be treated as a direct measure of causal estimator quality.

Paper Overview

Field: Machine Learning Author: Cong Cao Published: 2026-09-03 arXiv: 2509.00010

Abstract

Prediction error is widely used to evaluate nuisance-function estimators in causal inference, but its relationship with causal estimator performance may differ across performance measures. We studied this question in a partially linear model using Monte Carlo simulations. We compared ordinary least squares (OLS), generalized additive models (GAMs), XGBoost, and Double Machine Learning with XGBoost (DML-XGBoost), evaluating nuisance-function prediction error, bias, RMSE, and 95% confidence interval coverage. We also examined a simple joint-error measure based on the absolute cross-product of estimation errors from the exposure and outcome nuisance functions. Across the simulated settings, XGBoost had the lowest RMSE among the non-oracle methods, while DML-XGBoost generally provided better confidence interval coverage. Prediction error did not consistently track causal bias across methods and settings, and the method with the best point-estimate performance did not necessarily have the best confidence interval coverage. The joint-error measure was only weakly correlated with causal bias and did not provide a useful independent measure of causal performance. These results suggest that prediction error is useful for evaluating nuisance-function estimation, but should not be viewed as a direct measure of the quality of the resulting causal estimator.

Key Takeaways

Prediction error ≠ causal quality: Nuisance-function prediction error does not consistently track causal bias or interval coverage across methods and settings.
Method comparison: XGBoost achieved the lowest RMSE among non-oracle methods; DML-XGBoost generally delivered better 95% confidence interval coverage.
Joint-error measure falls short: A simple cross-product-based joint error metric showed only weak correlation with causal bias.
Practical implication: When choosing nuisance estimators in causal inference, evaluate causal-level performance (bias, RMSE, coverage) directly rather than relying solely on prediction error.

---

*Auto-collected on 2026-09-03*