When Prediction Error Is Not Enough: Evaluating Nuisance-Function Estimators in Causal Inference

Forum topic · 小凯 · 2026-09-03

Summary

A study by Cong Cao (arXiv:2509.00010) examines whether prediction error is a reliable proxy for causal estimator quality when evaluating nuisance-function estimators in causal inference. Using Monte Carlo simulations in a partially linear model, the paper compares OLS, generalized additive models (GAMs), XGBoost, and Double Machine Learning with XGBoost (DML-XGBoost), assessing nuisance-function prediction error, bias, RMSE, and 95% confidence interval coverage, plus a joint-error measure based on the absolute cross-product of exposure and outcome nuisance errors. Results show XGBoost achieved the lowest RMSE among non-oracle methods, while DML-XGBoost generally provided better confidence interval coverage. Prediction error did not consistently track causal bias across methods and settings, and the method with the best point-estimate performance was not necessarily best at coverage. The joint-error measure correlated only weakly with causal bias. The conclusion: prediction error is useful for evaluating nuisance-function estimation but should not be treated as a direct measure of causal estimator quality.

When Prediction Error Is Not Enough: Evaluating Nuisance-Function Estimators in Causal Inference

Field: Machine Learning / Causal Inference Author: Cong Cao Published: 2026-09-03 arXiv: 2509.00010

Abstract

Prediction error is widely used to evaluate nuisance-function estimators in causal inference, but its relationship with causal estimator performance may differ across performance measures. We studied this question in a partially linear model using Monte Carlo simulations. We compared ordinary least squares (OLS), generalized additive models (GAMs), XGBoost, and Double Machine Learning with XGBoost (DML-XGBoost), evaluating nuisance-function prediction error, bias, RMSE, and 95% confidence interval coverage. We also examined a simple joint-error measure based on the absolute cross-product of estimation errors from the exposure and outcome nuisance functions.

Key Findings

Across the simulated settings, XGBoost had the lowest RMSE among the non-oracle methods, while DML-XGBoost generally provided better confidence interval coverage.
Prediction error did not consistently track causal bias across methods and settings; the method with the best point-estimate performance did not necessarily have the best confidence interval coverage.
The joint-error measure correlated only weakly with causal bias and did not provide a useful independent measure of causal performance.
Overall, prediction error is useful for evaluating nuisance-function estimation but should not be viewed as a direct measure of the quality of the resulting causal estimator.

When Prediction Error Is Not Enough: Evaluating Nuisance-Function Estimators in Causal Inference

Summary

When Prediction Error Is Not Enough: Evaluating Nuisance-Function Estimators in Causal Inference

Abstract

Key Findings

Tags