Predictive Credit: Measuring What Scientific Explanations Add to Experimental Forecasts (arXiv 2610.00314)

Forum topic · 小凯 · 2026-10-05

Summary

A new arXiv paper (2610.00314) by Jingjie Ning, Xueqi Li, Yibo Kong, and Dongting Li introduces "predictive credit," a framework for measuring what scientific explanations add to forecasts of planned experiments. Using paired forecasts that share an intervention, forecaster, and outcome while varying description, matched explanation, and donor context, the authors run five checks covering commitment, delivery, predictive gain, alignment, and known-signal uptake. Across 336 prospective states in controlled learning, 12 Tox21 endpoints, and 24 OpenML tasks, the v5 frozen credit decision was inconclusive: Tox21's preregistered ROC AUC interval-score harm test was unmet, OpenML's joint formation, point-equivalence, and repeatability rules were unmet, and matched point-accuracy gains over plain descriptions were not confirmed. LLM-specific effects were observed, including reduced secondary drift with DeepSeek V4 Pro and widened intervals with full cards on OpenML. The protocol targets research-agent benchmarks and scientific forecasting; explanation credit at the measured donor resolution remains unconfirmed.

Paper Overview

Field: Machine Learning
Authors: Jingjie Ning, Xueqi Li, Yibo Kong, Dongting Li
Published: 2026-10-05
arXiv: 2610.00314

Abstract

Research agents explain planned experiments. The authors measure predictive credit with paired forecasts that share an intervention, forecaster, and outcome while varying description, matched explanation, and donor context. Five checks track commitment, delivery, predictive gain, alignment, and known-signal uptake.

Across 336 prospective states in controlled learning, 12 Tox21 endpoints, and 24 OpenML tasks, v5's frozen credit decision was inconclusive:

Tox21's preregistered ROC AUC interval-score harm test was unmet ( $$D-M=-.0026$$ , 95% interval [ $$-.0174$$ , .0104]).
OpenML's joint formation, point-equivalence, and repeatability rules were unmet.
Matched point-accuracy gains over description remained unconfirmed, and Tox21/OpenML seed-donor intervals spanned zero.

Under requested DeepSeek V4 Pro, matched cards and donor cards reduced Tox21 secondary drift by 64.5% and 59.1%, respectively. DeepSeek V4 Flash replay raised matched-point MAE from .01823 to .02020, failing matched-donor interval equivalence. OpenML full cards widened nominal 80% intervals by 21%, with coverage of 49.3% vs 51.4% for descriptions (66 of 144 cards). Direct-text Flash delivery covered all 144 notes but showed no matched point-accuracy gain. Researcher-written mechanistic positive controls reduced point MAE by 2.60 percentage points relative to descriptions.

Conclusion

The protocol is designed for research-agent benchmarks and scientific forecasting. Natural explanation credit remains unconfirmed at the measured donor resolution.

--- *Auto-collected on 2026-10-05*