Competence-Gated Pooling of Language Models and Priors for Event Forecasting

Paper Overview

• Field: Machine Learning
• Authors: Aditi Tiwari, Aashrith Bandaru, Heng Ji
• Published: 2026-09-15
• arXiv: 2609.12101

Summary

In hybrid forecasting, a language model is often one of several available signals. A system may already have a market, crowd, or statistical forecast and must decide whether the model adds useful information or should be ignored. The relevant target is therefore not standalone model accuracy, but relative competence — the model's marginal value beyond the available external forecast.

Under Brier loss, the authors characterize when model disagreement can improve an external forecast and derive the gain from using domain-specific rather than global pooling weights. They then introduce a competence gate that:

• Estimates domain-level source weights from resolved outcomes
• Shrinks uncertain estimates toward a global weight
• Recalibrates the pooled forecast

Key Results

• Across 2,357 resolved binary questions and five language models, the gate improves a strong external baseline from 0.0771 to 0.0732 Brier score, significantly outperforming global forecast pooling.
• Gains remain significant under leakage controls and leak-proof time-series priors for pooling structured sets, with independent evidence on FRED.
• In contrast, the gate shows no significant improvement on the official ForecastBench market subset, since it largely follows the market.
• Across four Qwen models, verbal confidence does not reliably identify when a model outperforms external forecasts, whereas outcome-estimated competence supports better abstention decisions.

These results offer a practical path toward selective model use based on measured marginal value.

Abstract (Original)

> In hybrid forecasting, a language model is often one of several available signals. A system may already have a market, crowd, or statistical forecast and must decide whether the model adds useful information or should be ignored. The relevant target is therefore not standalone model accuracy, but relative competence, defined as the model's marginal value beyond the available external forecast. Under Brier loss, we characterize when model disagreement can improve an external forecast and derive the gain from using domain-specific rather than global pooling weights. We then introduce a competence gate that estimates domain-level source weights from resolved outcomes, shrinks uncertain estimates toward a global weight, and recalibrates the pooled forecast. Across 2,357 resolved binary que...

---

*Auto-collected on 2026-09-15*