Paper Overview
Field: ML Authors: Priya Dhawka, Benjamin Wilson, Andrew Head Published: 2026-11-09 arXiv: 2611.12834
Summary
The growing complex reasoning capabilities of AI models have made them popular tools for forecasting—the systematic prediction of future events based on past and present data. Once AI forecasting systems are adopted, however, human builders shoulder the responsibility of constructing and evaluating them.
This paper explores what this work actually looks like through qualitative interviews with 15 forecasting bot builders, all successful participants from a recent AI forecasting competition. The analysis reveals three challenges:
1. Tuning factual research acquisition — configuring how bots gather relevant research and evidence 2. Implementing coherent reasoning — making the bot's reasoning process consistent and reliable 3. Understanding actual bot performance — evaluating how the bot is truly doing
Across this work, builders frequently introduced complexity that cost more than it delivered, and their testing approaches were opportunistic and fragmented.
This detailed characterization of builders' challenges helps clarify where innovations in forecasting pipelines, tooling, and human-AI collaboration should be targeted.
Original Abstract
We explore the work of building AI forecasting bots through qualitative interviews with 15 builders, surfacing challenges around tuning factual research acquisition, implementing coherent reasoning, and understanding actual bot performance.
*Auto-collected on 2026-11-11*