TokenCast: Forecasting Token Consumption During LLM Agent Execution

Forum topic · 小凯 · 2026-09-30

Summary

TokenCast is a method for predicting token consumption of LLM agents during task execution, where usage can vary by over an order of magnitude across runs of the same task. Because agents choose next actions based on tool feedback and intermediate results, and growing context inflates the input size of every subsequent call, total consumption is hard to predict in advance. TokenCast learns a composable cost representation for each execution segment, recording its own consumption and the context growth it introduces. Composing adjacent segments yields cumulative estimates that capture the extra input cost of re-reading earlier context. Predictions are refreshed as execution unfolds without additional LLM calls, averaging only 32.8ms of cumulative forecast time per run on SWE-bench Verified. Across 4 task suites and 6 agent models, TokenCast reduces mean absolute error by 14.5% on average over the strongest baseline across 96 evaluation combinations, and in offline budget-controlled replays uses 21.3% fewer tokens than fixed-budget policies at matched trajectory completion rates.

Overview

Research area: ML Authors: Chaoqian Ouyang, Ling Yue, Libin Zheng, Huanghui Guo, Shengxiang Xu, YiShu Wang, Ran Li, Jian Yin, Shaowu Pan, Shimin Di Published: 2026-09-28 arXiv: 2609.35760

Abstract

When a large language model (LLM) agent executes the same task, token consumption can vary by over an order of magnitude across runs. The agent chooses its next steps based on tool feedback and intermediate results, while the growing context steadily inflates the input size of every subsequent call. The total consumption of a task is therefore hard to predict before execution, and the prediction must be revised as the run unfolds.

Method

TokenCast learns a composable cost representation for each execution segment, recording its own consumption and the context growth it introduces. Composing adjacent segments yields a cumulative estimate that captures the extra input cost incurred when context from earlier segments is re-read by every later call. As execution unfolds, newly observed evidence refreshes the prediction without requiring additional LLM calls — on SWE-bench Verified, the average cumulative forecasting time is only 32.8ms per run.

Results

Across 4 task suites and 6 agent models, TokenCast reduces mean absolute error by an average of 14.5% relative to the strongest compared methods, over 96 evaluation combinations.
In offline budget-controlled replays, TokenCast uses on average 21.3% fewer tokens than fixed-budget policies at matched trajectory completion rates.