Summary
A new arXiv paper (2504.19773) by Longju Bai, Zhemin Huang, and Xingyao Wang presents the first systematic study of token consumption in agentic coding tasks. Key findings: (1) agentic tasks are exceptionally expensive, consuming roughly 1000x more tokens than code reasoning or code chat, with input tokens—largely from repeated context reading—driving overall cost; (2) token usage is highly variable and inherently stochastic, with runs on the same task differing by up to 30x, and higher token consumption does not translate into higher accuracy; (3) token efficiency varies significantly across models—Kimi-K2 and Claude-Sonnet-4.5 consume on average over 1.5 million more tokens than GPT-5; (4) human expert ratings of task difficulty only weakly correlate with actual token cost; and (5) frontier models cannot accurately predict their own token usage and systematically underestimate real costs. The work provides a foundation for cost-aware agent design and budget prediction, with important implications for developers and businesses deploying LLM agents in production workflows.
Overview
Field: NLP
Authors: Longju Bai, Zhemin Huang, Xingyao Wang
Published: 2025-04-28
arXiv: 2504.19773
Abstract
The wide adoption of AI agents in complex human workflows is driving rapid growth in LLM token consumption. When agents are deployed on tasks that require a significant amount of tokens, three questions naturally arise:
1. Where do AI agents spend the tokens?
2. Which models are more token-efficient?
3. Can agents predict their token usage before task execution?
This paper presents the first systematic study of token consumption patterns in agentic coding tasks. Key findings:
- Agentic tasks are uniquely expensive: they consume 1000x more tokens than code reasoning and code chat, with input tokens rather than output tokens driving the overall cost.
- Token usage is highly variable and inherently stochastic: runs on the same task can differ by up to 30x in total tokens, and higher token consumption does not translate into higher accuracy.
- Token efficiency differs significantly across models: Kimi-K2 and Claude-Sonnet-4.5 consume on average more than 1.5 million additional tokens compared to GPT-5.
- Human-perceived difficulty weakly predicts cost: task difficulty ratings by human experts only weakly correlate with actual token cost.
- Models cannot predict their own spend: frontier models fail to accurately predict their own token usage and systematically underestimate actual costs.
These results highlight the need for cost-aware agent design and better budget prediction mechanisms when deploying LLM agents at scale.
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/177618831