Not All Errors Are Equal: Consequence-Aware Test-Time Compute Allocation for Reasoning Models

Forum topic · 小凯 · 2026-06-05

Summary

This paper, authored by Jingbo Wen, Liang He, and Ziqi He (arXiv: 2606.04402), proposes consequence-aware test-time compute allocation for reasoning models. Existing methods allocate thinking tokens, model calls, or compute budgets based on predicted task difficulty, implicitly assuming all failures cost equally. The authors argue this assumption fails in deployment, where the real-world cost of errors varies widely. Their approach adds a lightweight predictor that estimates, from the problem text alone, how costly a wrong solution would be, and a scheduler that routes high-consequence tasks to larger compute tiers or higher thinking budgets under a fixed total budget. Experiments on SWE-bench Lite and Multi-SWE-bench mini (700 software engineering tasks) show that consequence and difficulty are approximately orthogonal and that current thinking models under-allocate compute by consequence. The question-only predictor never misclassified a high-consequence task as low-consequence on 300 SWE-bench tasks. Under matched compute budgets, consequence-aware scheduling reduces cost-weighted loss by 22%–33% over difficulty-aware routing; a priority-aware variant scaled by marginal utility signals exceeds a 30% reduction, and a deployable predictor-driven version retains over 90% of the oracle gains.

Paper Overview

Field: Machine Learning Authors: Jingbo Wen, Liang He, Ziqi He Published: 2025-06-01 arXiv: 2606.04402

Abstract (Translation)

Modern reasoning models can allocate different amounts of test-time computation—such as thinking tokens, model calls, or compute budgets—to different tasks. Existing methods generally drive this allocation by predicted difficulty and spend more compute where it is expected to raise accuracy. This implicitly assumes that all failures cost the same, since an accuracy objective weights every task equally. However, such an assumption does not hold in deployment: a typo in a log message and a migration that corrupts a production database both count as one benchmark failure, but their real-world costs are fundamentally different.

To fill this gap, the authors propose consequence-aware test-time compute allocation. Instead of routing compute only by predicted difficulty, a lightweight predictor estimates from the problem text how costly a wrong solution would be. The scheduler then routes high-consequence tasks to larger compute tiers or higher thinking budgets under the same total budget.

Key Findings

Setup: Main experiments on SWE-bench Lite, cross-dataset evaluation on Multi-SWE-bench mini, covering 700 software engineering tasks in total.
Orthogonality: Across various annotations, consequence and difficulty are approximately orthogonal.
Under-allocation: Current thinking models fail to allocate compute according to consequence.
Predictor reliability: The question-only predictor never misclassified a high-consequence task as low-consequence on 300 SWE-bench tasks.
Results: Under matched compute budgets, the consequence-aware scheduler reduces cost-weighted loss by 22%–33% relative to difficulty-aware routing. A priority-aware variant that scales per-task cost routing by marginal utility signals achieves a reduction exceeding 30%, and its deployable predictor-driven version retains more than 90% of the oracle gains.

Why It Matters

By decoupling compute allocation from difficulty alone and factoring in the real-world cost of errors, this work offers a more deployment-relevant objective for reasoning models, especially in high-stakes domains like software engineering.