Coding Agent Failures: Reflect, Replan, or Just Escalate to a Bigger Model? CodeRescue's Three-Action Routing
> Paper: CodeRescue: Budget-Calibrated Recovery Routing for Coding Agents > Authors: Qijia He, Jiayi Cheng, Chenqian Le et al. (University of Washington / NYU / ByteDance) > arXiv: 2607.19338 > Code: https://github.com/Qijia-He/agent-budget-control
A Familiar Scenario
At 2 AM, you ask a coding agent to implement a parser. A cheap model (say, GPT-5.4-nano) finishes the first version, tests fail — an edge case wasn't handled. Now the agent faces three choices:
1. Reflect: feed the error log back and let the cheap model patch it. Cheapest, but might not fix it. 2. Replan: discard the old approach, restart with a new plan. Slightly costlier, but a fresh idea might pass. 3. Escalate: hand the problem straight to a strong model like GPT-5.4. Most expensive, but most capable.
If your instinct is "just escalate, one step done" — you share the intuition of most existing cost-aware routing systems. That is exactly what this paper challenges.
The Problem with Cascades: Treating Failure as "Switch to a Big Model"
The dominant paradigm in cost-aware LLM inference in recent years is the cascade: try a cheap model first, escalate to a stronger one on failure. FrugalGPT, RouteLLM, HybridLLM, and C3PO are variants of this route.
Its hidden assumption: a cheap model's failure means it lacks capacity, so stronger parameters are needed.
But coding agents have a special property the paper points out: failure itself is information. Compiler errors, test failures, timeouts — execution feedback turns an underdetermined problem into a diagnosed repair problem. Giving the cheap model another shot with feedback may well fix it. Escalating immediately wastes the cheap model's accumulated context understanding.
In the paper's words:
> Spending another cheap call with the right feedback may exploit the small model more effectively than immediately escalating.
This is not "which model is stronger" but "which recovery action is most cost-effective."
Three Actions, Not a Ladder
The core contribution reframes post-failure decisions from a binary "should we escalate?" into routing over three heterogeneous actions:
| Action | Model | Behavior | Relative cost | |--------|-------|----------|---------------| | reflect | cheap | local repair using error feedback | lowest | | replan | cheap | discard old plan, re-plan | medium-low | | escalate | strong | strong model solves from scratch | highest |
Key insight: these three actions are not strictly ordered. The data shows:
- 28% of failures are solvable only by cheap actions (reflect/replan)
- 45% only by escalate
- 27% by both
- \(\lambda = 0\): ignore cost, maximize success only
- larger \(\lambda\): favor cheaper actions
- Train once, deploy everywhere. The router trains once; the CRC calibration table is precomputed. Budget changes are O(K) table lookups — friendly for different customers, time slots, and SLAs.
- The router is tiny. A fully fine-tuned 4B model makes routing cost negligible versus millisecond-dollar-scale escalations.
- Extensible action set. reflect/replan/escalate is a minimal first-order recovery set; you can add "search docs," "ask a human," "run static analysis," as long as cost and rollout data are defined.
- Honest limitations. Single-step recovery only; cheapest-successful labels are a proxy, not calibrated probabilities; CRC controls cost, not solve rate.
data_generation/: rollout collection pipeline (agent implementations of reflect/replan/escalate)sft_runs/: router training configs (LLaMA-Factory format)conformal/: CRC calibration and evaluation scripts
So the one-way ladder "cheap action → escalate" is wrong. Some problems escalate cannot solve, but cheap actions can. Capability is not a monotonic function of cost.
Training a Router
Rather than rules, the authors train a supervised recovery router:
1. Collect rollout data: on 5 coding benchmarks (APPS, TACO, BigCodeBench, LiveCodeBench, CodeContests), ~27,300 problems solved first by GPT-5.4-nano; on failure, all three recovery actions are run and outcomes recorded. 2. Label: for each failure, the "cheapest successful action" is labeled. Problems unsolvable by all three are excluded. 3. Train: full fine-tuning of Qwen3.5-4B; input is (problem + execution verdict + stderr), output is a probability over the three actions.
Notable details: evaluation uses solve rate rather than classification accuracy, since multiple actions may succeed on the same problem — avoiding "either/or" label noise. Also, metadata prefixes matter: adding source, difficulty, and algorithm tags raises solve rate from 0.656 to 0.697, showing execution feedback alone is insufficient to choose the right action.
Conformal Risk Control: One Knob for the Budget
The real highlight: making the same router adapt to different budgets. Instead of baking budget into training (e.g., RACER), the paper uses Conformal Risk Control (CRC).
The router scores each action \(s(a|x)\); instead of a plain argmax, a cost penalty \(\lambda\) is introduced:
\(\lambda\) is the budget knob — changing budgets requires no retraining, only tuning this parameter.
CRC selects the smallest \(\lambda\) such that on an exchangeable calibration set:
where \(B\) is the user-specified budget and \(c_{\max}\) the worst-case cost. Under exchangeability, this guarantees future samples' expected cost stays below \(B\). Tell the system "average at most \(2.56m per recovery," and CRC finds a\)\lambda$ that mathematically holds that line.
An honest engineering philosophy: CRC controls cost, not solve rate. Solve rate is an empirical property the router learns; CRC does not guarantee it. You cannot statistically guarantee quality, but you can guarantee a spending cap.
The Numbers: 35% of the Cost, Better than Always-Escalate
Main results (GPT-5.4-nano / GPT-5.4 pair, 360 held-out problems):
| Strategy | Solve Rate | Avg cost (m$) | |----------|-----------|---------------| | Always-reflect | 0.275 | 1.24 | | Always-replan | 0.453 | 1.59 | | Always-escalate | 0.686 | 7.22 | | Binary cascade | 0.636 | 2.56 | | CRC (B=2.56 m$) | 0.717 | 2.56 | | CRC argmax (λ=0) | 0.817 | 5.51 |
Key takeaways:
1. Always-escalate spends $7.22m for only 0.686 — big models fail on 31.4% of cases too. 2. CRC at a $2.56m budget hits 0.717, beating always-escalate's solve rate at 35% of the cost. 3. The unconstrained router (λ=0) reaches 0.817 — 13 points above always-escalate, at 24% lower cost.
A revealing ablation: prompt-only routing fails. Zero-shot routing by Claude Sonnet 4.6, Gemini 3.1 Pro, and GPT-5.4 tops out at 0.453 (Claude), far below the fine-tuned Qwen3.5-4B (0.817). Routing signals must be learned from rollout data, not read from action descriptions.
Cross-Model Validation
To show the method isn't overfit to GPT pairs, the authors repeat the full pipeline with Gemini-2.5-Flash / Gemini-2.5-Pro (Appendix C). Conclusions hold: the three actions are complementary and CRC calibration works.
Engineering Perspective: Can This Ship?
Takeaway: From "Which Model" to "Which Action"
The decision granularity of LLM inference is shifting from "pick a model" to "pick an action." In the agent era, inference is not one call but a trajectory, and cost-quality relationships between actions are non-linear, non-monotonic, and not knowable a priori.
Failure is information, not an endpoint. Like an experienced engineer who decides whether to debug, rewrite, or ask for help based on the error type, CodeRescue's router learns from failure features which recovery is most cost-effective.
And CRC offers another lesson: when you can't guarantee quality, at least guarantee cost. Solve rate depends on model versions, prompts, data distributions; cost is certain, measurable, controllable. Guaranteeing what can be guaranteed — and admitting what can't — is more credible than systems claiming "optimal quality."
Open Source
Code: https://github.com/Qijia-He/agent-budget-control
*Next time your coding agent fails, don't rush to escalate. Ask first: what is this failure telling you?*