CodeRescue: Budget-Calibrated Recovery Routing for Coding Agents — Reflect, Replan, or Escalate?

Forum topic · 小凯 · 2026-07-22

Summary

CodeRescue (arXiv: 2607.19338, from University of Washington, NYU, and ByteDance) rethinks how coding agents recover from failures. Instead of the classic cascade paradigm that treats a cheap model's failure as a signal to escalate to a stronger model, it frames recovery as routing among three heterogeneous actions: reflect (cheap model fixes using error feedback), replan (cheap model restarts from a new plan), and escalate (strong model solves from scratch). Experiments show these actions are not strictly ordered: 28% of failures are only solvable by cheap actions, 45% only by escalation, and 27% by both. A supervised router fine-tuned from Qwen3.5-4B on ~27,300 problems across five coding benchmarks selects the cheapest successful action, and a Conformal Risk Control layer adapts it to any budget via a single cost-penalty parameter lambda without retraining. At a $2.56m budget, CRC achieves a 0.717 solve rate—beating always-escalate (0.686) at 35% of its cost; the unconstrained router reaches 0.817. Code and data pipelines are open-sourced.

After a Coding Agent Fails: Reflect, Replan, or Escalate? CodeRescue's Three-Action Routing Solution

> Paper: CodeRescue: Budget-Calibrated Recovery Routing for Coding Agents > Authors: Qijia He, Jiayi Cheng, Chenqian Le, et al. (University of Washington / NYU / ByteDance) > arXiv: 2607.19338 > Code: https://github.com/Qijia-He/agent-budget-control

A Familiar Scenario

It's 2 AM. You ask a coding agent to implement a parser. A cheap model (say, GPT-5.4-nano) produces a first version, and the tests fail—an edge case isn't handled.

The agent now has three options:

1. Reflect: feed the error logs back and let the cheap model fix it itself. Cheapest, but might not succeed. 2. Replan: have the cheap model discard the old approach and start from a new plan. Slightly costlier, but a fresh idea might pass. 3. Escalate: hand the problem to a big model like GPT-5.4. Most expensive, but most capable.

Which would you choose?

If your instinct is "just escalate, one and done"—congratulations, you share the intuition of most existing cost-aware routing systems. And that's exactly the intuition this paper challenges.

The Problem with the Old Paradigm: Treating Failure as "Time to Swap in a Bigger Model"

The dominant cost-aware LLM inference paradigm in recent years has been the cascade: try a cheap model first, escalate to a stronger one if it fails. FrugalGPT, RouteLLM, HybridLLM, and C3PO are all variants of this route.

The paradigm carries a hidden assumption: the cheap model failed, so its capability is insufficient and stronger parameters are needed.

But coding agents have a special property, which the paper makes explicit: failure itself is information. Compiler errors, failing tests, timeouts—these execution feedback signals turn an underdetermined problem into a diagnosed repair problem. Given one more cheap call with the right feedback, the small model may well fix it. Escalating immediately wastes the context the cheap model has already built up.

As the paper puts it:

> Spending another cheap call with the right feedback may exploit the small model more effectively than immediately escalating.

This is not a question of "which model is stronger" but of "which recovery action is more cost-effective."

Three Actions, Not a Ladder

The core innovation is expanding the post-failure decision from a binary "escalate or not" into routing over three heterogeneous actions:

| Action | Model | Behavior | Relative cost | |--------|-------|----------|---------------| | reflect | cheap model | local repair using error feedback | lowest | | replan | cheap model | discard plan, re-plan from scratch | low-mid | | escalate | strong model | solve from scratch with a strong model | highest |

The key insight: these three actions are not strictly ordered. The data shows:

28% of failures are solvable only by cheap actions (reflect/replan)
45% are solvable only by escalate
27% are solvable by both cheap actions and escalate

This means the one-way ladder "cheap action → escalate" is wrong. Some problems escalate can't solve, but cheap actions can. That finding alone is worth remembering—capability is not a monotonic function of cost.

Training a Router: Learning to Read the Situation

With three actions on the table, the next question is: how to choose?

Rather than using rules, the paper trains a supervised recovery router:

1. Collect rollout data: on 5 coding benchmarks (APPS, TACO, BigCodeBench, LiveCodeBench, CodeContests), ~27,300 problems; run GPT-5.4-nano first, then run all three recovery actions on failures, recording which actions succeed. 2. Label: for each failed problem, label the "cheapest successful action." Problems unsolvable by all three are excluded from training. 3. Train: full fine-tuning of Qwen3.5-4B; input is (problem + execution verdict + stderr), output is a probability distribution over the three actions.

One notable detail: evaluation uses solve rate, not classification accuracy. Since multiple actions may solve the same problem, predicting a different-but-still-successful action shouldn't count as an error. This design avoids "either/or" label noise.

Another key finding: metadata prefixes matter. Giving the model just (problem + error message) isn't enough; adding metadata like source, difficulty, and algorithm tags lifts solve rate from 0.656 to 0.697. Execution feedback alone is insufficient to decide the right action—distributional information about the problem is also needed.

Conformal Risk Control: One Knob for the Budget

So far the router is just an ordinary classifier. The paper's real highlight is the next step: making the same router adapt to different budgets.

The traditional approach bakes the budget into the training objective (e.g., RACER), requiring retraining for each budget. CodeRescue instead uses Conformal Risk Control (CRC), an elegantly simple idea.

The router scores each action $$s(a|x)$$ ; instead of a plain argmax, a cost penalty $\lambda$ is introduced:

\pi_\lambda(x) = \arg\max_a \{ s(a|x) - \lambda \cdot c(a,x) \}

$\lambda = 0$ : ignore cost entirely, maximize success only (most aggressive)
Larger $\lambda$ : favor cheaper actions (more conservative)

$\lambda$ is the "budget knob." Changing budgets requires no retraining—just tune this one parameter.

But $\lambda$ is not chosen arbitrarily. CRC provides a finite-sample guarantee: on an exchangeable calibration set, pick the smallest $\lambda$ such that

\frac{n \cdot \hat{C}_n(\lambda) + c_{\max}}{n+1} \leq B

where $$B$$ is the user-specified budget, $c_{\max}$ the worst-case cost, and $$n$$ the calibration set size. Under the exchangeability assumption, this mathematically guarantees that the expected cost of future samples stays under $$B$$ .

In plain terms: tell the system "average recovery cost should not exceed 2.56 millidollars," and CRC finds a $\lambda$ that provably keeps long-run average cost under that number.

There's a subtle engineering philosophy here: CRC controls cost, not solve rate. Solve rate is an empirical property learned by the router; CRC makes no promise about it. That's honest—you can't statistically guarantee correctness, but you can guarantee a spending ceiling.

The Numbers: 35% of the Cost, More Solve Rate Than Always-Escalate

Main results (GPT-5.4-nano / GPT-5.4 pair, 360 held-out test problems):

| Strategy | Solve Rate | Avg. cost (m$) | |----------|-----------|----------------| | Always-reflect | 0.275 | 1.24 | | Always-replan | 0.453 | 1.59 | | Always-escalate | 0.686 | 7.22 | | Binary cascade | 0.636 | 2.56 | | CRC (B=2.56 m$) | 0.717 | 2.56 | | CRC argmax (λ=0) | 0.817 | 5.51 |

Key readings:

1. Always-escalate spends 7.22 m$ yet solves only 0.686. Big models aren't omnipotent; 31.4% of failures remain unsolved even by escalation. 2. CRC at a 2.56 m$ budget reaches 0.717—higher than always-escalate's solve rate at just 35% of the cost. This is the paper's headline result. 3. The unconstrained router (λ=0) reaches 0.817, 13 points above always-escalate at 24% lower cost.

An interesting ablation: prompt-only routing doesn't work. Asking Claude Sonnet 4.6, Gemini 3.1 Pro, or GPT-5.4 to route zero-shot tops out at 0.453 (Claude), far below the fine-tuned Qwen3.5-4B (0.817). The routing signal is not written in the action descriptions—it must be learned from rollout data.

Cross-Model Validation: Not GPT-Specific

To show the method isn't overfit to the GPT pair, the authors rerun the entire pipeline with Gemini-2.5-Flash / Gemini-2.5-Pro (Appendix C). The conclusions hold: the three actions are complementary and CRC calibration remains effective. This cross-model check matters—if it only worked on GPT, the method's value would be discounted.

Engineering Perspective: Can This Ship?

From an engineering standpoint, CodeRescue has several appealing properties:

1. Train once, deploy many times. The router trains once; the CRC calibration table is precomputed once. Switching budgets is a table lookup for $\lambda$ , O(K) time (K breakpoints, typically a few dozen). Production-friendly: different customers, time slots, and SLAs can share one model.

2. The router is tiny. Full fine-tuning of Qwen3.5-4B; a 4B-parameter router's inference cost is negligible next to a single escalation costing millidollars.

3. Extensible action set. The paper picks reflect/replan/escalate as a "minimal first-order recovery set," but the framework doesn't limit the action count. You can add "search docs," "ask a human," "run static analysis," etc., as long as you can define costs and collect rollout data.

4. Honest limitations. The paper explicitly notes: only single-step recovery is modeled (real agents may recover over multiple rounds); the cheapest-successful label is a proxy, not calibrated probability; CRC controls cost, not solve rate. That honesty makes the conclusions more credible.

My Take: From "Which Model" to "Which Action"

This paper points to a broader trend: the decision granularity of LLM inference is shifting from "pick a model" to "pick an action."

Early cost-aware systems asked: "Should this problem go to a small or large model?" CodeRescue asks: "After this failure, should I reflect, replan, or escalate?"

Behind this shift is a recognition: in the agent era, inference is not a single call but a trajectory. Every decision point on the trajectory offers multiple actions, and the cost-quality relationships among them are nonlinear, non-monotonic, and not knowable a priori. You must learn them from data.

Deeper still: failure is information, not an endpoint. In the classic cascade, failure triggers exactly one action (escalation). In CodeRescue, failure triggers a routing decision, and the router learns from failure characteristics which recovery is most cost-effective. This matches human engineer intuition—an experienced developer doesn't call in the expert on every failure; they read the error and decide whether to debug it themselves, rewrite, or ask for help.

The CRC layer adds another engineering lesson: when you cannot guarantee quality, at least guarantee cost. Solve rate depends on too many factors (model versions, prompts, data distributions) to guarantee statistically. But cost is definite, measurable, and controllable. CRC cleverly places the guarantee on the cost side and lets empirical data speak for quality. This attitude—"guarantee what can be guaranteed, admit what cannot"—is far more credible than systems claiming "optimal quality."

Open-Source Code

The paper's code is open-sourced: https://github.com/Qijia-He/agent-budget-control

The repository is well organized and includes:

data_generation/: rollout collection pipeline (including agent implementations of reflect/replan/escalate)
sft_runs/: router training configs (LLaMA-Factory format)
conformal/: CRC calibration and evaluation scripts

For teams looking to reproduce or extend, the repo is quite complete. In particular, the three recovery action implementations in data_generation/agents/ can serve directly as reference designs for coding agents.

---

Paper: https://arxiv.org/abs/2607.19338 Code: https://github.com/Qijia-He/agent-budget-control

---

*The next time your coding agent fails, don't rush to escalate. First ask: what is this failure actually telling you?*