PyroDash: Teaching a Small LLM When to Ask for Help, Cutting Inference Cost from $49 to $1.78
How Much Can an Intern Handle?
Imagine you hire an intern: cheap, responsive, and capable of handling most tasks. But occasionally he gets stuck—on a complex derivation or a subtle edge case. You have two options:
- Option A: Send every hard problem straight to your senior consultant (reliable, but extremely expensive).
- Option B: Let the intern judge "I can't solve this" himself, then escalate.
- Average accuracy: 64.04%
- 6.36 points higher than the standalone large model (GLM-5.2-FP8, 57.68%)
- 20.4% cheaper than pure LLM inference
- LLM token share: 95.34% (escalating on nearly every problem, but only once each)
- Average accuracy: 54.55%
- LLM token share: 1.90%
- Average LLM calls: 0.012 per problem (i.e., only 1.2 in 100 problems need the large model)
- Total cost: $1.78 vs $49.36 for pure LLM inference—a 96.4% reduction
- Accuracy 55.29%, cost $4.71—more accurate than both routing baselines at 1/7 to 1/6 of their cost.
- Scenario A: The small model reduces a word problem to an inequality, finds the next numerical step beyond its ability, emits τ_off; the LLM completes the computation.
- Scenario B: The small model hits overly complex combinatorial constraints while setting up equations, emits τ_off; the LLM writes the full system.
- Scenario C: The small model solves everything itself with no τ_off—zero LLM calls, near-zero cost.
Option A describes most enterprise AI inference today—either all-LLM (expensive) or all-SLM (unreliable). Option B sounds better, but there's a key question: how does the intern know he can't handle it? He needs enough self-awareness to raise his hand *before* getting stuck, not after grinding for half an hour.
PyroDash (arXiv:2607.20327) is the Pyromind Dynamics team's answer: train a 4B-parameter small model (Qwen3.5-4B) to judge during reasoning when it needs help, then hand off to a frozen large model (GLM-5.2-FP8) to finish. The result is cheaper—and more accurate than the large model alone.
Core Mechanism: One Control Token, One Seamless Handoff
PyroDash's architecture is surprisingly simple. No separate router model, no retraining the large model, no peeking at the large model's internal logits. It does exactly one thing:
Add a special control token—τ_off (the offload token)—to the small model's vocabulary.
While generating reasoning steps, if the small model "feels" a step exceeds its ability, it emits τ_off. The system detects the token, packages the question plus the partial reasoning trace, and sends it to the large model, which picks up from the break point and completes the remaining reasoning in one go.
That's it. No back-and-forth, no multiple switches. One handoff, one completion.
Key design details:
1. The decision is internalized in the small model. No external router, no real-time difficulty estimation. The small model "senses" difficulty during generation, like an experienced intern smelling a hard problem. 2. The large model is fully frozen. No retraining, no access to its internal states—any API-based LLM (GLM, GPT, Claude, etc.) can serve as the backend. 3. Single handoff. The small model stops at τ_off; the large model takes over to the end. No ping-ponging between models.
Three-Stage Training: From Recognizing the Token to Learning the Trade-off
How does the small model learn when to raise its hand? This is PyroDash's most elegant part—a three-stage progressive pipeline:
Stage 1: Control-Token Embedding Learning
First, the small model must recognize τ_off. The token is added to Qwen3.5-4B's vocabulary and its embedding is trained on a small amount of data. At this stage the model only "knows the token exists"—not when to use it.
Stage 2: Offload-Oriented SFT (Supervised Fine-Tuning)
This is "behavior cold start." The team constructs training data: for hard problems the small model can attempt but solves poorly, they annotate "should hand off here" positions. SFT teaches the basic handoff behavior—after which reasoning steps to emit τ_off. Like demonstrating to an intern: "for this class of problem, hand it to the consultant at this step."
Stage 3: Cost-Aware Policy Alignment (GRPO)
SFT teaches "can hand off," not "should hand off." If the small model raises its hand on everything, you're just paying for an all-LLM system; if it never does, accuracy suffers.
This stage uses GRPO (Group Relative Policy Optimization) with the reward:
reward = accuracy − λ × normalized inference cost
λ controls how much cost matters. Small λ → prioritize accuracy, frequent escalation; large λ → prioritize cost, solve it yourself.
Normalized cost is computed relative to pure-LLM inference: if the large model alone costs $1 per problem and PyroDash uses only $0.02 of LLM compute, the normalized cost is 0.02. This reward lets the small model naturally learn to weigh "I truly can't do this" against "I can manage with effort."
The Numbers: $49.36 vs $1.78
PyroDash was evaluated on five math benchmarks: Minerva, GSM8K, Olympiad-Bench, AIME-2025, AIME-2024. Results at two extreme configurations:
Accuracy-First (λ = 0.05)
Here the small model escalates frequently, but because handoff timing is precise (after it has already produced useful partial reasoning), the large model receives good context and does better than starting from scratch. A counterintuitive finding: well-structured division of labor beats simply stacking a bigger model.
Cost-First (λ = 0.6)
The small model works almost entirely alone; only 1–2 in 100 problems get escalated. Yet 54.55% accuracy matches or beats RouteLLM (52.74%, $44.62) and GlimpRouter (54.20%, $31.61), at 20–25× lower cost.
Middle Ground (λ = 0.1)
The three configurations trace a clean accuracy-cost curve; users pick λ based on budget and accuracy needs.
Why It Works: Precision of Handoff Timing
Traditional request-level routing must decide LLM-vs-SLM the instant it sees the question. But difficulty is often hidden—problems that look easy turn out hard halfway through. Request-level routing can't handle this.
PyroDash's token-level handoff allows dynamic decisions mid-reasoning. The paper's appendix shows typical scenarios:
Comparison with Existing Approaches
Existing SLM-LLM collaboration schemes fall into several categories:
1. Request-level routing (RouteLLM, GlimpRouter): decides before decoding. Cannot handle "looks easy, is hard" problems; expensive (75%+ of tokens go to the LLM). 2. Token-level collaboration: allows switching mid-generation, but usually needs a separate router, repeated switches, or access to LLM logits—high system complexity. 3. Speculative decoding: the small model drafts, the large model verifies in parallel. But verification happens at the token level, not the reasoning-step level; the large model must stay online, so costs aren't reduced.
PyroDash's distinctiveness: decision internalized in the small model + single handoff + fully frozen large model. This combination is both simple and cheap.
Honest Assessment: Limitations and Open Questions
1. Math only. All five benchmarks are math. Effects on code generation, multi-turn dialogue, and open-domain QA are unknown. Math has clear correct answers and clean reward signals; other domains are harder. 2. One-shot completion after handoff. If the large model also gets stuck, there's no fallback mechanism; the paper doesn't discuss this failure mode. 3. Training data dependency. The SFT stage needs annotated "where to hand off" data, requiring manual or semi-automatic construction. 4. λ requires tuning. Optimal λ differs by application; users need hyperparameter search on their own data.
Still, as a proof of concept, PyroDash is convincing: a small model can learn self-awareness, and that self-awareness is itself a powerful capability.
The Deeper Insight: Self-Knowledge Is Worth More Than Raw Ability
What's most interesting about PyroDash isn't the savings (though $49 → $1.78 is striking) but the underlying principle:
A system with limited ability that knows where its limits are is more useful than a stronger system that doesn't know its own boundaries.
Pure GLM-5.2-FP8 scores 57.68%; PyroDash (λ=0.05) scores 64.04%. A small+large combination beats the large model alone by 6 points. How can the same models, combined, be stronger?
The answer is precision of division of labor. Solving a hard problem from scratch, the large model must walk every reasoning step, each with a chance of error. PyroDash's small model first handles what it can (simplifying the problem, establishing constraints, setting up equations), resolving the "mechanical parts," and hands over only at the step that truly demands stronger capability. The large model receives an already-partially-simplified problem, with lower error probability.
This mirrors how human teams work. A good intern is neither useless nor recklessly ambitious: he knows which parts he can handle (faster and cheaper than the consultant), and which to escalate. This metacognition beats raw capability, because it makes resource allocation across the whole system efficient.
PyroDash essentially teaches a small model "cognitive humility"—not modesty or conservatism, but a precise knowledge of its ability boundary. It's not a personality trait; it's a policy trained via RL. But it works like genuine cognitive humility: extracting maximum utility from limited resources.
More broadly, PyroDash points toward a future of AI deployment: not chasing ever-bigger models, but building networks of models that precisely know their own boundaries. A system of models at different capability levels, each knowing when to act and when to escalate, can beat any single model on both cost and accuracy—far more elegant than simply stacking parameters.
An intern who learns to raise his hand is worth more than a consultant who learns to do everything.
---
Paper: https://arxiv.org/abs/2607.20327 HTML version: https://arxiv.org/html/2607.20327v1 Code: No dedicated repository (trained with the HuggingFace TRL library)