PyroDash: Teaching a Small Model When to Raise Its Hand, Cutting Inference Cost from $49 to $1.78

How Much Can an Intern Handle?

Imagine you've hired an intern. Low salary, good attitude, fast response — handles most of the work. But occasionally they hit a wall on a hard problem — a complex mathematical derivation, a subtle edge case. You have two options:

• Option A: Forward every hard problem directly to your senior consultant (extremely expensive, but reliable).
• Option B: Let the intern judge "I can't solve this one" themselves, then escalate.

Option A is the status quo of enterprise AI inference — either use a large model (LLM) for everything and burn money, or a small model (SLM) for everything and fail. Option B sounds more sensible, but there's a critical question: how does the intern know they can't handle it? They need enough self-awareness to raise a hand *before* getting stuck, not after half an hour of flailing.

PyroDash (arXiv:2607.20327) is the Pyromind Dynamics team's answer: teach a 4-billion-parameter small model (Qwen3.5-4B) to judge during reasoning when it should ask for help, then pass the baton to a frozen large model (GLM-5.2-FP8) to finish. The result saves money — and is *more accurate* than the large model alone.

Core Mechanism: One Control Token, One Seamless Handoff

PyroDash's architecture is surprisingly simple. No separate router model, no retraining the large model, no peeking at the LLM's internal logits. It does one thing:

Add a special control token — τ_off (offload token) — to the small model's vocabulary.

While the small model generates reasoning steps, if it "feels" a step is beyond its ability, it emits τ_off. The system detects this token, packages the problem plus the small model's partial reasoning trace, and sends it to the large model, which continues from the breakpoint and completes the remaining reasoning in one go.

That's it. No ping-ponging, no multiple switches. One handoff, one completion.

Key design details:

1. The decision is internalized in the small model. No external router, no real-time difficulty estimation. The small model "senses" difficulty during generation — like an experienced intern who can smell a hard problem. 2. The large model is fully frozen. No retraining, no access to its internal states. Any API-based LLM works as the backend — GLM, GPT, Claude, etc. 3. A single handoff. The small model stops at τ_off; the large model takes over to the end. No back-and-forth passing.

Three-Stage Training: From Knowing the Token to Learning the Trade-off

How does the small model learn *when* to raise its hand? This is PyroDash's most elegant part — a three-stage progressive training pipeline:

Stage 1: Control Token Embedding Learning

First, the model must recognize τ_off. The token is added to Qwen3.5-4B's vocabulary and its embedding is trained on a small amount of data. At this point the model only "knows the token exists" — not when to use it.

Stage 2: Offload-Oriented SFT

This is behavioral cold-start. The team constructs training data: for problems the small model can solve but poorly, they annotate "where the handoff should happen." SFT teaches the basic handoff behavior — after which reasoning steps to emit τ_off. Like showing the intern: "on problems like this, once you reach this step, escalate."

Stage 3: Cost-Aware Policy Alignment (GRPO)

SFT taught the model it *can* hand off; this stage teaches it whether it *should*. If it raises its hand on everything, you're just paying for the LLM; if it never does, accuracy suffers.

GRPO (Group Relative Policy Optimization) is used with the reward:

Reward = accuracy − λ × normalized inference cost

λ controls how much the model "cares about money." Small λ favors accuracy (more frequent escalation); large λ favors cost (more self-reliance). The normalized cost is measured relative to pure-LLM inference — if the LLM alone costs $1 per problem and PyroDash uses only $0.02 of LLM compute, the normalized cost is 0.02. This reward lets the small model naturally learn to trade off "I really can't do this" vs. "I can solve this with more effort."

The Numbers: $49.36 vs $1.78

PyroDash was evaluated on five math reasoning benchmarks: Minerva, GSM8K, Olympiad-Bench, AIME-2025, AIME-2024. Results at two extreme configurations:

Accuracy-First (λ=0.05)

• Average accuracy: 64.04%
• +6.36 points over the pure large model (GLM-5.2-FP8, 57.68%)
• 20.4% cheaper than the pure LLM
• LLM token share: 95.34% (escalates on nearly every problem, but only once)

Here the small model escalates constantly — but because the handoff timing is precise (it hands over only after doing useful partial reasoning), the LLM receives good context and outperforms solving from scratch. A counterintuitive finding: well-structured division of labor beats simply stacking a large model.

Cost-First (λ=0.6)

• Average accuracy: 54.55%
• LLM token share: 1.90%
• Average LLM invocations: 0.012 per problem (only ~1.2 out of 100 problems need the LLM)
• Total cost: $1.78 vs $49.36 for the pure LLM — a 96.4% reduction

Here the small model works almost entirely alone. Yet 54.55% accuracy matches or beats RouteLLM (52.74%, $44.62) and GlimpRouter (54.20%, $31.61) at 20-25x lower cost.

Middle Ground (λ=0.1)

• Accuracy 55.29%, cost $4.71
• More accurate than both routing baselines at 1/7 to 1/6 of their cost

The three configurations trace a clean accuracy-cost curve; users pick λ based on budget and accuracy needs.

Why It Works: Precise Handoff Timing

Traditional request-level routing must decide which model to use the instant it sees the query. But many problems are deceptively hard — they look easy until you're halfway through. Request-level routing can't handle that.

PyroDash's token-level handoff allows dynamic decisions mid-reasoning. The paper's appendix shows typical scenarios:

• Scenario A: The small model reduces a word problem to an inequality, finds the numerical computation beyond its ability, emits τ_off. The LLM finishes the computation.
• Scenario B: The small model finds the combinatorial constraints too complex while setting up equations, emits τ_off. The LLM completes the full equation system.
• Scenario C: The small model solves the whole problem itself, emitting no τ_off. Zero LLM calls, near-zero cost.

The small model learns a form of "metacognition" — not simply "I'm stuck, help," but "*this specific step* needs stronger capability." This emerges naturally from the GRPO reward signal, not hand-designed rules.

Comparison with Other Approaches

Existing SLM-LLM collaboration schemes fall into several categories:

1. Request-level routing (RouteLLM, GlimpRouter): decides before decoding. Cannot handle deceptively hard problems. Expensive (75%+ of tokens go to the LLM). 2. Token-level collaboration: allows mid-generation switching, but usually needs a separate router, repeated switching, or access to LLM logits. High system complexity. 3. Speculative decoding: the small model drafts, the large model verifies in parallel. But verification is at the token level, not the reasoning-step level, and the LLM must stay online throughout — no cost savings.

PyroDash's uniqueness: decision internalized in the small model + single handoff + fully frozen LLM. This combination makes it both simple and cheap.

Honest Assessment: Limitations and Open Questions

The paper candidly discusses limitations:

1. Tested only on math reasoning. All five benchmarks are math. Effectiveness on code generation, multi-turn dialogue, and open-domain QA is unknown. Math has clear correct answers and clean reward signals; other domains make reward design harder. 2. One-shot completion after handoff. If the LLM also gets stuck, there's no fallback mechanism. The paper doesn't discuss this failure mode. 3. Training data dependency. The SFT stage requires data annotated with "where to hand off," which itself needs manual or semi-automatic construction. 4. λ requires tuning. Optimal λ varies by application; users need hyperparameter search on their own data.

But as a proof of concept, PyroDash is persuasive: a small model can learn self-awareness — and that self-awareness is itself a powerful capability.

The Deeper Insight: Self-Knowledge Is Worth More Than Capability

What's most interesting about PyroDash isn't the savings (though $49 to $1.78 is striking) but the underlying principle it reveals:

A system with limited capability that knows where it's limited is more useful than a stronger system that doesn't know its boundaries.

The pure LLM (GLM-5.2-FP8) scores 57.68%; PyroDash (λ=0.05) scores 64.04%. The small-plus-large combo beats the large model alone by 6 points. Counterintuitive — how can combining the same models make it *stronger*?

The answer is precision of division of labor. Solving a hard problem from scratch, the LLM must walk through every reasoning step, each with error probability. PyroDash's small model first handles what it can (simplifying the problem, establishing constraints, setting up equations) — the "mechanical parts" — and only hands over the step that truly requires strong capability. The LLM receives an already-partially-simplified problem and is less likely to err.

This mirrors how human teams work. A good intern isn't useless at everything, nor recklessly attempting everything. They know which parts they can handle (faster and cheaper than the senior consultant), and which parts to escalate. That "metacognition" is worth more than raw capability, because it makes resource allocation across the whole system more efficient.

PyroDash essentially teaches a small model "cognitive humility" — not modesty or conservatism, but precisely knowing where its ability boundaries lie. It's not a personality trait; it's a policy trained through RL. But its effect is the same as genuine cognitive humility: maximizing the utility of limited resources.

From a broader perspective, PyroDash points toward a future of AI deployment: not chasing ever-larger models, but building networks of models that precisely know their own boundaries. A system of models at different capability levels, each knowing when to act and when to escalate, beats any single model on both cost and accuracy. Far more elegant than just stacking parameters.

An intern who learns to raise their hand is worth more than a consultant who learns everything.

---

Paper: https://arxiv.org/abs/2607.20327 HTML version: https://arxiv.org/html/2607.20327v1 Code: No dedicated repository provided (trained with the HuggingFace TRL library)