PyroDash: Teaching Small LLMs When to Ask for Help, Cutting Inference Cost from $49 to $1.78

An Intern Analogy

Most enterprise LLM serving today routes every request either to a large model (expensive, accurate) or a small model (cheap, error-prone). A smarter pattern is to let the small model admit "I can't handle this step" and hand off to the large model mid-stream. The hard part is teaching the small model when to raise its hand.

PyroDash (arXiv:2607.20327) from Pyromind Dynamics answers this by training a 4B-parameter Qwen3.5-4B to judge its own limits during generation, then passing the partial trace to a frozen GLM-5.2-FP8, which finishes the reasoning in one shot. The combined system is both cheaper and more accurate than the large model alone.

Core Mechanism: One Control Token, One Seamless Handoff

The architecture is intentionally minimal:

• A new vocabulary token tau_off (offload token) is added to the small model.
• During generation, if the small model decides a step exceeds its competence, it emits tau_off.
• The system immediately packages the original question plus the partial reasoning and sends it to the large model, which continues from the handoff point to the end.

Key design points:

1. Decision lives inside the small model. No external router, no difficulty classifier, no latency overhead. 2. Large model is frozen. No retraining, no logits access. Any API-served model (GLM, GPT, Claude) can serve as the backend. 3. Single handoff per request. No back-and-forth, no ping-pong. The small model stops at tau_off; the large model finishes.

Three-Stage Training: From Token Awareness to Cost-Aware Strategy

Stage 1 — Embedding learning for the control token

The team adds tau_off to Qwen3.5-4B's vocabulary and trains its embedding with a small dataset so the model knows the token exists and what it conceptually means.

Stage 2 — Offload-oriented SFT (behavioral cold start)

Training data is constructed by annotating handoff positions on problems the small model solves partially but poorly. SFT teaches the model the basic handoff behavior — after which kinds of reasoning steps it should emit tau_off.

Stage 3 — Cost-aware policy alignment (GRPO)

SFT teaches *when it can hand off*; GRPO teaches *when it should*. The reward is:

reward = accuracy − lambda × normalized_inference_cost

where cost is measured relative to a pure large-model run. Tuning lambda shifts the model along the accuracy-cost frontier:

• small lambda: bias toward accuracy, hand off often
• large lambda: bias toward cost efficiency, self-solve aggressively

Results: $49.36 vs $1.78

Across five math reasoning benchmarks (Minerva, GSM8K, Olympiad-Bench, AIME-2025, AIME-2024):

Accuracy-tuned (lambda = 0.05)

• Average accuracy: 64.04%
• Beats GLM-5.2-FP8 alone (57.68%) by 6.36 points
• ~20.4% cheaper than the large model
• 95.34% of tokens come from the large model, but handoffs are precisely timed so the LLM benefits from the small model's partial progress

Cost-tuned (lambda = 0.6)

• Average accuracy: 54.55%
• Large-model token share: 1.90%
• Average large-model calls per problem: 0.012 (~1.2 out of 100 problems)
• Total cost: $1.78 versus $49.36 for the large model alone, a 96.4% reduction
• Matches or slightly beats RouteLLM (52.74%, $44.62) and GlimpRouter (54.20%, $31.61) on accuracy at 20-25x lower cost

Mid-range (lambda = 0.1)

• 55.29% accuracy at $4.71 — more accurate than either routing baseline at 1/6 to 1/7 the cost

These three points trace a clean accuracy-cost Pareto curve that users can navigate by tuning lambda.

Why It Works: Token-Level Handoff Beats Request-Level Routing

Request-level routers must classify difficulty before generation begins and fail on "looks easy, turns out hard" problems, driving large-model token shares above 75%. PyroDash's token-level handoff lets the small model defer the decision until it actually encounters the hard step. The paper shows qualitative examples:

• The small model reduces a word problem to an inequality, senses the numeric computation is beyond it, emits tau_off; the LLM completes the calculation.
• The small model starts setting up equations, finds the combinatorial constraints too complex, hands off; the LLM writes the full system.
• The small model solves the whole problem without emitting tau_off, costing nothing extra.

This metacognitive ability — recognizing *which step* requires stronger capabilities, not merely *whether the problem* is hard — emerges from the GRPO reward rather than hand-coded rules.

Comparison to Existing Approaches

• Request-level routing (RouteLLM, GlimpRouter): decides before decoding, no response to mid-stream difficulty, high LLM share.
• Token-level collaboration: typically requires a separate router, multiple switches, or logits access; high system complexity.
• Speculative decoding: token-level verification, not step-level handoff; the LLM stays online throughout, so cost savings are limited.

PyroDash's combination of an internalized decision, a single handoff, and a frozen LLM backend is what makes it both simple and cheap.

Limitations

1. Validated only on math reasoning; performance on coding, multi-turn dialogue, and open-domain QA is unknown, and reward shaping is harder in those domains. 2. No fallback if the large model also fails partway through. 3. Stage-2 SFT requires annotated handoff positions, which itself takes manual or semi-automated work. 4. The optimal lambda is dataset-dependent and must be tuned per deployment.

The Deeper Insight

Pure GLM-5.2-FP8 scores 57.68%; PyroDash (lambda = 0.05) scores 64.04% — the small-plus-large combination is more accurate than the large model on its own. The reason is division of labor: the small model handles the mechanical parts (simplifying the problem, establishing constraints, setting up equations), and the LLM tackles only the genuinely hard step, on a partially reduced problem with a lower error surface.

This is a useful model for AI deployment more broadly: instead of endlessly scaling single models, build networks of differently-capable models that know their own boundaries. Metacognition — knowing the limits of what you know — is, in many settings, more valuable than raw capability.

---

Paper: https://arxiv.org/abs/2607.20327 HTML: https://arxiv.org/html/2607.20327v1 Code: No dedicated repository (training uses HuggingFace TRL)