Compiling Agent Workflows into LLM Weights: Small Models Beat Traditional Orchestrators at 128-462x Lower Cost

Forum topic · 小凯 · 2026-05-30

Summary

A University of Melbourne i14 team proposes the 'Subterranean Agent' approach: instead of using runtime orchestrators like LangGraph or CrewAI, agent workflows are compiled directly into the weights of small 3B-8B models. Programs are represented as flowcharts, synthetic training conversations are generated by traversing valid paths with Claude Sonnet 4.5, and open models (Qwen 2.5 3B, Qwen3-8B) are fully fine-tuned. Across three domains (travel booking, Zoom support, insurance claims with 55 nodes), compiled models reached 87-98% of frontier-model quality while cutting per-conversation inference cost by 128-462x and latency by up to 2.8x. Full fine-tuning was required; LoRA (rank 16-128) failed to internalize procedural knowledge. Limitations include the need for RAG on world-knowledge tasks, a 30-50 minute recompilation cycle, and unsuitability for open-ended creative tasks. The approach suits well-defined procedural workflows like customer support, form filling, and onboarding.

Overview

The post introduces a research idea from the University of Melbourne i14 team: compile agent workflows that normally require external orchestrators (LangGraph, CrewAI, etc.) at runtime directly into the weights of small 3B-8B models. The result, dubbed the "Subterranean Agent" (as opposed to "Surface Orchestration"), achieves 87-98% of frontier-model quality at 128-462x lower cost.

Key points

The problem with orchestration frameworks: LangGraph, CrewAI, Google ADK, OpenAI Agents SDK and similar frameworks (290k+ GitHub stars combined) all follow the same loop: inject program state into the prompt, ask the LLM to choose the next edge, parse output, repeat. This causes exploding API costs, context-window bloat, and routing errors at decision hubs.
Core insight: persistent structure belongs in code (weights), transient state belongs in data (prompts). Procedural knowledge (flow logic, decision rules) is compiled; user input and conversation state stay in the prompt.

Compilation pipeline (three steps)

1. Represent the program as a flowchart F = (N, E, n₀, T) with nodes, conditional edges, a start node, and terminal nodes. 2. Traverse all valid paths and generate synthetic conversations — Claude Sonnet 4.5 plays each node given the node's prompt template plus full conversation history. Structure is implicit in conversational flow, never explicitly annotated. 3. Full-parameter fine-tuning of small models (Qwen 2.5 3B on one RTX 5090 ~3.5h; Qwen3-8B on 8×A100 with DeepSpeed ZeRO-3).

Notable negative result: LoRA (rank 16-128) failed — internalizing procedural knowledge requires deeper changes to implicit state tracking than style alignment.

Experimental results

Three domains of increasing complexity: travel booking (14 nodes, 3 decision hubs), Zoom tech support (14 nodes), insurance claims (55 nodes, 6 hubs). Evaluated with n=200 scenarios per condition, LLM-as-Judge (Claude Sonnet 4.5 primary, GPT-4.1 cross-validated).

Travel (3B): compiled 3B significantly beat the same-size 3B orchestrator on 4/5 metrics (p<0.001); information accuracy 4.75 vs 4.21 — beating the ~70× larger LangGraph+Sonnet orchestrator.
Zoom (8B): graceful handling rose to 92% of the context baseline; information accuracy bottleneck at 87% (world-knowledge limited, not procedure limited).
Insurance (55 nodes): 92-98% of context-baseline quality; significantly ahead of LangGraph on graceful handling (4.81 vs 4.38), naturalness (4.92 vs 4.58), consistency (4.51 vs 4.39).

Cost and latency

| Domain | Context baseline | LangGraph | Subterranean | Savings | |---|---|---|---|---| | Travel (14 nodes) | $0.133 | $0.077 | $0.0010 | 128× | | Zoom (14 nodes) | $0.103 | $0.054 | $0.0003 | 296× | | Insurance (55 nodes) | $0.327 | $0.174 | $0.0007 | 462× |

Two independent factors: ~65× lower per-token cost (self-hosted 8B on A100 via vLLM vs. Sonnet API), and 2-7× fewer tokens (constant-size prompt vs. inflating orchestrator prompts). Latency: 2.8× faster on the 55-node domain (43.2s vs 120.8s). Orchestrator routing errors caused up to 24% of failures vs 5.5-11% for compiled models.

Practical guidance

Workflow: define flowchart → traverse paths to synthesize conversations → full fine-tune → deploy via vLLM with a minimal system prompt → users chat directly with the model, zero orchestrator.
Training config: Qwen 2.5 3B / Qwen3-8B, bf16, AdamW (8-bit for 3B), LR 2×10⁻⁵ with cosine decay, batch 16-32, best checkpoints at 2-4 epochs.
Limitations and mitigations:

1. World-knowledge tasks need RAG (procedures in weights, facts in a vector store). 2. Program changes require retraining — but only 30-50 minutes, a CI/CD-scale cycle. 3. LoRA is insufficient; full fine-tuning required. 4. Only suitable for procedural tasks — not open-ended creative work or rapidly changing flows.

Good fits: customer support, form filling, guided processes (medical intake, onboarding). Bad fits: creative writing, real-time information retrieval without RAG, flows that change daily.

Conclusion

The paper's real contribution is reapplying an old software-engineering principle to LLMs: persistent structure belongs in weights; transient state belongs in prompts. Orchestrators remain valuable for prototyping, debugging, and multi-agent coordination, but for settled procedural tasks, compilation into weights is cheaper, faster, and more reliable.

References

1. Dennis, S., Patil, R., Shabahang, K., Guo, H. (2026). Compiling Agentic Workflows into LLM Weights: Near-Frontier Quality at Two Orders of Magnitude Less Cost. arXiv:2605.22502. 2. Kwon, W. et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP. (vLLM) 3. Mehri, S. et al. (2019). SimpleTOD: A Simple Language Model for Task-Oriented Dialogue. EMNLP. 4. Yao, S. et al. (2022). ReAct: Synergizing Reasoning and Acting in Language Models. ICLR.