Compiling Agentic Workflows into LLM Weights: An 8B Model, Single-Step Inference Replacing Seven-Layer Orchestration

Paper info

| | | |---|---| | Title | Compiling Agentic Workflows into LLM Weights: Near-Frontier Quality at Two Orders of Magnitude Less Cost | | Authors | Simon Dennis, Rivaan Patil, Kevin Shabahang, Hao Guo | | Institution | University of Melbourne | | arXiv ID | 2605.22502 | | Date | May 21, 2026 | | Categories | cs.AI / cs.LG | | Core claim | Compile multi-step agent workflows into small-model weights; no external orchestrator at runtime—an 8B model reaches 87–98% of frontier quality, costs 128–462x less, recompiles in 30–50 minutes (CI/CD level), with lower failure rates |

---

Background: A Tower of Babel

The world's LLM agent frameworks collectively hold ~290,000 GitHub stars. LangGraph, CrewAI, Google ADK, OpenAI Agents SDK, Semantic Kernel, Strands, LlamaIndex—seven frameworks, seven orchestration styles, one architecture: a scheduler sits outside the model, injecting instructions each turn, parsing output, deciding the next step.

The orchestrator is like a person standing behind the model, tapping its shoulder at every fork: "Left." "Now right." "Ask the user's budget." "Now give three options based on the budget."

This architecture has held for three years, widely assumed to be the only form an AI agent can take. Simon Dennis of the University of Melbourne asked a more fundamental question: what if no orchestrator is needed at all?

Three Architectures

• Surface orchestration. The orchestrator sits between user and LLM: inject node prompts, parse output, consult the flow graph, re-inject. Cost: extra API calls per turn, prompt bloat (workflow instructions re-stuffed into context every turn), and routing-decision failures—LLMs deciding "which edge to fire next" are not error-free.
• In-context prompting. Put the whole flow graph in the system prompt and let a frontier model navigate itself. Prior work shows near-perfect quality on procedural tasks (4.53–5.00/5.0), but every conversation needs a frontier model, every turn re-sends the graph, and private business logic is exposed to third-party APIs.
• Subterranean agent. Bake the flow graph's knowledge into weights at training time. At runtime, user and model talk directly. No orchestrator, no graph injection, no routing decisions. The model "knows" what to do—like you don't need an external manager to tell you to ask someone's name first.

The idea isn't new—SimpleTOD (2020), FireAct (2023), WorkflowLLM and Agent Lumos (2024) all proved the path works. Yet orchestration frameworks have 290k stars while compilation approaches have under 3,000 combined. Nobody had asked why.

Three Walls, Measured

Dennis attributes developer avoidance to three fears, tested on three real scenarios: travel booking (14 nodes), Zoom tech support (14 nodes), and insurance claims (55 nodes).

Wall 1: Quality

Travel booking: 14-node graph, 86 acyclic paths, 4–17 turn dialogues. Four conditions (200 scenarios each): 3B subterranean agent vs. the same 3B with an orchestrator vs. Claude Sonnet 4.5 + LangGraph vs. Claude in-context prompting.

Compilation itself is a gain. The same 3B base model, after compiling the workflow into weights, significantly outperformed explicit orchestration of the same model on task success, consistency, graceful handling, and naturalness (p < 0.001). Why? Orchestrators have three structural costs:

1. Responses are generated from local node context—no global awareness (hence the chatbot that asks your departure date three times). 2. Routing decisions are a failure source: the LangGraph orchestrator failed 24% of the time on travel booking vs. 5.5% for the subterranean agent. 3. Template injection kills conversational rhythm.

Scaling to 8B:

• Zoom support (14 nodes, 6,264 synthetic dialogues): naturalness 4.87/5 (97% of in-context prompting), graceful handling 4.62/5 (92%). Information accuracy was the biggest gap (87%)—a model-size issue, not a compilation issue. Against a LangGraph orchestrator using a 70x larger model, the 8B compiled model scored significantly higher on naturalness (4.87 vs 4.64, p < 0.001).
• Insurance claims (55 nodes, 6 decision hubs, 2,381 paths, nested loops, cross-phase dependencies): graceful handling 4.81/5, naturalness 4.92/5, consistency 4.51/5—all significantly above the LangGraph orchestrator (p < 0.001), reaching 92–98% of in-context prompting.

Wall 2: Cost

Two multiplicative layers:

• Per-token cost: self-hosted 8B on A100 (vLLM, $2.50/hr) works out to ~$0.05/M input and $0.23/M output tokens vs. Claude Sonnet 4.5's $3/$15—roughly 65x cheaper per token.
• Token volume: in-context prompting bloats tokens ~2x (14 nodes) to ~7x (55 nodes) by re-sending the graph every turn. The subterranean agent's prompt is constant-size—15 tokens—regardless of workflow complexity.

Combined results per conversation:

• Travel booking (14 nodes): 128x cheaper ($0.13 → $0.0010)
• Zoom support (14 nodes): 296x cheaper ($0.10 → $0.0003)
• Insurance claims (55 nodes): 462x cheaper ($0.33 → $0.0007); 249x vs. the LangGraph orchestrator

The advantage grows with workflow complexity. One-time compilation cost: ~$40 data generation + $10–40 fine-tuning compute = $50–80, breaking even within 500 conversations. A customer-support agent at 10,000 daily active users costs under $7/day; the LangGraph + Claude Sonnet 4.5 equivalent: $1,740.

Wall 3: Flexibility

30–50 minutes from changed flow graph to deployed model: data generation 15–30 min (parallel Claude API traversal), fine-tuning 10–15 min (8×H200, data-parallel full-precision AdamW, 12 epochs), evaluation 5–15 min. On a single A100 80GB: about 3 hours—still "change it in the morning, ship it in the afternoon."

Compiler, Not Interpreter

The most striking data is a dialogue transcript. Travel booking: two friends, six days in Japan, $1,100 each, water sports and festivals.

• 3B compiled model: presented three options by turn 5, confirmed booking by turn 7.
• Same 3B model + orchestration framework: asked the departure date at turns 4, 6, and 8; presented options at turn 14; finished at turn 18.

Same model, same base capability. The orchestrator makes the model look amnesiac. The subterranean agent isn't being steered node-by-node—it acts on a procedure internalized as statistical structure in its weights.

LoRA Doesn't Work

A notable side finding: LoRA fails to internalize procedural knowledge. All tests at rank 16–128 failed—low-rank update space is too shallow to change the model's implicit state-tracking behavior. A companion paper concludes: procedural knowledge is not low-rank. Full-parameter training is required.

Honest Limitations

• Scope is procedural dialogue tasks (booking, support triage, claims)—explicit steps, bounded branching, closed conclusions. Don't ask a booking agent to write poetry or do math olympiads.
• World knowledge is the real bottleneck: Zoom accuracy capped at 87% even as naturalness hit 97%. Models above 8B were not tested.
• Third-party API calls are out of scope: production "confirm booking" needs GDS/CRS/payment gateways; integrating tool calling while staying compiled was not tested.
• Synthetic data risk: all training dialogues were generated by Claude Sonnet 4.5; systematic errors could be baked into weights. Evaluation used GPT-4.1 as an independent judge, but generation bias wasn't ruled out.
• The LoRA failure is an experimental observation, not a theoretical proof.

The Deeper Principle

The paper's most fundamental contribution is a principle, not an algorithm:

Durable structure belongs in weights. Transient state belongs in the prompt.

For three years, agent development has stuffed everything—flow graphs, routing rules, current state, dialogue history—into an ever-bloating context window, re-read by the model every turn. Orchestration frameworks just help engineers organize the stuffing. The subterranean approach says: refine "how to do things" into the model; write "what the situation is now" in the prompt. Like a human brain: your neural circuits know "how to drive"—you don't recite traffic rules each time you get in the car. But you do need to know today's route to the airport.

An orchestrator is a crutch—useful for learning to walk, a brake once you can.

References

1. Dennis, S., Patil, R., Shabahang, K., & Guo, H. (2026). Compiling Agentic Workflows into LLM Weights: Near-Frontier Quality at Two Orders of Magnitude Less Cost. *arXiv:2605.22502*. 2. Dennis, S., Diamond, M., Patil, R., Shabahang, K., & Guo, H. (2026a). In-Context Prompting Obsoletes Agent Orchestration for Procedural Tasks. *arXiv preprint*. 3. Dennis, S., Shabahang, K., Guo, H., & Patil, R. (2026b). Procedural Knowledge is Not Low-Rank: Why LoRA Fails to Internalize Multi-Step Procedures. *arXiv preprint*. 4. Hosseini-Asl, E., et al. (2020). A Simple Language Model for Task-Oriented Dialogue. *NeurIPS 2020*. 5. Yin, D., et al. (2024). Agent Lumos: Unified and Modular Training for Open-Source Language Agents. *ACL 2024*.