Your LangGraph May Be Making Claude Dumber: A Controlled Study of Agent Orchestration vs. In-Context Prompting
| Item | Detail | |------|--------| | Paper | In-Context Prompting Obsoletes Agent Orchestration for Procedural Tasks | | Authors | Simon Dennis, Michael Diamond, Rivaan Patil, Kevin Shabahang, Hao Guo (University of Melbourne) | | arXiv | 2604.27891 (cs.AI, cs.LG) | | Date | April 30, 2026 (v2: May 5, 2026), 20 pages | | Core contribution | First controlled experiment showing that placing the entire workflow graph in the system prompt beats LangGraph external orchestration on all 15 metrics for procedural tasks — orchestration failure rates of 9-24% vs. 0.5-11.5% for the full-graph approach | | Link | https://arxiv.org/abs/2604.27891 |
Imagine you are an engineer. Your boss says: "We need a customer service AI. Build an orchestration workflow with LangGraph."
So you spend four days drawing a graph: 14 nodes, 3 decision hubs, multiple branches. You test 200 scenarios and find that 24% of conversations get stuck at some decision point. You assume your graph design isn't good enough.
The problem may not be your graph. It may be the graph-based architecture itself.
The Experiment
Researchers at the University of Melbourne ran a direct controlled experiment: the same Claude Sonnet model, the same tasks, the same workflow specifications.
- Condition A (LangGraph orchestration): each node's prompt template is injected per call; an orchestrator uses extra LLM calls to route between nodes.
- Condition B (In-context): the entire workflow graph is placed in the system prompt as plain text, and the model decides how to proceed itself.
- Task completion: in-context 4.53-4.92 vs. LangGraph 4.17-4.62
- Information accuracy: in-context 4.64-4.92 vs. LangGraph 4.21-4.75
- Consistency: in-context 4.83-4.99 vs. LangGraph 4.32-4.55
- Graceful deviation handling: in-context 4.96-5.00 vs. LangGraph 4.38-4.62
- Naturalness: in-context 5.00 vs. LangGraph 4.58-4.84
- Travel booking: 10.8 LLM calls/conversation (orchestration) vs. 8.7 (in-context) — 1.2x
- Insurance: 17.3 vs. 10.0 — 1.7x
Three domains — travel booking (14 nodes), Zoom tech support (14 nodes), insurance claims processing (55 nodes) — with 200 scenarios each.
Result: Condition B (in-context) outperformed Condition A (LangGraph) on all 15 metrics. Without exception.
The Cold Numbers
With Claude Sonnet as judge (1-5 scale) across five dimensions in three domains:
All 15 comparisons had p-values below 0.005 (Mann-Whitney U, Holm-Bonferroni corrected). Effect sizes ranged from d = 0.37 to 1.01.
The authors are notably honest: a 5.00 naturalness score may reflect LLM self-preference bias (Claude judging its own outputs). So they had GPT-4.1 independently re-judge all 1,200 conversations. The result: 11/15 comparisons still favored in-context — none favored LangGraph. The three most critical metrics (task completion, accuracy, consistency) held under both judges. The naturalness gap shrank to near zero, but the core conclusion stands.
Why Orchestration Performs Worse: Three Structural Problems
The paper identifies three inherent failure sources of external orchestration — not because LangGraph is poorly implemented, but because the architecture truncates information by design:
1. Fragmented reasoning. LangGraph injects only the current node's prompt template at each LLM call. The model sees a local fragment of the flowchart — it searches products without knowing a price comparison follows; it confirms information without knowing the confirmation feeds a claims decision. Confined to one node's time window, the model loses awareness of the global conversational arc — repeating questions, skipping steps, or truncating dialogues.
2. Orchestration-specific failure modes. The orchestrator requires extra LLM calls at decision hubs to pick the next branch — and those calls can fail. Three failure types exist only under orchestration: routing failures (the LLM misclassifies the dialogue state → wrong node), decision ambiguity (multiple edge conditions trigger → the default, most dangerous branch is chosen), and template conflicts (the node prompt clashes with conversation context). In the 55-node insurance domain, errors cascade — one wrong classification → wrong information gathering → wrong coverage judgment → a failed claim.
3. Constrained naturalness. Per-node template injection locks each node's language into a small scripted space. Dialogue becomes hard switching between nodes, losing the natural breathing of human conversation.
The Economics You Might Be Missing
On the surface, orchestration saves tokens (no full graph transmitted per call). In the 55-node insurance domain: in-context used 68K tokens/conversation vs. 43K for orchestration — orchestration looks cheaper.
But that ignores the orchestrator's routing overhead.
Total cost: in-context is 1.3-1.4x more expensive — travel $0.10 vs. $0.08, insurance $0.22 vs. $0.17 per conversation (Claude Sonnet API pricing). But that ~$0.05 buys a drop in failure rate from 9-24% to 0.5-11.5%. Factoring in recovered churn, human-escalation costs, and brand trust, it may be the highest-ROI five cents you spend this year.
Failure Rates: Where Orchestration Collapses
| Domain | In-context failure rate | LangGraph failure rate | Gap | |--------|------------------------|------------------------|-----| | Travel (14 nodes) | 11.5% | 24.0% | 2.1× | | Zoom (14 nodes) | 0.5% | 9.0% | 18× | | Insurance (55 nodes) | 5.0% | 17.0% | 3.4× |
The 18x Zoom gap is striking. In travel orchestration failures, the model skipped decision steps into wrong branches or looped collecting information without progressing. In insurance, routing errors at the 6 decision hubs cascaded systematically — one wrong type classification polluted the entire downstream path. In-context succeeds because the model, seeing the whole graph, keeps global goals in mind during local actions; the orchestrator strips that awareness away at every decision point.
Honest Caveats: What This Paper Does NOT Show
1. This is not an attack on LangGraph specifically. The implementation was standard — nodes mapped to graph nodes, extra LLM calls at hubs. Other frameworks (CrewAI, Google ADK, OpenAI Agents SDK) share the same paradigm. The problem is the "per-node template injection + LLM routing" paradigm itself.
2. All three domains used synthetic scenarios — LLM-simulated users, not production conversations. Real user unpredictability or adversarial behavior could shift the relative performance, though the synthetic domains were carefully designed to cover the full range of graph paths, user styles, and satisfaction.
3. In-context token consumption has limits. The 55-node graph serialized to ~4,000 tokens — only 2% of a 200K+ context window. But a 500-node flow with complex business logic might exceed the window. The authors cite a companion paper that "compiles" workflows into an 8B model's weights via fine-tuning, achieving comparable quality at 128-462x lower inference cost — a third path between cost and quality.
4. Orchestration still has valid uses: multi-model pipelines (vision + code models), tool calling and external state management (databases, APIs, file systems), non-procedural open-ended tasks (creative writing, research), and smaller models that need guardrails for weaker instruction following. The paper's actual conclusion: for procedural multi-turn conversations with frontier models and a known workflow, orchestration provides zero benefit and measurable harm.
Commentary: The Redux Parallel
This echoes a historical pattern in framework evolution. When React first appeared, you needed Redux, redux-saga, redux-thunk to manage state and side effects. As React hooks and React Query matured, Redux became over-engineered legacy baggage in many cases. External orchestration frameworks provided valuable structure when GPT-3-era models needed precise instructions at every step — but that necessity evaporates as model capability jumps.
A survey cited in the paper found 80% of developers struggle to choose between LangGraph, CrewAI, Google ADK, and OpenAI Agents SDK. This paper's underlying message: perhaps the question isn't which framework to choose, but whether procedural tasks need any framework at all.
If Claude Sonnet (or GPT-4.1, Gemini 2.5 Pro) can self-orchestrate a 55-node insurance flow at 95% success while LangGraph orchestration of the same model reaches only 83%, the scaffolding isn't support — it's a ceiling.
A final observation: the paper's appendix includes complete dialogues from both systems on identical scenarios. The orchestrated conversations read like an automated phone tree — "Now I'll help you search for flights. Please wait." → "Now I'll help you select a hotel. Please wait." — mechanical, scripted, with a faint sense of node-switching in every segment. The in-context conversations read like a person: "Okay, Mumbai to Goa, Jan 15-25 — let me look at flights and hotels together — any particular preferences?"
Once you see the same model, same scenario, two different conversation qualities, it's clear: optimizing the orchestrator isn't the solution — the orchestrator is the problem.
References
1. Dennis, S., Diamond, M., Patil, R., Shabahang, K., Guo, H. (2026). In-Context Prompting Obsoletes Agent Orchestration for Procedural Tasks. arXiv:2604.27891. 2. LangChain, Inc. (2024). LangGraph: Build Resilient Language Agents as Graphs. 3. Wang, Y. et al. (2026). An Empirical Study of Agent Developer Practices in AI Agent Frameworks. arXiv:2512.01939. 4. Cemri, M. et al. (2026). Why Do Multi-Agent LLM Systems Fail? arXiv:2503.13657. 5. Kapoor, S. et al. (2024). AI Agents That Matter. arXiv:2407.01502. 6. Dennis, S. et al. (2026). Compiling Agentic Workflows into LLM Weights: Near-Frontier Quality at Two Orders of Magnitude Less Cost. Companion paper.