FAPO: Turning Claude Code into a Fully Autonomous Prompt Optimizer for Multi-Step LLM Pipelines

> Kassianik, P. et al. *FAPO: Fully Autonomous Prompt Optimization of Multi-Step LLM Pipelines.* arXiv:2606.19605, 2026. > Cisco Foundation AI & Yale University. > Code: https://github.com/cisco-foundation-ai/fully-automated-prompt-optimization

The Problem: Pipelines Fail at Every Stage

Enterprise LLM pipelines typically chain multiple steps—retrieval, reranking, LLM reasoning, formatting—each with its own prompt. Manual tuning is slow, and existing automatic optimizers like GEPA/DSPy only optimize single instruction strings, which can mask root causes elsewhere in the pipeline. FAPO's goal: let the whole pipeline diagnose and optimize itself.

Core Philosophy: Attribution-Driven, Minimal Changes

FAPO treats prompt edits as the cheapest, most controllable optimization. It escalates only when attribution shows prompts cannot solve the bottleneck:

1. Level 1 – Prompt text modification (lightest) 2. Level 2 – Chain parameter changes (temperature, top-p, retrieval counts) 3. Level 3 – Chain structure changes (new steps, routing, post-processing)

This progression is controllable, explainable, and cheap to deploy.

Architecture: Three-Agent System on Claude Code

FAPO uses Claude Code as the optimization orchestration layer, governed by enterprise-style "playbooks":

• Optimization Agent — reads the tenant playbook, drives the optimization loop, decides when to escalate levels.
• Step-Attribution Subagent — acts like a forensic analyst: classifies failures per step (e.g., "13 cases failed from verbose answers; add brevity constraints") and labels each as prompt-fixable vs. structural bottleneck.
• Variant-Reviewer Subagent — audits every proposed change for scope compliance, data isolation across tenants, placeholder integrity, and scorer compatibility.

The optimization loop runs six steps: evaluate → attribute failures → propose minimal change → review → test variant → iterate or escalate.

Four Guardrails Against Overfitting

1. Data isolation — the optimizer sees only individual training cases; validation/test sets return aggregate scores only. 2. Scope constraints — tenant playbooks define what may and may not be changed, independently enforced by both agents. 3. Iteration history — structured logs of all variants, scores, and failures prevent repeated bad proposals and enable human audits. 4. Variant immutability — every variant is a new file; nothing is overwritten, so the full trajectory is reproducible and rollback-safe.

Results: 15 Wins Out of 18 Comparisons

Six benchmarks (HotpotQA, HoVer, IFBench, Papillon, LiveBench-Math, AIME), three task models (GPT-4.1-mini, GPT-5.4-mini, Gemma 3-12B), against GEPA as the baseline:

| Benchmark | Model | GEPA | FAPO | Gain | |---|---|---|---|---| | HotpotQA | GPT-4.1-mini | 37.11% | 46.67% | +9.56pp | | HoVer | GPT-5.4-mini | 45.56% | 94.11% | +48.56pp ‡ | | IFBench | GPT-5.4-mini | 45.56% | 84.51% | +38.95pp ‡ | | Papillon | GPT-4.1-mini | 55.33% | 60.44% | +5.11pp | | AIME | GPT-4.1-mini | 16.00% | 13.78% | −2.22pp |

‡ = escalated to structural optimization.

FAPO won 15/18 comparisons (11 statistically significant), averaging +14.1pp.

Key Findings

• HoVer: attribution found insufficient retrieval coverage; expanding 3-hop to 4–5-hop retrieval with multi-query BM25 and entity-aware rescue yielded the largest gains (+20 to +48pp).
• IFBench: format-constraint failures were fixed by adding deterministic post-processing nodes (+20 to +39pp).
• HotpotQA: prompt-only edits (brevity constraints, must-answer rules) lifted validation EM from 39.2% to 70.3% before hitting a retrieval-bound structural ceiling.
• AIME: the only loss. With only ~150 validation cases, optimization signal was too weak—automatic optimization needs sufficient signal density.
• CTIBench-RCM (security, CVE→CWE classification, 263 classes): prompt-only constraints yielded +4.0pp (GPT-5), +7.1pp (Foundation-Sec-8B-Instruct), +2.0pp (Reasoning variant). Notably, each model's best strategy differed—there is no universally optimal prompt; optimization must be model-aware.

Technical Design

• Tenant model: a shared core engine (runner, evaluation, attribution, scorers) serves isolated tenant workspaces, each with its own chain.py, prompts, scorer, config, playbook, and history. Tenants enable per-BU, per-customer, or per-security-domain isolation.
• LangGraph representation: pipelines are state graphs (LLM calls, code execution, retrieval, tools, conditional routing), enabling topology-aware structural upgrades and per-node I/O logging for attribution.
• Failure taxonomy: failures are mapped to fixes—missing evidence → more retrieval/multi-query; unsupported abstention → must-answer rules; verbose answers → brevity constraints; malformed output → schema enforcement and post-processing; etc.

Limitations

• Claude Code dependency: the optimizer layer is bound to Claude, raising reproducibility and trust concerns (especially in security contexts where code is auto-edited).
• Sparse-signal scenarios: AIME shows that with small validation sets, attribution-driven optimization may not beat simple baselines.
• Structural escalation risk: structure changes have a much larger decision space and potential side effects, only partially mitigated by guardrails.
• Cost: dozens of variants × full validation runs × multi-step pipelines is not cheap; the paper omits concrete cost figures.

Takeaway

FAPO's significance is less "yet another prompt optimizer" and more a shift from prompt tuning to attribution-driven pipeline engineering: every change is evidence-backed, logged, and auditable. For enterprises, the tenant model, guardrails, isolation, and auditability matter as much as accuracy gains—buyers want a *controllable* optimization process, not just higher scores.