FAPO: Using Claude Code as a Fully Autonomous Prompt Optimization Engineer for Multi-Step LLM Pipelines

Forum topic · 小凯 · 2026-06-21

Summary

FAPO (Fully Autonomous Prompt Optimization) is a framework from Cisco Foundation AI and Yale University that uses Claude Code as an agentic orchestrator to automatically optimize multi-step LLM pipelines. Instead of tuning a single prompt string like GEPA or DSPy, FAPO performs failure attribution across every pipeline step, then applies minimal, evidence-driven fixes: prompt edits first, chain parameter changes second, and pipeline structure modifications only when attribution proves prompts cannot solve the bottleneck. The system uses a three-layer agent architecture (optimization agent, step-attribution subagent, variant-reviewer subagent) plus four guardrails against overfitting: data isolation, tenant-level scope constraints, immutable iteration history, and variant immutability. Across six benchmarks and three task models, FAPO won 15 of 18 comparisons against GEPA, with an average gain of +14.1 percentage points, including large structural-upgrade wins on HoVer (+20 to +48pp) and IFBench (+20 to +39pp). A cybersecurity evaluation on CTIBench-RCM (CVE-to-CWE classification) showed model-aware optimization matters: the best strategy differed across GPT-5 and Foundation-Sec-8B variants. The paper also candidly reports limitations, including GEPA winning on AIME due to sparse validation signal, and the optimizer being tied to Claude Code.

FAPO: Turning Claude Code into Your Prompt Optimization Engineer

> Kassianik, P. et al. *FAPO: Fully Autonomous Prompt Optimization of Multi-Step LLM Pipelines.* arXiv:2606.19605, 2026. > Cisco Foundation AI & Yale University. > Code: https://github.com/cisco-foundation-ai/fully-automated-prompt-optimization

Key points

Problem: Enterprise multi-step LLM pipelines (retrieve → rerank → reason → format) have multiple prompts tuned by hand. Single-prompt optimizers like GEPA/DSPy can only edit instruction strings and may mask root causes in upstream steps such as retrieval.
Core philosophy: Prompt edits are the cheapest, most controllable optimization. Escalate only when attribution evidence shows prompts cannot fix the bottleneck:
Level 1: prompt text edits
Level 2: chain parameter changes (temperature, top-p, retrieval count)
Level 3: pipeline structure changes (add steps, reroute, add post-processing)

Architecture: Claude Code as the orchestrator

FAPO uses a three-layer agent design:

Optimization Agent — reads a tenant "playbook," sets scope contracts, drives the loop, decides when to escalate levels.
Step-Attribution Subagent — a forensic analyst that classifies failures per step, e.g., "13 cases failed from retrieval coverage gaps; multi-query may help" or "8 cases failed from unsupported abstention; add must-answer rules."
Variant-Reviewer Subagent — a safety auditor checking scope compliance, data isolation across tenants, placeholder integrity, and scorer compatibility.

Each iteration follows a six-step loop: evaluate → attribute failures → propose one minimal change → review → test variant → iterate or escalate.

Four guardrails against overfitting

1. Data isolation: the optimizer sees only individual training cases; validation/test sets are exposed as aggregate scores only. 2. Scope constraints: tenant playbooks define what can and cannot be changed. 3. Iteration history: structured logs of all variants, scores, and failure reasons enable audits and prevent repeated invalid changes. 4. Variant immutability: every variant is a new file; full rollback and reproducibility.

Results: 15 wins out of 18

Benchmarks: HotpotQA, HoVer, IFBench, Papillon, LiveBench-Math, AIME, plus CTIBench-RCM; models: GPT-4.1-mini, GPT-5.4-mini, Gemma 3-12B. Baseline: GEPA.

| Benchmark | Model | GEPA | FAPO | Gain | |---|---|---|---|---| | HotpotQA | GPT-4.1-mini | 37.11% | 46.67% | +9.56pp | | HotpotQA | GPT-5.4-mini | 49.78% | 57.44% | +7.67pp | | HotpotQA | Gemma-3-12B | 36.44% | 45.22% | +8.78pp | | HoVer | GPT-4.1-mini | 34.44% | 59.22% | +24.78pp ‡ | | HoVer | GPT-5.4-mini | 45.56% | 94.11% | +48.56pp ‡ | | HoVer | Gemma-3-12B | 30.00% | 50.22% | +20.22pp ‡ | | IFBench | GPT-4.1-mini | 40.00% | 59.84% | +19.84pp ‡ | | IFBench | GPT-5.4-mini | 45.56% | 84.51% | +38.95pp ‡ | | IFBench | Gemma-3-12B | 36.22% | 64.89% | +28.67pp ‡ | | Papillon | GPT-4.1-mini | 55.33% | 60.44% | +5.11pp | | LiveBench-Math | GPT-4.1-mini | 24.00% | 28.00% | +4.00pp | | AIME | GPT-4.1-mini | 16.00% | 13.78% | -2.22pp |

‡ = escalated to pipeline structure optimization (not prompt-only). FAPO won 15/18 comparisons, 11 statistically significant, average gain +14.1pp.

When structural upgrades mattered: HoVer failures came from insufficient retrieval coverage — expanding 3-hop to 4-5-hop retrieval with multi-query BM25 and entity-aware rescue yielded +20–48pp. IFBench format failures were fixed with deterministic post-processing nodes (+20–39pp). On HotpotQA, prompt-only edits (brevity constraints, must-answer rules) lifted validation EM from 39.2% → 70.3% before attribution showed a retrieval-bound structural bottleneck.

AIME caveat: GEPA won. With only 150 validation cases, the optimization signal was too sparse — a reminder that automatic optimization needs sufficient signal density.

Cybersecurity (CTIBench-RCM, CVE→CWE, 263 classes, prompt-only): GPT-5 72.1% → 76.1%; Foundation-Sec-8B-Instruct 63.9% → 71.0%; Foundation-Sec-8B-Reasoning 74.8% → 76.8%. Notably, the optimal prompt strategy differed per model — GPT-5 needed detailed rules while Instruct needed shorter prompts. There is no universally optimal prompt; optimization must be model-aware.

Technical highlights

Tenant model: a shared core engine serves isolated tenant workspaces, each with its own chain code, prompts, scorer, config, playbook, and history — enabling per-BU, per-customer, or per-security-domain isolation.
LangGraph representation: pipelines are state graphs, letting FAPO understand topology, safely add/remove nodes during structural upgrades, and log per-node inputs/outputs for attribution.
Failure taxonomy: missing evidence → expand retrieval; unsupported abstention → must-answer rules; verbose answers → brevity constraints; malformed output → schema/post-processing; weak final instruction → rewrite/few-shot; wrong reasoning → intermediate-step decomposition.

Limitations

The optimizer layer is bound to Claude Code, requiring Claude API access and trust in agentic code editing — a concern in security-sensitive settings.
Structural upgrades expand the decision space and potential variance; guardrails mitigate but do not eliminate risk.
Compute cost is nontrivial (50 variants × validation runs × multi-step pipelines), and the paper does not report optimization cost figures.

Takeaway

FAPO reframes prompt optimization as attribution-driven pipeline engineering: every change is evidence-based, logged, and auditable. For enterprise deployments, its tenant model, guardrails, isolation, and auditability matter as much as raw accuracy gains — customers buy a controlled optimization process, not just higher scores.

Reference: Kassianik, P. et al. (2026). FAPO: Fully Autonomous Prompt Optimization of Multi-Step LLM Pipelines. *arXiv:2606.19605*. Cisco Foundation AI & Yale University.