Ploy Migrates Production AI Agent from Claude Opus 4.8 to GPT-5.6 Sol: 2.2x Faster, 27% Cheaper

Ploy, an AI website-building platform, ran its core agent on Claude Opus (4.7, then 4.8) for four months—until GPT-5.6 Sol launched. Their engineering blog, Migrating a production AI agent to GPT-5.6, documents the full cross-model migration. This post summarizes and translates the key findings.

Key points

• Headline results (redesign suite, real brand assets):
• Cost per build: $2.22 vs $3.06 (−27%)
• Wall-clock: 3m42s vs 8m00s (2.2x faster)
• Input tokens: 1.70M vs 2.60M; output tokens: 17.1K vs 33.0K
• Visual score: 0.970 vs 0.936 (near Ploy's scoring ceiling)
• Step 0 — Fix your harness first. Ploy's eval harness was tuned to Opus's behavior: the tool-call budget assumed sequential calls (GPT-5.6 makes parallel calls), the executor didn't support batched file reads, and a missing minScore silently defaulted to 1.0. Lesson: "Your harness is tuned to your incumbent model, and you don't know it." Any serious cross-model comparison must first verify the harness is fair to both models.
• Step 1 — Tool schema behavior gap. GPT-5.6 sent all 25 optional parameters of Ploy's code tool in 100% of 6,635 calls (Opus: 0.1%), inventing plausible values. Invented offset: 0 caused 52–64% of file reads to return empty content—while reporting success: true. Prompt fixes failed entirely. The working fix: at the provider boundary, rewrite optional fields as required but nullable (anyOf: [T, null]), then strip nulls at the tool invocation seam. Empty reads dropped to 0%, and the agent used ~30% fewer tool calls.
• Step 2 — Rebuild prompt caching. GPT-5.6 removed partial prefix matching; only full-prompt hits with a prompt_cache_breakpoint and matching prompt_cache_key cache, with ~15 RPM per cache node before fan-out to cold nodes. Ploy chose per-workspace keys with layered breakpoints (static prefix / workspace context / session chain). Result: first-call cache hit rate 0% → 83.7%, 28% fewer uncached input tokens—and GPT-5.6 became cheaper than Opus per call. "If you're cost-comparing models and one of them has a cold cache, you are comparing your config, not the models."
• Step 3 — Reasoning replay must be self-contained. GPT-5.6's Responses API replays prior reasoning as server-side item references, intermittently failing with Item 'rs_...' not found. Fix: store: false so the SDK requests encrypted reasoning content instead of server-state pointers.

Four truths for cross-model migration

1. Your eval harness is tuned to your incumbent model—fixing it is a precondition for evaluation, not tech-debt cleanup. 2. Tool calling has the largest model-level behavioral divergence; cross-model agents need schema-layer abstraction. 3. Prompt caching is a decisive cost variable—the same model can vary 50%+ in cost by cache config alone. 4. Reasoning replay is a new state-management battleground; it must be explicitly controlled in multi-turn conversations.

Caveats

• Small sample sizes (n=11 and n=10 builds); Ploy's domain is marketing-site builds, so results may not transfer to backend or data-pipeline workloads.
• Ploy's own visual judge may be biased toward GPT-5.6 outputs.
• Deep reliance on OpenAI-specific APIs (Responses API, prompt_cache_key, tool schema behavior) creates reverse lock-in—the real cost of multi-model strategy is migrating API behavior differences.
• Designers noted GPT-5.6 produces clean, modern, grid-strong code but tends to converge on one style unless actively steered—relevant for brand-consistency workloads.

References

• Ploy engineering blog: https://ploy.ai/blog/migrating-a-production-ai-agent-to-gpt-5-6
• Vercel AI SDK: https://ai-sdk.dev/docs/introduction
• OpenAI Responses API (function calling): https://developers.openai.com/api/docs/guides/function-calling
• OpenAI Prompt Caching: https://developers.openai.com/api/docs/guides/prompt-caching