Ploy Migrates Production AI Agent from Claude Opus 4.8 to GPT-5.6 Sol: 2.2x Faster, 27% Cheaper

Forum topic · 小凯 · 2026-07-13

Summary

Ploy, an AI website-building platform, published a detailed engineering post on migrating its production agent from Claude Opus 4.8 to OpenAI's GPT-5.6 Sol. In real-world marketing-site build tests, GPT-5.6 Sol completed pages in 3m42s versus 8m00s (2.2x faster), cost $2.22 versus $3.06 per build (-27%), cut output tokens from 33.0K to 17.1K, and raised visual scores from 0.936 to 0.970. Beyond benchmarks, Ploy documented the full migration playbook: fixing harness assumptions tuned to Claude, resolving a tool-schema issue where GPT-5.6 sent all 25 optional parameters (including invented values like offset: 0 that caused 52-64% empty file reads), rebuilding prompt caching after GPT-5.6 dropped partial prefix matching (per-workspace prompt_cache_key strategy lifted first-call hit rate to 83.7%), and making reasoning replay self-contained with store: false. The post argues that harness fairness, schema-layer abstraction, and cache configuration are decisive for cross-model comparisons.

Ploy, an AI website-building platform, ran its core production agent on Claude Opus (4.7, then 4.8) for four months before migrating to GPT-5.6 Sol. Their engineering blog is essentially a field manual for cross-model migration in production. Below are the key findings.

Key points

Results (redesign suite, real brand assets): GPT-5.6 Sol vs Opus 4.8 — cost $2.22 vs $3.06 (-27%); wall-clock 3m42s vs 8m00s (2.2x faster); input tokens 1.70M vs 2.60M; output tokens 17.1K vs 33.0K (-48%); visual score 0.970 vs 0.936 (+3.4pp, near Ploy's ceiling).
Step 0 — fix the harness first: Ploy's eval suite was tuned to Claude's behavior. Issues found: GPT-5.6 makes parallel tool calls and burned tool-call budgets sized for Opus's sequential style; the eval executor didn't support batched file reads; a missing minScore silently defaulted to 1.0. Core lesson: "Your harness is tuned to your incumbent model, and you don't know it."
Step 1 — tool schema parameter inflation: Ploy's code tool has 25 top-level parameters, 24 optional. Production traces: 6,635 of 6,635 GPT-5.6 calls (100%) carried all 25 parameters, vs 4 of 2,898 (0.1%) for Opus 4.8. Invented values like offset: 0 caused 52–64% of file reads to return empty content — while returning success: true. Prompt-based fixes all failed, including OpenAI strict mode. The working fix: a provider-boundary schema transform making optional fields required but nullable (anyOf: [T, null]), then stripping nulls at the tool invocation seam. Empty reads dropped from 52% to 0%, and the agent used ~30% fewer tool calls.
Step 2 — prompt caching rebuild: Ploy's ~29K-token static prefix was shared org-wide via Claude's cache_control. GPT-5.6 removed partial prefix matching; caching now requires a full-prompt hit plus a prompt_cache_breakpoint and a prompt_cache_key (part of cache identity), with per-node ~15 RPM before fan-out to cold caches. Ploy chose per-workspace keys with layered breakpoints (static prefix / workspace context / session chain). First-call cache hit rate went from 0% to 83.7%, uncached input tokens fell 28%, and GPT-5.6's effective cost flipped below Opus. Ploy's warning: "If you're cost-comparing models and one of them has a cold cache, you are comparing your config, not the models."
Step 3 — reasoning replay: GPT-5.6's Responses API replays prior-turn reasoning as server-side item references, causing intermittent Item 'rs_...' not found failures mid-conversation. Fix: store: false so the SDK fetches encrypted, self-contained reasoning content.

Four truths for cross-model migration

1. Your eval harness is tuned to your incumbent model — verifying harness fairness is a precondition for any serious comparison, not tech-debt cleanup. 2. Tool calling is the largest model-layer behavioral gap: OpenAI-family models fill optional parameters by default; Anthropic models omit them. Agents must abstract at the schema layer. 3. Prompt caching can swing cost by 50%+ on identical inputs — cache both sides optimally before comparing prices, and design prompts around which tokens truly never change. 4. Reasoning replay is a new state-management battleground — control it explicitly (store: false/true, encryption) in multi-turn conversations.

Caveats

Ploy's workload is visual marketing-site builds; small sample sizes (n=11 and n=10); their own visual-judge model may be biased toward GPT-5.6 outputs.
Ploy is now tightly coupled to OpenAI-specific APIs (Responses API, prompt_cache_key, schema behavior) — reverse lock-in that will cost real engineering time on the next migration.
Notably, GPT-5.6's output converges to a single aesthetic unless actively steered — a drawback for per-brand design work; Ploy says steering details will come in a follow-up post.

References

Ploy engineering blog: https://ploy.ai/blog/migrating-a-production-ai-agent-to-gpt-5-6
AI SDK (Vercel cross-model SDK): https://ai-sdk.dev/docs/introduction
OpenAI Responses API / function calling: https://developers.openai.com/api/docs/guides/function-calling
OpenAI Prompt Caching: https://developers.openai.com/api/docs/guides/prompt-caching