Same Coding Agents, Different Results: Why Some Teams Get 3x and Others 10x — Evidence Weighed

Forum topic · 小凯 · 2026-09-15

Summary

This in-depth analysis synthesizes a talk by AWS Senior Principal Engineer Clare Liguori on AI-native development with five parallel lines of external research. In Amazon's Stores pilot, over 90% of 50 teams used the same internal tooling (including Kiro), yet productivity diverged from under 3x to a median 4.5x, with some teams exceeding 10x — the difference was workflow, not tools. The report's sharpest finding: documentation itself adds little value. Independent experiments (ETH Zurich CTXbench and arXiv 2604.05278) show LLM auto-generated context files slightly decrease task success, while full spec-driven development yields only +0.05 over direct vibe coding; the statistically significant gain (+0.15, p<0.05) comes from post-hoc validation hooks. External evidence tempers the hype: the best independent result is 2.09x (single-company DiD), GitClear's 70,000 developer-years show median +9%, and METR's RCT found a -19% slowdown for experienced maintainers. The bottleneck has shifted to code review (+441.5% median review time) and decision-making. Actionable guidance includes deterministic local validation loops, hand-written short context files, EARS-style testable specs, mentorship ratios for juniors, and rethinking metrics beyond commit counts.

Key points

This report integrates Clare Liguori's talk *"From AI-Assisted to AI-Native: Building a Frontier Development Team"* (AI Engineer summit, AWS Senior Principal Engineer) with cross-validated external research.

1. Tools are not the variable

In Amazon Stores' 50-team pilot, 90%+ of teams used the same internal tooling (including Kiro), yet results diverged: half under 3x, the other half a median 4.5x, some above 10x (measured by production deployment speed).
Caveats on metrics: Bedrock Mantle's "20x" was commits-based; Prime Video's 24 weeks was a revised *estimate*, not delivery; the 50-team study had no control group.
Liguori's "frontier developer" definition: hand-writing only 1–2% of code, agents running autonomously for hours, multiple agents running in parallel. The five habits: invest in agent context, slow down to speed up (codebase refactoring), feed agents don't babysit, make intent explicit, shift testing left.

2. The hardest reversal: documentation is cheap, executable validation is valuable

ETH Zurich (arXiv 2602.11988, CTXbench): LLM auto-generated context files *decreased* success (48.3% vs 48.8% baseline on SWE-bench Lite, n.s. or worse) and cost 19–23% more tokens. Hand-written files gave only a narrow +2.4pp.
arXiv 2604.05278 (Spec Kit Agents, 128 runs): Full SDD process vs direct vibe coding = 3.51 vs 3.46 (+0.05, not significant). Full SDD + hooks = 3.66 (+0.15, Wilcoxon p<0.05). Post-hoc validation hooks matter more than pre-hoc discovery hooks.
Context file best practices: hand-written, <200 lines (Claude Code), <60–300 lines (AGENTS.md), loaded by path, pruned as models upgrade. Anthropic confirms @import does not save context; RAG-MCP shows tool selection accuracy 13.62% → 43.13% (3.17x) by retrieving top-1 schemas instead of stuffing all.

3. Feedback loops are the real multiplier

SWE-Dev: 30→75 turns only +2.6pp (diminishing returns). CRUST-Bench: compiler feedback took o1 from 15% → 28%. R2E-Gym: execution-only verifiers saturate at ~42–43%.
Deterministic local testing (mocks over real remote services) prevents agent thrash and "fake fixes". Anthropic: hooks are deterministic, CLAUDE.md instructions are advisory.
No public evidence supports migrating languages (e.g., to Rust/TypeScript) for agent performance — two multilingual benchmarks contradict each other on Rust.

4. External evidence tempers the 4.5x claim

Best independent evidence: 2.09x (single-company staggered DiD, arXiv 2607.01904).
GitClear (70,000 developer-years): median +9%.
METR RCT: experienced open-source maintainers were 19% slower with AI; the perception–reality gap was 39 percentage points.
DORA 2025 (~5,000 respondents): AI is an "amplifier"; delivery throughput standardized effect only 0.03.
Demirer et al.: commits +180% with agentic AI, but releases only +30%.

5. The bottleneck has shifted

Faros AI 2026 (22,000 developers): median PR review time +441.5%, production incidents per PR +242.7%, churn +861%.
LinearB 2026: AI-assisted PRs' 30-day merge rate only 32.7%; agentic PR first-review wait 17.6h.
Liguori's new costs: harder code review (especially for juniors), decision-making as the new bottleneck, FOMAT, management expectation mismatch (a ~2-month productivity dip is normal during codebase investment).

6. Junior engineers: a real problem

Anthropic RCT (n=52): AI group scored 50% vs handwritten group 67% on a new library (Trio), largest gap in debugging — and the AI group was only ~2 minutes faster (not significant).
Proposed fix (Russinovich & Hanselman, CACM): "preceptorship" — pair juniors with senior mentors at 3:1–5:1, make learning an explicit goal.

7. Practical checklist (ordered by evidence strength)

1. Manage code review as an explicit bottleneck (limit PR size, AI pre-review, cap senior reviewer load). 2. Build fast, local, deterministic validation loops — in hooks, not CLAUDE.md. 3. Make specs land in testable criteria (EARS-style), not markdown completeness. 4. Context files: hand-written, short, path-loaded, regularly pruned; never /init-auto-generated. 5. Improve error messages into actionable repair guidance. 6. Establish mentorship before scaling AI adoption. 7. Demote diffs/commits to secondary metrics; add rework rate, PR size distribution, lead time, mutation score. 8. Don't migrate languages for AI's sake; don't expect SDD rituals alone to deliver multiples; budget ~2 months for the initial productivity dip.

Bottom line

The lever is making machines able to judge correctness themselves, not writing more human documentation. The credible external ceiling is ~2x; Amazon's 4.5x is an internal observation by deployment-speed measurement, not a universal law — and "intent engineering" should produce criteria, not documents: good specs are testable, bad specs are merely explainable.