Why Some Teams Get 3x from Coding Agents and Others Get 10x: Tools Don't Explain the Gap

Forum topic · 小凯 · 2026-09-15

Summary

This in-depth analysis, built around an AWS Senior Principal Engineer Clare Liguori's talk "From AI-Assisted to AI-Native", examines why productivity differences among teams using identical coding agents range from under 3x to over 10x. In an Amazon Stores pilot, 90%+ of 50 teams used the same internal tools (including Kiro), yet median deployment-speed gains split below 3x versus 4.5x (some exceeding 10x), indicating working methods—not tools—drive the gap. The post cross-weights five research streams and finds that LLM auto-generated context files actually reduce task success rates (ETH Zurich CTXbench), while full spec-driven development adds only +0.05 points versus direct vibe coding; the statistically significant gain (+0.15, p<0.05) comes from validation hooks that let machines judge correctness autonomously. External evidence tempers the hype: the best independent estimate is 2.09x (single-company DiD study), GitClear's 70,000 developer-years show only +9% median, and METR's RCT found a -19% slowdown. The article concludes with a prioritized checklist for engineering leaders: manage code review as a bottleneck, build local deterministic feedback loops, keep context files short and hand-written, avoid language migrations for AI's sake, and budget two months for workflow transformation.

Why Some Teams Get 3x from Coding Agents and Others Get 10x

> Based on Clare Liguori's (AWS Senior Principal Engineer) AI Engineer conference talk *"From AI-Assisted to AI-Native: Building a Frontier Development Team"*, cross-weighed against five parallel research streams.

Key points

1. Tools are not the variable

In the Amazon Stores pilot (50 teams), 90%+ used the same internal toolset (including Kiro), yet the gap stretched from under 3x to a median 4.5x, with some teams exceeding 10x.
The sub-3x teams sprinkled AI on old workflows; the 4.5x teams restructured all five stages: requirement expression, coding delegation, local validation, automated review, and decision release. Per Amdahl's Law, compressing only the coding segment barely moves total cycle time.

2. Three Amazon case studies use incomparable metrics

| Case | Original estimate | Actual result | Metric | |---|---|---|---| | Bedrock Mantle | 30 people × 18 months | 6 people × 76 days ("up to 20x") | commits | | Prime Video | 90 weeks | Revised estimate down to 24 weeks | revised estimate, not delivery | | Amazon Stores (50 teams) | — | Half <3x, median 4.5x, some >10x | deployment speed |

The "20x" figure is commit-count based and self-qualified by Liguori; Prime Video's sprint had no oncall, minimal meetings, and 3 weeks of pre-granular task breakdown; the 50-team study covers only "the better part of last year".
These three numbers must not be chained into a single "evolution curve".

3. "Frontier developer" definition

Hand-writing only 1–2% of shipped code; low-frequency interaction (agents running autonomously for hours); multiple parallel agents on a task backlog instead of waiting on one result. "Babysitting" agents in 30-second loops consumes your attention and caps gains.

4. Five daily habits

Invest in agent context

Slow down to speed up

Feed agents, don't babysit

Make intent explicit

Shift testing left

5. The hardest reversal: documentation is cheap, executable validation is valuable

ETH Zurich (arXiv:2602.11988, CTXbench): LLM auto-generated context files *decreased* success rates (SWE-bench Lite 48.8% → 48.3%, p=0.87) while raising cost 19–23%. Developer hand-written files give only +2.4pp.
arXiv:2604.05278 (Spec Kit Agents): Full SDD process vs direct vibe coding = 3.51 vs 3.46 (+0.05, noise). The statistically significant gain is post-hoc validation hooks: +0.15, Wilcoxon p<0.05.
Efficiency evidence is stronger: AGENTS.md files cut median task time 98.57s → 70.34s (−28.6%) and output tokens −16.6% (arXiv:2601.20404). Best practice: hand-written, 100–150 lines, path-scoped loading, pruned as models upgrade.
Tool-count decay (RAG-MCP): success >90% below 30 tools; retrieval of top-1 schema instead of stuffing all raises tool-selection accuracy 13.62% → 43.13% (3.17×) — the only clean "3x" number, achieved purely by changing context organization.

6. Feedback loops are the real multiplier

SWE-Dev: 30 → 75 turns raises resolve rate only 34.0% → 36.6% (diminishing returns).
CRUST-Bench: compiler feedback lifts o1 from 15% → 28%.
R2E-Gym: execution-only or verifier-only strategies saturate at 42–43%; hybrid reaches 51%.
Make errors agent-readable: INTERVENOR doubles fixes via chained compiler diagnostics; Anthropic reports refined tool descriptions alone reached SWE-bench SOTA for Sonnet 3.5.
Language migration claims are unsupported: SWE-bench Multilingual and Multi-SWE-bench contradict each other on Rust and both rank JS/TS low; Rust's borrow checker is itself agents' top failure source.

7. External evidence caps the hype

METR RCT: experienced open-source devs on their own repos were 19% slower with AI allowed, while believing they were 20% faster — a 39-point perception gap that undermines all self-reported gains.
GitClear (70,000 developer-years): median delivery gain only +9%.
Best independent evidence: 2.09× (single-company staggered DiD, arXiv:2607.01904).
DORA 2025: AI is an "amplifier"; delivery throughput standardized effect is only 0.03; flow efficiency ~21% supports the "decisions and waiting are the new bottleneck" thesis.

8. The bottleneck really has shifted

Faros AI 2026 (22,000 devs): tasks +33.7% but bugs +54%, per-PR production incidents +242.7%, code churn +861%, median PR review time +441.5%.
LinearB 2026: AI-assisted PRs' 30-day merge rate only 32.7%; agentic PRs wait 17.6h for first review.
GitClear: copy-pasted lines exceeded moved lines for the first time in 2024; refactoring −70%.

9. The junior engineer problem

Anthropic RCT (arXiv:2601.20245, n=52): AI-assisted learners scored 50% vs 67% hand-written on a new library, worst on debugging — and were only ~2 minutes faster. "AI-enhanced productivity is not a shortcut to competence."
Russinovich/Hanselman (CACM) prescribe preceptorship: pair juniors with seniors at 3:1–5:1 ratios, with learning as an explicit goal.

Actionable checklist for engineering leaders

Tier 1 (strong evidence): 1. Manage code review as an explicit bottleneck (limit PR size, AI pre-review, senior reviewer load caps). 2. Build local, deterministic, fast validation loops in hooks (deterministic), not CLAUDE.md (advisory) — the only intervention with statistically significant measured gains. 3. Provide spec templates landing on criteria (EARS acceptance standards, positive/negative examples, non-goals), not exhaustive markdown. 4. Context files: hand-written, <200 lines, path-scoped, pruned per model upgrade. Don't auto-generate.

Tier 2 (worth it, measure cost): improve error messages to give fixes; build mentorship before scaling AI; replace diffs/commits metrics with rework rate, PR size distribution, lead time, mutation score.

Tier 3 (insufficient evidence): don't migrate languages for AI's sake; don't expect SDD ceremony alone to yield multiples; budget ~2 months of upfront productivity dip before expecting returns.

Bottom line

The leverage is in making machines able to judge correctness themselves, not in writing more instructions for them. The 4.5x figure is an internal Amazon observation on deployment-speed metrics without a control group — cite it cautiously. The best external independent evidence is 2.09x.

---

*Key sources: ai.engineer talk transcript; arXiv:2602.11988, 2601.20404, 2604.05278, 2505.03275, 2506.07636, 2601.20245, 2607.01904; Chroma Context Rot; Anthropic context engineering guide; METR RCT; DORA 2025; GitClear; Faros AI 2025/2026; LinearB 2026 benchmark; martinfowler.com (Böckeler on SDD).*