Feynman-Style Breakdown of Anthropic's Agent Evals Framework: Why Unevaluated Agents Can't Scale

Forum topic · 小凯 · 2026-05-16

Summary

A detailed analysis of Anthropic's engineering blog "Demystifying evals for AI agents," explaining why the same qualities that make agents powerful (autonomy, intelligence, flexibility) make them hard to evaluate. The framework organizes agent evaluation into eight elements: three dynamic runtime components (Trial, Transcript, Outcome), three static definitions (Task, Suite, Grader), and two infrastructure pieces (Eval Harness, Agent Harness). Key insights include: you evaluate the coupled system of model plus scaffolding, not the model alone; Transcript versus Outcome distinction (what the agent said vs. what actually changed in the environment); and a hierarchy of graders—code-based first, model-based second, human as calibration benchmark. The article highlights two statistical traps: pass@k versus pass^k (an agent with 75% per-run success fails at least once in three consecutive runs 58% of the time) and eval saturation, where near-perfect scores signal benchmarks that are too easy rather than agent perfection. It closes with evaluation strategies for coding, conversational, research, and computer-use agents, plus an eight-step roadmap for building an eval system from real failure cases.

Feynman-Style Breakdown of Anthropic's Agent Evals Framework

> Core conclusion up front: The very capabilities that make agents impressive—autonomy, intelligence, flexibility—are exactly what makes them hard to evaluate. Anthropic's framework turns agent evaluation from mysticism into engineering with an eight-element structure: three dynamic execution components (Trial / Transcript / Outcome), three static definition components (Task / Suite / Grader), and two infrastructure components (Eval Harness / Agent Harness). The most counterintuitive insight: you're not evaluating the model—you're evaluating the coupled system of "model + scaffolding." And one statistical trap: an agent with 90% success rate run 3 times has only a 73% chance of passing all three—users expect "works every time," while engineering defines "works sometimes."

1. Why This Deep Dive Matters

This is not a paper—it's hard-won lessons from Anthropic's engineering team. The evaluation framework distilled from Claude Code's iteration is becoming a de facto industry standard for agent development.

Context: The turning point for Claude Code came when users started complaining that "updates keep making it worse." Without evals, the team was flying blind.

2. The Eight Elements of Agent Evaluation

Group 1: Dynamic Execution (Runtime)

| Element | What it is | Key insight | |---------|-----------|-------------| | Trial | A single attempt | Agents are stochastic—run multiple trials for reliable conclusions | | Transcript | Full trajectory record | Don't just check the final answer—examine tool call sequences, reasoning chains, intermediate errors | | Outcome | Final environment state | The agent saying "booked!" isn't the same as the booking existing—check the database |

The hardest distinction: Transcript vs. Outcome

Imagine a flight-booking agent:

Transcript = what it said, which APIs it called, how it reasoned
Outcome = whether the reservation actually exists in the environment's SQL database

Anthropic's own words: *"Agent might say 'Your flight has been booked' at the end of the transcript, but the outcome is whether a reservation exists in the environment's SQL database."*

Too many teams only look at what the agent said, not how the world changed.

Group 2: Static Definitions (Test Cases)

| Element | What it is | Practical advice | |---------|-----------|------------------| | Task | A single test case (input + success criteria) | Extract from real failure cases, don't invent them from thin air | | Suite | A collection of related Tasks | Distinguish "capability assessment" (how hard can it go) from "regression testing" (does it still work) | | Grader | Scoring logic | If code can judge it, never use a model; if a model can judge it, never use a human |

Group 3: Infrastructure

| Element | What it is | Why it matters | |---------|-----------|----------------| | Eval Harness | The engine that runs evals | Concurrent execution, step logging, result aggregation—CI/CD for agents | | Agent Harness | The agent's scaffolding | Claude Code is a harness; the Agent SDK is a lower-level one. You evaluate "model + harness" as a unit |

3. Three Types of Graders

Code-based Grader (First choice)

Methods: string matching, unit tests, static analysis, state checks, tool-call verification.

Pros: fast, cheap, objective, reproducible.

Cons: brittle—an agent that solves the problem legitimately but unexpectedly gets marked wrong.

Real Anthropic case: Opus 4.5 discovered a policy loophole in a τ2-bench flight-booking task, technically "failing" the eval while actually delivering a better user outcome. This exposes the blind spot: code graders judge "did it follow the expected path," not "did it find a better solution."

Model-based Grader (Second choice)

Methods: rubric scoring, natural language assertions, pairwise comparison, reference comparison.

Pros: flexible, captures nuance, fits open-ended tasks.

Cons: non-deterministic, expensive, requires human calibration.

Key trap: LLM grader scores drift—the same output might get 8 today and 6 tomorrow. Calibrate regularly against human labels.

Human Grader (Benchmark)

Pros: gold standard. Cons: slow, expensive, doesn't scale.

Best practice: use human annotation to calibrate LLM graders, not to judge every case directly.

4. Two Overlooked Statistical Traps

Trap 1: pass@k vs pass^k

pass@k: probability of at least one success in k attempts. pass@1 = 50% → pass@4 ≈ 94%.
pass^k: probability that all k attempts succeed. 75% per-run success → pass^3 = 0.75³ ≈ 42%.

Engineering reports "75% success rate"; the PM hears "it usually works." But the real user experience: a 58% chance of at least one failure in three consecutive runs.

Users want pass^k (works every time); engineering measures pass@k (works sometimes). The gap between them is exponential math.

Trap 2: Eval Saturation

When your eval suite scores approach 100%, the agent isn't perfect—your eval is too easy. The agent may still fail on unseen tasks your eval doesn't cover.

Anthropic's advice: when scores saturate, add harder tasks, don't celebrate.

5. Evaluation Strategies by Agent Type

Coding Agent

Conversational Agent (support/sales)

Research Agent

Computer Use Agent

6. Feynman Perspective: Do We Actually "Understand"?

Naming ≠ understanding. We have Task, Trial, Grader, Harness... but terminology is a map, not the territory.

The real question: who evaluates the grader's accuracy? LLM graders need human calibration, but human annotators disagree with each other. This introduces a meta-problem: the tool evaluating the agent needs evaluating, and the tool evaluating that tool... infinite regress. The engineering answer: set a "good enough" threshold instead of pursuing absolute correctness.

Cargo-cult detection: Many teams see Anthropic's framework and start writing YAML, building CI/CD, running eval suites. But if Tasks are invented rather than extracted from real failures, the eval is testing a fictional world. Anthropic repeatedly stresses: extract Tasks from real user complaints, production incidents, and core paths—otherwise you're testing "can the agent pass an exam," not "can it solve user problems."

The most interesting question: Anthropic's framework evaluates "agent harness + model," but users experience the *product*—UI, post-processing, human fallback, monitoring. A 100% agent eval score doesn't mean good product experience, because evals test ideal environments while products face messy reality: network jitter, chaotic user input, degraded third-party APIs. Agent evals are the first line of quality defense, not the last.

7. Practical Roadmap: Building Evals from 0 to 1

Anthropic's 8-step path:

1. Start: write 20–50 Tasks, extracted from real failures 2. Define success criteria: unambiguous + with reference solutions 3. Choose graders: code-based first, model-based second 4. Build the harness: environment isolation (each Trial independent), explicit resource limits 5. Run it: test regression first (protect known capabilities), then capability (expand new ones) 6. Analyze failures: read Transcripts, not just scores 7. Prevent saturation: regularly add harder tasks 8. Keep iterating: version-control the eval suite, close the loop with production monitoring

8. References

Core source: Anthropic Engineering Blog. *Demystifying evals for AI agents*. https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents
Related: Anthropic. *Quantifying infrastructure noise in agentic coding evals*. https://www.anthropic.com/engineering/infrastructure-noise
Related: Anthropic. *Designing AI-resistant technical evaluations*. https://www.anthropic.com/engineering/AI-resistant-technical-evaluations
Related benchmarks: SWE-bench Verified, Terminal-Bench, τ2-Bench, WebArena, OSWorld, BrowseComp
Video walkthrough: https://www.bilibili.com/video/BV1DCrVBREA8

---

> Final thoughts: The most striking detail isn't the terminology or framework—it's Anthropic's candid admission that Opus 4.5 "failed" an eval because it found a better solution than the test designer anticipated. This reveals evaluation's fundamental dilemma: you test "does it match expectations," but an agent's value lies precisely in "exceeding expectations." > > The delicate balance: evals must protect the floor (regression tests) without killing the ceiling (creative solutions). Code graders protect the floor, human review protects the ceiling, and LLM graders handle the middle ground. All three are indispensable. > > And remember the pass@k vs pass^k trap: if your agent has 75% per-run success, don't tell users "it'll probably work." Be honest: "across three consecutive uses, there's a 58% chance of at least one failure." Numbers don't lie, but how you present them can deceive.

*Research date: 2026-05-16 · Source: Anthropic Engineering Blog*