Anthropic's Agent Evals Framework, Explained Feynman-Style
> Core conclusion upfront: The very capabilities that make agents powerful (autonomy, intelligence, flexibility) are what make evaluation hard. Anthropic turns agent evals from mysticism into engineering with an "eight-component" framework: three dynamic execution components (Trial/Transcript/Outcome), three static definition components (Task/Suite/Grader), and two infrastructure components (Eval Harness/Agent Harness). The most counterintuitive insight: you're not evaluating the model — you're evaluating the coupled "model + scaffolding" system. Plus a statistical trap: an agent with a 90% success rate run three times passes all three attempts only 73% of the time — between users' expectation of "works every time" and engineering's "works sometimes" lies an entire mathematical world.
1. Why This Deserves a Deep Dive
This is not a paper — it's a battle-tested summary from Anthropic's engineering team, distilled from iterating on Claude Code, and it's becoming the de facto industry standard for agent evaluation.
Background: Claude Code's turning point from "iterate fast on intuition" to "scale with evals" happened when users started complaining "why does every update make it worse?" Without evals, the team was flying blind.
2. The Eight Components of Agent Evaluation
The eight components fall into three groups:
Group 1: Dynamic Execution (Runtime)
| Component | What it is | Key insight | |------|--------|----------| | Trial | One attempt | Agents are stochastic; you must run multiple trials for reliable conclusions | | Transcript | Full trajectory log | Don't just look at the final answer — look at *how* it got there: tool call order, reasoning chain, intermediate errors | | Outcome | Final environment state | An agent saying "booked" doesn't mean it's actually booked — check the database |
The hardest distinction: Transcript vs Outcome
Imagine a flight-booking agent:
- Transcript = what it said, which APIs it called, how it reasoned
- Outcome = whether the booking record actually exists in the SQL database
- pass@1 = 50% → one run, half the time it works
- pass@4 ≈ 94% → four runs, very likely at least one success
- 75% single-run success → pass^3 = 0.75³ ≈ 42%
- SWE-bench Verified: agent gets a GitHub issue; run the test suite after it edits code
- Terminal-Bench: end-to-end technical tasks (compiling kernels, training models)
- Evaluate: outcome (tests pass) + trajectory (code quality, tool call efficiency)
- tau-Bench / tau2-Bench: an LLM plays the user; the agent handles multi-turn interaction
- Evaluate: ticket resolved (state check) + under 10 turns (trajectory constraint) + appropriate tone (LLM rubric)
- BrowseComp: tests finding a specific answer in a sea of information
- Strategy: groundedness (are citations real), coverage (are key points covered), source quality (are sources authoritative)
- WebArena: browser tasks, checking URLs and page state
- OSWorld: full OS control, checking file systems, app configs, databases
- Source article: Anthropic Engineering Blog. *Demystifying evals for AI agents*. https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents
- Companion: Anthropic. *Quantifying infrastructure noise in agentic coding evals*. https://www.anthropic.com/engineering/infrastructure-noise
- Companion: Anthropic. *Designing AI-resistant technical evaluations*. https://www.anthropic.com/engineering/AI-resistant-technical-evaluations
- Related benchmarks: SWE-bench Verified, Terminal-Bench, tau2-Bench, WebArena, OSWorld, BrowseComp
Anthropic's own words: *"Agent might say 'Your flight has been booked' at the end of the transcript, but the outcome is whether a reservation exists in the environment's SQL database."*
Too many teams only look at what the agent said, not how the world changed.
Group 2: Static Definitions (Test Cases)
| Component | What it is | Practical advice | |------|--------|----------| | Task | A single test case (input + success criteria) | Extract from real failure cases, don't invent from thin air | | Suite | A collection of related Tasks | Distinguish "capability evals" (how hard can it go) from "regression tests" (does it still work) | | Grader | Scoring logic | If code can judge it, never use a model; if a model can judge it, never use a human |
Group 3: Infrastructure
| Component | What it is | Why it matters | |------|--------|----------| | Eval Harness | The engine that runs evals | Concurrent execution, step logging, result aggregation — it's CI/CD for agents | | Agent Harness | The agent's scaffolding | Claude Code is a harness; the Agent SDK is a lower-level harness. You evaluate the "model + harness" combination |
3. Three Types of Graders: Code, Model, Human
Code-based Grader (preferred)
Methods: string matching, unit tests, static analysis, state checks, tool call verification.
Pros: fast, cheap, objective, reproducible.
Cons: brittle — an agent solving the task in a legitimate way you didn't anticipate gets marked wrong.
Anthropic's real case: Opus 4.5, on a tau2-bench flight booking task, discovered a policy loophole — technically "failing" the eval while actually delivering a better solution for the user. This exposes the blind spot of code-based graders: they can only judge "did you follow the expected path," not "did you find a better solution."
Model-based Grader (second choice)
Methods: rubric scoring, natural language assertions, pairwise comparison, reference comparison.
Pros: flexible, captures nuance, suited to open-ended tasks.
Cons: non-deterministic, expensive, needs human calibration.
Key trap: LLM grader scores drift — the same output might get 8 today and 6 tomorrow. Calibrate regularly against human labels.
Human Grader (the baseline)
Methods: expert review, crowd judgments, sampling checks, A/B testing.
Pros: gold standard. Cons: slow, expensive, doesn't scale.
Best practice: use human labels to calibrate LLM graders, not to grade every case.
4. Two Overlooked Statistical Traps
Trap 1: pass@k vs pass^k
pass@k: probability of at least one success in k attempts
pass^k: probability all k attempts succeed
What does this mean?
Engineering reports "our agent has a 75% success rate"; the product manager hears "users succeed most of the time." But the actual user experience: a 58% chance of at least one failure across three consecutive uses.
Users want pass^k (works every time); engineering measures pass@k (works sometimes). The gap between them is exponential math.
Trap 2: Eval Saturation
When your eval suite scores near 100%, it doesn't mean "the agent is perfect" — it means "the eval is too easy." Agents may still fail on unseen tasks; your eval just doesn't cover them.
Anthropic's advice: when eval scores saturate, add harder tasks instead of celebrating.
5. Evaluation Strategies for Four Agent Types
Coding Agent
Core logic: code is naturally verifiable
Conversational Agent (support/sales)
Core challenge: "task completion" and "experience quality" are two dimensions
Research Agent
Core challenge: no single correct answer
Computer Use Agent
Core challenge: token efficiency vs latency in GUI interaction
6. The Feynman Lens: Do We Really "Understand"?
"Naming ≠ understanding"
We now have Task, Trial, Grader, Harness... a full vocabulary. But naming is a map, not the territory.
The real question: who evaluates the accuracy of the grader itself?
Anthropic admits LLM graders need human calibration — but human annotators disagree with each other too. This introduces a meta-problem: your agent-evaluating tool itself needs evaluating, and that meta-evaluation tool needs meta-meta-evaluation... an infinite regress.
The engineering answer: set a "good enough" threshold instead of chasing absolute correctness.
"Cargo-cult detection"
Many teams see Anthropic's framework and start writing YAML, building CI/CD, running eval suites. But if Tasks are invented from imagination (rather than extracted from real failures), the eval is testing a fictional world.
Anthropic repeatedly stresses: Tasks should come from real user complaints, production incidents, and critical paths. Otherwise you're testing "can the agent pass an exam," not "can the agent solve users' problems."
The most interesting question: evaluating the agent vs evaluating the system
Anthropic's framework evaluates "agent harness + model." But users actually use the "product" — including UI, post-processing, human fallback, monitoring and alerting.
A 100% agent eval score doesn't mean a good product experience, because evals test an idealized environment while products face messy reality: network jitter, chaotic user input, degraded third-party APIs.
Implication: agent evals are the first line of quality assurance, not the last.
7. A Practical Roadmap: Building an Eval System from 0 to 1
Anthropic's 8-step roadmap:
1. Start: write 20–50 Tasks, extracted from real failures 2. Define success criteria: unambiguous + with reference solutions 3. Choose graders: code-based first, model-based second 4. Build the harness: environment isolation (each Trial independent), explicit resource limits 5. Run it: test regression first (protect known capabilities), then capability (extend new ones) 6. Analyze failures: read transcripts, not just scores 7. Prevent saturation: regularly add harder tasks 8. Keep iterating: version-control the eval suite; close the loop with production monitoring
8. References
> Closing thought: The most striking part of Anthropic's post isn't the terminology or framework — it's their candid admission: Opus 4.5 "failed" an eval because it found a better solution than the question writer. This reveals the essential dilemma of evaluation — you test "did it match expectations," but an agent's value lies precisely in "exceeding expectations." > > The finest balance: evals should protect the floor (regression tests) without killing the ceiling (creative solutions). Code graders protect the floor, human review protects the ceiling, and LLM graders handle the middle ground. You need all three. > > And that pass@k vs pass^k statistical trap: if your agent's single-run success rate is 75%, don't tell users "it'll probably work." Be honest: "across three consecutive uses, there's a 58% chance of at least one failure." Numbers don't lie, but the way you present them can deceive.
*Research date: 2026-05-16* *Source: Anthropic Engineering Blog* *Feynman thinking framework applied*