> Core takeaway upfront: The very capabilities that make agents powerful—autonomy, intelligence, flexibility—are what make them hard to evaluate. Anthropic's 'eight components' framework turns agent evaluation from mysticism into engineering: three dynamic execution components (Trial/Transcript/Outcome), three static definition components (Task/Suite/Grader), and two infrastructure components (Eval Harness/Agent Harness). The most counterintuitive insight: you are not evaluating the model, but the coupled 'model + scaffold' system. And there's a statistical trap: an agent with 90% success rate run 3 times has only a 73% chance of passing all three runs—between the user's expectation of 'works every time' and engineering's definition of 'works sometimes' lies an entire world of math.
---
1. Why Deep-Dive This Blog Post?
This isn't a paper—it's hard-won lessons from Anthropic's engineering team, distilled from iterating on Claude Code, and it's becoming the de facto industry standard for agent evaluation.
Background: The turning point for Claude Code—from 'fast iteration on intuition' to 'scale requires evaluation'—came when users started complaining that updates made things worse. Without evals, the team was flying blind.
---
2. The Eight Components of Evaluation: One Diagram for Agent Testing
The eight components fall into three groups:
Group 1: Dynamic Execution (Runtime)
| Component | What it is | Key insight | |------|--------|----------| | Trial | One attempt | Agents are stochastic—run multiple times for reliable conclusions | | Transcript | Full trajectory record | Don't just look at the final answer; look at *how* it got there—tool call order, reasoning chains, intermediate errors | | Outcome | Final environment state | The agent saying 'booked!' doesn't mean it's booked—check the database |
The hardest distinction: Transcript vs Outcome
Imagine a flight-booking agent:
- Transcript = what it said, which APIs it called, how it reasoned
- Outcome = whether the reservation actually exists in the SQL database
- pass@1 = 50% → one run, half the chance
- pass@4 ≈ 94% → four runs, likely at least one success
- 75% single-run success → pass^3 = 0.75³ ≈ 42%
- SWE-bench Verified: give the agent a GitHub issue, run the test suite after its changes
- Terminal-Bench: end-to-end technical tasks (compiling kernels, training models)
- Evaluate: outcomes (tests pass) + trajectory (code quality, tool-call efficiency)
- 𝜏-Bench / 𝜏2-Bench: an LLM plays the user; the agent handles multi-turn interactions
- Evaluate: ticket resolved (state check) + under 10 turns (trajectory constraint) + appropriate tone (LLM rubric)
- BrowseComp: tests finding specific answers in a sea of information
- Strategy: groundedness (are citations real), coverage (key points covered), source quality (authoritative sources)
- WebArena: browser tasks, check URLs and page state
- OSWorld: full OS control, check filesystems, app configs, databases
- Core source: Anthropic Engineering Blog. *Demystifying evals for AI agents*. https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents
- Related: Anthropic. *Quantifying infrastructure noise in agentic coding evals*. https://www.anthropic.com/engineering/infrastructure-noise
- Related: Anthropic. *Designing AI-resistant technical evaluations*. https://www.anthropic.com/engineering/AI-resistant-technical-evaluations
- Related benchmarks: SWE-bench Verified, Terminal-Bench, 𝜏2-Bench, WebArena, OSWorld, BrowseComp
- Video walkthrough: 慢学AI. *没有评估的Agent，注定不可规模化*. https://www.bilibili.com/video/BV1DCrVBREA8
Anthropic's own words: *"Agent might say 'Your flight has been booked' at the end of the transcript, but the outcome is whether a reservation exists in the environment's SQL database."*
Too many teams make the same mistake: only looking at what the agent *said*, not how the *world changed*.
Group 2: Static Definitions (Test Cases)
| Component | What it is | Practical advice | |------|--------|----------| | Task | A single test case (input + success criteria) | Extract from real failure cases, don't invent from thin air | | Suite | A collection of related Tasks | Distinguish 'capability evaluation' (how hard can it go) from 'regression testing' (does it still work) | | Grader | Scoring logic | If code can judge it, never use a model; if a model can judge it, never use a human |
Group 3: Infrastructure
| Component | What it is | Why it matters | |------|--------|----------| | Eval Harness | The engine that runs evals | Concurrent execution, step logging, result aggregation—this is CI/CD for agents | | Agent Harness | The agent's scaffold | Claude Code is a harness; the Agent SDK is a lower-level one. You evaluate the 'model + harness' combination |
---
3. Three Types of Graders: Code, Model, Human
Code-based Grader (First Choice)
Methods: string matching, unit tests, static analysis, state checks, tool-call verification.
Pros: fast, cheap, objective, reproducible.
Cons: brittle—an agent that solves the task in a valid but unexpected way gets marked wrong.
Anthropic's real case: Opus 4.5, on a 𝜏2-bench flight-booking task, found a policy loophole—technically 'failing' the eval while actually delivering a better user outcome. This exposes the blind spot of code-based graders: they can only judge 'did it follow the expected path', not 'did it find a better solution'.
Model-based Grader (Second Choice)
Methods: rubric scoring, natural-language assertions, pairwise comparison, reference comparison.
Pros: flexible, captures nuance, suits open-ended tasks.
Cons: non-deterministic, expensive, requires calibration against human judgment.
Key trap: LLM grader scores drift—the same output might score 8 today and 6 tomorrow. Calibrate regularly with human labels.
Human Grader (The Benchmark)
Methods: expert review, crowdsourced judgment, sampling audits, A/B testing.
Pros: gold standard.
Cons: slow, expensive, unscalable.
Best practice: use human labels to calibrate LLM graders, not to grade every case.
---
4. Two Overlooked Statistical Traps
Trap 1: pass@k vs pass^k
pass@k: probability of at least one success in k attempts
pass^k: probability all k attempts succeed
What this means: Engineering reports 'our agent has 75% success rate'; the PM understands 'users will probably succeed within three tries'. But the actual user experience: a 58% chance of at least one failure in three consecutive attempts.
Users want pass^k (works every time); engineering measures pass@k (works sometimes). The gap between them is exponential math.
Trap 2: Eval Saturation
When your eval suite scores approach 100%, it doesn't mean 'the agent is perfect'—it means 'the eval is too easy'. The agent may still fail on unseen tasks; your eval just doesn't cover them.
Anthropic's advice: when eval scores saturate, add harder tasks—don't celebrate.
---
5. Evaluation Strategies for Four Agent Types
Coding Agent
Core logic: code is naturally verifiable.
Conversational Agent (Support/Sales)
Core challenge: 'task completion' and 'experience quality' are two dimensions.
Research Agent
Core challenge: no single correct answer.
Computer Use Agent
Core challenge: token-efficiency vs latency trade-offs in GUI interaction.
---
6. The Feynman View: Do We Really 'Understand'?
'Naming ≠ Understanding'
We now have Task, Trial, Grader, Harness... a whole vocabulary. But naming is a map, not the territory.
The real question: who evaluates the accuracy of the grader itself?
Anthropic admits LLM graders need human calibration. But human annotators have their own biases (limited inter-annotator agreement). This introduces a meta-problem: the tool you use to evaluate agents needs evaluating itself, and that meta-evaluation tool needs meta-meta-evaluation... infinite regress.
The engineering solution: set a 'good enough' threshold instead of pursuing absolute correctness.
'Cargo Cult Detection'
Many teams see Anthropic's framework and start writing YAML, building CI/CD, running eval suites. But if Tasks are invented from imagination (rather than extracted from real failures), the eval is only testing a fictional world.
Anthropic repeatedly stresses: Tasks should be extracted from real user complaints, production incidents, critical paths. Otherwise you're testing 'can the agent pass the exam', not 'can the agent solve user problems'.
The Most Interesting Question: Evaluating Agents vs Evaluating Systems
Anthropic's framework evaluates 'agent harness + model'. But users actually use the *product*—including UI, post-processing, human fallback, monitoring and alerting.
A 100% agent eval score doesn't mean good product experience, because evals test an idealized environment while products face messy reality: network flakiness, chaotic user input, degraded third-party APIs.
This means: agent evals are the first line of quality defense, not the last.
---
7. Practical Roadmap: Building an Eval System from 0 to 1
Anthropic's 8-step roadmap:
1. Start: write 20–50 Tasks, extracted from real failures 2. Define success criteria: unambiguous + with reference solutions 3. Choose graders: code-based first, model-based second 4. Build the harness: environment isolation (each Trial independent), explicit resource limits 5. Run it: regression testing first (protect known capability), then capability testing (extend new capability) 6. Analyze failures: read the Transcript, don't just look at scores 7. Prevent saturation: regularly add harder tasks 8. Keep iterating: version-control the eval suite, close the loop with production monitoring
---
8. References
> Closing thoughts: What's most striking about this blog post isn't the terminology or the framework, but a fact Anthropic candidly admits: Opus 4.5 'failed' an eval because it found a better solution than the question-setter. This reveals evaluation's essential dilemma—you test 'does it match expectations', but the agent's value lies precisely in 'exceeding expectations'. > > The subtlest balance: evals must protect the floor (regression tests) without killing the ceiling (creative solutions). Code graders protect the floor, human review protects the ceiling, and LLM graders do the work in between. All three are indispensable. > > And that pass@k vs pass^k statistical trap: if your agent has a 75% single-run success rate, don't tell users 'it will probably work'. Be honest: 'across three consecutive uses, there's a 58% chance of at least one failure.' Numbers don't lie, but the way you present them can deceive.
*Research date: 2026-05-16* *Source: Anthropic Engineering Blog* *Deep research by Xiao Kai* *Feynman thinking framework applied*