Stop Grading AI Agents Like Verdicts: Why They Need a Full Diagnostic Report

Forum topic · QianXun · 2026-05-16

Summary

A Chinese tech forum post discusses a Deepchecks arXiv paper, "Holistic Evaluation and Failure Diagnosis of AI Agents," which argues that the bottleneck in AI agent progress is not model intelligence but outdated evaluation methods. Typical LLM-as-judge approaches ask a stronger model to read an agent's entire trace and give an overall pass/fail verdict, but long traces cause attention drift and vague, imprecise failure attributions. The paper proposes a holistic diagnosis combining top-down macro review (checking plan, strategy, and resource allocation) with bottom-up span-level inspection, where each LLM call and tool use is evaluated independently and in depth. According to the post, experiments show that switching from a single holistic judgment to this layered, span-based diagnosis improves failure localization accuracy by 12.5x using the exact same underlying model. The takeaway: agent progress depends on transparently decomposing failures rather than labeling them, and evaluation should examine an agent's full trajectory, not just its endpoint.

Imagine spending weeks designing an AI agent for a 100-step task, only to watch it fail at the very last step. Most people's reaction is a disappointed sigh — "AI still isn't good enough" — followed by a simple Fail label.

That approach, argues a recent forum post on zhichai.net, is like seeing a student fail a math exam and concluding they're "dumb" without ever checking which problem they got wrong or which formula they mixed up. Result-oriented evaluation, the post says, is becoming the biggest roadblock to progress in AI agents.

In May 2026, a research team from Deepchecks published an arXiv paper aimed at changing this: "Holistic Evaluation and Failure Diagnosis of AI Agents." Their core claim is provocative: the current bottleneck isn't that AI isn't smart enough — it's that our evaluation methods are outdated.

Why the "All-Powerful Judge" Goes Blind

Today's standard evaluation approach uses an "all-powerful judge" — a stronger LLM (e.g., GPT-5) — to read an agent's entire work trace and ask: "How did it do?"

The fatal flaw: judges get tired too. When a trace stretches to tens of thousands of words, the judge model tends to forget earlier details by the time it reaches the end. It can tell the task failed, but it can't pinpoint whether the error was a bad plan at step 5 or a misplaced bracket in a tool call at step 58.

The result is diagnostic error: a small local component fails, but the judge delivers only a vague overall verdict.

The Fix: A Full-Hospital Consultation (Holistic Diagnosis)

Invoking Feynman — "The first principle is that you must not fool yourself and you are the easiest person to fool" — the paper proposes a pincer-style diagnosis from two directions:

1. Top-Down Macro Diagnosis

Like a hospital's attending physician, this step looks at overall condition first: Is the plan reasonable? Is the strategy clear? Are resources allocated properly? It's about catching big-picture drift before diving into details.

2. Bottom-Up Span-Level Inspection

This is the paper's most distinctive contribution. Instead of asking the judge to read everything at once, the task trajectory is split into tiny "spans":

Each LLM call is one span
Each tool use is one span

The judge then examines each span independently and in depth — like examining every single pass with a magnifying glass instead of watching a highlight reel.

How Well Does It Work?

According to the post, the paper's experiments reveal a striking result:

Using the exact same LLM, switching the evaluation method from "one holistic read" to this "layered span-based" consultation model improved failure localization accuracy by 12.5x.

The implication: much of the time we think an AI is "hopeless," we simply haven't located the right lesion.

Why It Matters

The post argues that the future of AI agents no longer depends on who can build the biggest black box, but on who can most transparently decompose it. When we can diagnose each of an agent's errors with the precision of a watchmaker — through layered diagnosis — we take the most solid step toward truly autonomous driving, automated programming, and fully automated office work.

Bottom line: Don't judge an agent by its endpoint; examine its footprints. The next time your agent fails, don't rush to mark it wrong — slice its long trace into fragments and find out exactly which step slipped on the banana peel.

True understanding comes from exhaustively dissecting the details of failure.