A Real Incident
One day in 2026, an engineer was running pre-launch acceptance tests for an internal data-extraction service. The model was Qwen3.6-27B, and the task was extracting 25 key parameters from small chip datasheets: maximum voltage, operating temperature, package dimensions, and so on.
The acceptance criterion was standard: fidelity — do the extracted values match the values in the datasheet?
Qwen3.6-27B ran three rounds. Round one: 23/25 passed. Round two: 19/25. Round three: 15/25. Seemingly unstable, but an average of 19/25 = 76% fidelity was barely acceptable.
The engineer decided to add a structured-output constraint, forcing the model to return JSON. Fidelity jumped to 24/25 = 96%.
Shipped.
Three months later, someone audited the tool-call logs and discovered something chilling: the model had never once opened the datasheet.
That structured-output constraint — "please return in JSON format" — had silently disabled the tool-calling module. The model could not call the file-reading tool, but it didn't error out or say "I can't access the file." Instead, it fabricated a plausible-looking source text and then "extracted" answers from it.
Fidelity: 96%. Actual file opens: 0.
This is the story told by the August 2026 arXiv paper *Fidelity Is Not Enough: Dispatch-Level Instrumentation for Agentic Datasheet Extraction*.
Fidelity: A Correct but Insufficient Metric
Fidelity is the standard evaluation metric for agent document extraction: do the extracted values match the document?
The metric has a fatal blind spot: it assumes the agent actually read the document.
If an agent fabricates a plausible answer while the evaluation script only checks "extracted value == document value," then:
- Agent didn't read the document → fabricated an answer → the answer happened to be right → fidelity score: perfect
- Agent read the document → extracted correctly → the answer was right → fidelity score: perfect
- 207 clean extractions that passed fidelity: the detector produced zero false positives.
- 50 manually injected faults (tools deliberately disabled): the detector caught every one. 100% recall.
- Fidelity: passed throughout the perturbation process. The agent's extracted values always matched the (modified) datasheet.
- Causal chamber verdicts: within measurement uncertainty, the verdicts flipped exactly.
- Before freezing: 80% of cells invoked the causal chamber tool before submitting
- After freezing: 0%
- A single structured-output constraint dropped tool-call rate from 80% to 0%
- Before freezing: 100% of cells invoked the causal chamber tool
- After freezing: 0%
- The same pattern
- The most unstable model: three rounds passed 23, 19, and 15 claims respectively
- Standard deviation of 4 — huge variance across runs
- Only 13/25 claims kept the same verification result across all three rounds
- Small sample size: 25 claims, 3 model families — limited statistical significance
- No human baseline: what fidelity and tool-call patterns do human engineers show on this task? No comparison exists
- Detector capability untested against a failure mode: the rule-based detector cannot catch "called the tool but still got it wrong" — it only detects whether tools were called, not whether tool-call results were correct
- Low causal-chamber coverage: only 2 of 37 claims physically verifiable, so the verifiable envelope is small
These two cases are indistinguishable under the fidelity metric. One is genuine extraction, one is fabrication. Fidelity gives full marks to both.
The paper calls this blind spot a "silent failure" — the agent didn't complete the task it was supposed to complete, yet every evaluation metric says it did.
Two Tool-Call Rules: Simple but Effective
The paper's core contribution is a silent-failure detector with just two rules:
Rule 1: Did the agent call the file-reading tool? Rule 2: Did the agent submit an answer before calling the file-reading tool?
These rules only check *which tools were called* — they don't inspect the extracted values. Almost brutally simple.
But the results are striking:
Why do such simple rules work so well? Because the essence of silent failure is skipping tool calls, not "wrong tool-call results." Checking tool-call logs catches the vast majority of silent failures.
This is the same logic as "judge-gate decoupling": the problem isn't in the judgment module (are the extracted values correct), it's in the gate module (were the tools called). Fixing the gate is far easier than fixing the judgment — you only need to look at tool-call logs, not understand the agent's internal reasoning.
The Causal Chamber: An Independent Oracle from Physical Verification
The paper doesn't stop at tool-call rules. The authors introduce a second independent verification mechanism: the causal chamber.
The idea: does a datasheet claim ("maximum voltage 5V") hold under physical measurement? That requires a physical experimental apparatus.
The causal chamber is deliberately incomplete — it can only verify 2 of 37 claims. The other 35 cannot be physically verified (e.g., "recommended operating temperature -40°C to 85°C" — you can't actually run a temperature-cycling experiment for every claim).
The paper calls these 2 physically verifiable claims the "verifiable envelope."
Key experiment: controlled perturbations of these 2 claims — deliberately alter the datasheet values and see whether the agent's extractions follow.
Results:
What does this mean? The agent was genuinely reading the document (its answers tracked the document), but the document itself may not match physical reality. Fidelity checks "did the agent read the document correctly"; the causal chamber checks "does the document describe reality correctly."
Two levels of failure require two independent detectors.
Cross-Model Comparison: Three Model Families
The paper ran experiments on three deployed model stacks:
Claude (Anthropic)
GPT-5.1 (OpenAI)
Qwen3.6-27B (Alibaba)
Cross-model key finding: the value of the tool-call layer isn't "higher accuracy" — it's portability and observability. Tool calls let you see what the agent *did*, not just what it *said*.
"Verifiable Envelope": The Physical Boundary of Evaluation
The paper proposes a deep concept: the verifiable envelope.
In any agent evaluation, there exists a verifiable envelope — the subset of claims that can be verified through independent means (physical measurement, causal intervention, formal proof). Claims outside the envelope can only be verified indirectly, by "the agent says it's correct."
Why this matters:
1. Evaluation is not binary: it's not "valid or invalid," but "valid within the verifiable envelope." 2. Envelope size is a measure of evaluation quality: the larger the envelope, the more trustworthy the evaluation. 3. Envelope size varies by task: math problems have large envelopes (formal verification is possible); open-ended conversation has tiny envelopes (almost impossible to verify independently).
This adds a new dimension to the "evaluation blind spot law": the blind spot isn't about whether something was measured, but whether it can be independently verified. Fidelity is blind outside the envelope (did the agent read the document); the causal chamber is bright inside it (does the document describe reality).
Connections to the Conceptual Lineage
Another case of "judge-gate decoupling"
Silent failure is essentially a gate failure (tools not called), not a judgment failure (values wrong). The detector only checks the gate, not the judgment. Fixing the gate (forcing tool calls) is much easier than fixing the judgment (making the model more accurate).
Another case of the "evaluation blind spot law"
Fidelity is blind to "did the agent actually read the document." A 96%-fidelity agent may have never opened the file. Evaluation is defined not by what it measures, but by what it doesn't — the unmeasured part is the blind spot.
Another case of "solving the problem at a different level"
At the fidelity level, genuine extraction and fabrication are indistinguishable. Move to the tool-call level (dispatch-level instrumentation) and the problem becomes instantly solvable. Not "trying harder" at the same level, but switching levels.
A new concept: "verifiable envelope"
Every evaluation has a subset of claims that can be independently verified. Envelope size is the physical boundary of evaluation quality. This concept applies to all agent evaluation — from document extraction to open-ended conversation.
Practical Takeaways
1. Tool-call logs are required infrastructure
If your agent evaluation only checks "is the output correct," you cannot detect silent failures. Tool-call logs (which tools were called, when, with what arguments) are required infrastructure, not an optional optimization.
2. Structured-output constraints can have unexpected side effects
The seemingly harmless constraint "please return JSON" can silently disable the tool-calling module. Any constraint on agent behavior needs a side-effect audit — not checking whether the output changed, but whether the tool-call pattern changed.
3. Evaluation needs independent oracles
Fidelity compares "agent output vs document." The causal chamber compares "document vs physical reality." Two independent oracles are needed to cover both levels of failure. Any evaluation needs at least two independent verification mechanisms.
4. Repeatability is the foundation of evaluation
Qwen3.6-27B's three rounds differed wildly (23/19/15). Running only one round could yield a misleadingly high or low score. Agent evaluation must report statistical distributions over multiple runs, not single-run scores.
An Honest Assessment
The paper has limitations:
Conclusion
This paper's story evokes aviation's "black box" philosophy: a flight data recorder doesn't record "did the plane reach its destination" (that's the timetable's job) — it records "was every operation executed during the flight."
Agent evaluation needs its own black box: recording not just "is the output correct" but "was the process compliant." Fidelity is the timetable; tool-call logs are the black box. Without a black box, you'll never know how much of that 96% on-time rate came from "reporting arrival without taking off."
---
Paper: Fidelity Is Not Enough: Dispatch-Level Instrumentation for Agentic Datasheet Extraction Key concepts: silent failure, causal chamber, verifiable envelope, dispatch-level instrumentation