The warehouse that always audits perfectly
Every month, the warehouse supervisor receives an immaculate inventory sheet: opening stock, closing stock, in/out flows—zero discrepancy. Yet inside the warehouse, goods were shuffled to hit the final number: stock borrowed between zones, defects mixed with good units, a crate left on the dock forgotten. The sheet is perfect; the warehouse is a mess.
That is the metaphor for the final installment of the "price of capability" series. After Mímir (immutable gates cost money) and HydroJEV (fast screening with dual judges cuts review load 35-38% but misses 66% of covert attacks), today's paper turns the bill to page three: what happens when an enterprise AI agent's acceptance testing only checks "is the final answer right?"
The paper is Continual Learning for Enterprise AI Agents (CLEA), posted to arXiv in October (arXiv:2610.01833, cs.AI), by Aarya Doshi (Georgia Tech) and Vadim Sheinin (IBM Research), completed during Doshi's summer internship at IBM. It runs on IBM's real enterprise system—not a toy benchmark.
The pipeline where one error pollutes the whole village
IBM's VAR (Value Aware Resiliency) system includes a skill called BVD (Business Value Determination): given applications, a date range, and annual revenue, it queries observability APIs for CPU, memory, and call metrics, then apportions revenue proportionally (default weights: CPU 40%, memory 40%, calls 20%) and writes results to a persistent database that all downstream skills read. BVD is the upstream water source—when its output is wrong, it doesn't explode locally; it pollutes every downstream ledger.
The paper studies two skills with different structure:
- Revenue Apportionment: 9 applications, CPU/memory/call inputs, host- and app-level revenue tables, 104 checks.
- Productivity Apportionment: additionally pulls Kubernetes costs (Kubecost) and EC2 unblended costs (Cloudability), requires EC2 resource identifiers, host-cost apportionment, productivity (revenue minus cost, with explicit NULL semantics), 80 checks.
- 175 of 240 runs passed all applicable final-value checks. Of these, 162 (92.6%, Wilson 95% CI 87.7-95.6%) still hid at least one evaluator-flagged deviation. Final checks alone missed deviations in 67.5% of trials. Overall, 227 runs (94.6%) failed at least one check.
- Revenue's extreme consistency: all 120 trials passed the final check; all 120 contained trajectory deviations. Zero exceptions. Robustness checks (excluding externally-errored runs) barely move the number: 91.0% and 92.5%.
- Expanding to the seven-item Level 2 doesn't close the crack: of 164 runs passing all seven, 151 (92.1%, CI 86.9-95.3%) still violated trajectory checks. Revenue: 109/109; Productivity: 42/55 (76.4%).
- Data/numeric consistency deviations: 106 — stored values don't match the values implied by the process.
- Missing stages or required tools: 66 — e.g., writing results without a database read-back.
- Allowlist violations: 55 — using tools the spec didn't approve.
- Redundant/call-count violations: 43.
- Scope/filter violations: 13 — global queries where cluster/host-scoped ones were required.
- Tail: 2 response-format, 1 tool-execution.
- Doshi, A., & Sheinin, V. Continual Learning for Enterprise AI Agents (CLEA): Continual Process-Level Evaluation for Evolving Enterprise AI Agent Skills. arXiv:2610.01833 (cs.AI, 2026). https://arxiv.org/abs/2610.01833
- Institutions: Georgia Tech, IBM Research
Real enterprise setup: 9 applications, 3 infrastructure platforms (2 K8s clusters + 1 EC2 host), $1.2M annual revenue, utilization from Instana. The agent has 41 tools (23 stateless + 18 with SQLite persistence).
What "final check" means: an exam that only grades the last question
CLEA defines three evaluation levels. Level 1: preset final numeric results. Level 2: seven items (Level 1 values plus stored values, normalized sums, response structure). Level 3: the remaining ~100 trajectory checks—tool choice, parameters, call order, query scope.
Industry practice validates agent skills at Level 1: run once, compare final numbers, stamp, deploy. But a check that only verifies the final inventory number cannot see how goods were moved. And the crack is especially wide for enterprise agents because skills are not frozen artifacts—they evolve along four axes:
1. Tool API evolution: renamed parameters, changed scope semantics—skills silently break. 2. Spec revisions: documentation updates can change planning strategies in ways final checks can't catch. 3. LLM version changes: default planning choices drift from conservative to aggressive. 4. Harness changes: migrating Claude Code to Codex can transform process-level deviation patterns even when final-check precision looks the same.
CLEA's core claim: enterprise agent evaluation must be continual, process-level evaluation; the final check is only the cheapest and least reliable layer.
240 trials, and a 92.6% crack
The matrix: 2 skills x 2 spec variants x 2 harnesses x 3 models x 10 repetitions = 240 fully automated trials. Spec variants: MD (SKILL.md) vs TXT (Skill.txt); harnesses: Claude Code and Codex; models: GPT-5.6-Sol, Claude Opus 4.8, Claude Sonnet 4.6.
Results:
What the deviations look like: root-cause families and cascading attribution
Across the 162 "numbers right, process wrong" runs (categories non-exclusive):
An ablation confirms these aren't artifacts of one pedantic check: removing the "irrelevant tool" check, all 162 runs still fail on other dimensions.
Since a run averages 6.34 failing checks, CLEA adds cascading attribution: a hand-written dependency graph distinguishes root failures from downstream effects, compressing 6.34 to 2.65 root failures per run (58% reduction); median 4 tickets to 2. Revenue's top roots: incorrect staged utilization parameters (75/120), allowlist-violating tools (42), missing DB read-back (24). Productivity's: incorrect app inputs to computation tools (73), incorrect total cost (40), missing app-filtered K8s queries (38), missing app-filtered EC2 queries (31).
Harness inversion: no universally "obedient" model
The most counterintuitive result: spec sensitivity rankings completely invert across harnesses. On Claude Code, GPT-5.6-Sol is robust to the MD/TXT spec variant (−1.2pp) while Opus 4.8 and Sonnet 4.6 show moderate sensitivity (+9.7, +6.3pp). On Codex the pattern flips: Sonnet 4.6 is most robust (−0.4pp) while GPT-5.6-Sol and Opus 4.8 show the largest TXT degradation (+12.9, +11.8pp).
"Which model follows the spec better" has no universal answer—it lives in the coupling of a specific model and a specific harness. Any comfort from "we tested it on Claude Code" is void the moment you migrate harnesses.
Two more empirical notes: Claude Code was consistently more precise on Revenue's TXT variant (86.8–95.7% vs 82.1–92.6%); unrecovered errors clustered on Codex (38 vs 5), with the Productivity Codex-Sonnet configuration hitting at least one unrecovered error in all 20 runs.
The bill: 240 runs cost $859.34 total (mean $3.58, median $1.23, max $30.23). Claude Code averaged $1.04 vs Codex's $6.12—nearly 6x—driven by Claude models' cache-read share (90.7–92.9% on Claude Code, 0% on Codex). Same models, same skill, six-fold cost difference by harness—another irony for "process over outcome."
The five-stage pipeline: turning "process" into a reusable artifact
1. Capture: a one-line spec instruction logs parsed inputs; full tool-call trajectories (names, parameters, returns, turn indices) come from harness logs.
2. Independent ground truth: a separate Python script calls the same real APIs via a path fully independent of the skill. LLMs never generate oracles—the paper cites HumanEval's lesson: 88.89% of erroneous test cases came from wrong output oracles.
3. Templated test cases: placeholders (e.g., $k8s_host_date_calls) resolved from ground truth at evaluation time, with dependency links between checks. The same registered suite was reused across all 24 configurations; one check ID covered 4 different Revenue and 16 different Productivity expected states. This is unit testing for agent skills—something most enterprises don't have.
4. Three-level evaluation: Level 1 finals, Level 2 seven items, Level 3 ~100 trajectory checks along four dimensions: tool selection, tool parameters, tool ordering (fetch→stage→readback→allocate→write), and parameter scope. An LLM judge (claude-sonnet-4-6) is used only where exact matching is brittle; all numeric comparisons go through a calculator tool (the judge is from the same family as an evaluated model—self-preference risk). Manual review of 60 sampled LLM judgments showed 95% agreement.
5. Report root failures: cascading attribution, 6.34 → 2.65.
A stress test: freezing Productivity's modal expectations and running 120 runs caused 80 runs to break at least one value—runtime parsing works under observed variance, but persistence under controlled evolution is unproven. Templates are living artifacts; they age with the data.
Bonus: writing tests itself improves the spec
Three times while writing test cases, the evaluator hit "cannot compute the expected value"—each time exposing an implicit policy choice the spec never stated. The most typical: EC2 memory returns host-level or JVM-level readings that cannot physically be split per application—yet the spec never said what to store when splitting is impossible. Similar issues surfaced with DB read-back ordering and EC2 call scope.
Continual evaluation doubles as a spec-completeness tool. Every test is a probe poked at the spec document; wherever it feels soft, the spec gets patched. Test-driven development returns in agent-era clothing.
Limitations, laid on the table
Two skills in one enterprise system—generalization open. Only 10 repetitions per cell—limited power for small effects. MD vs TXT are bundled variants, not controlled format interventions. Aggregate scores mix heterogeneous checks without severity weighting. 83 runs hit at least one tool/harness error. The LLM judge shares a family with an evaluated model. The comparison is cross-sectional, not longitudinal tracking of API/model upgrades—the dynamic "time" axis is barely touched. Finally, the framework diagnoses but does not repair.
Closing: gates, bills, and cracks
Three papers, one wall. Mímir: entering the physical world requires building a gate—unlocking costs money. HydroJEV: fast screening saves review cost—but carries a blind-spot bill. CLEA: skipping process checks to keep only final checks—92.6% of trajectory deviations slip silently through the crack; in Revenue's cell, 120 runs, no exceptions.
Capability is not a switch; it's a contract. You buy "looks right," and the fine print says "the process may be entirely wrong." The 92.6% figure isn't meant to scare anyone—it's a line of fine print to add to every "acceptance passed" report: this conclusion covers final values only; process health is not insured.
The inventory sheet is still the inventory sheet. From now on, push open the door, watch how goods are moved, then decide whether to trust the numbers.
---
References