Paper: APEX-Accounting Authors: Benchek Julien, Bennett Austin, Kern Jasmin, Stevens Ryan, Sultan Rene, et al. Institutions: Mercor × Ramp arXiv: 2607.27189 Link: https://arxiv.org/abs/2607.27189 Code: https://github.com/Mercor-Intelligence/archipelago
---
A Simple Question: Can AI Actually Do the Books?
Over the past two years we've seen headlines like "AI passes the CPA exam." These scores create the illusion that AI can already do professional work. But there is a huge gap between passing an exam and doing actual work. Exam questions are clean and self-contained. Real accounting work means:
- Finding relevant data across spreadsheets, PDFs, emails, and bank statements
- Judging which account a transaction belongs to
- Multi-step reasoning: reconcile, adjust entries, post, generate statements
- Any error in one step cascades into everything after it
- A full accounting system (ledger, chart of accounts, historical transactions)
- A set of related documents (spreadsheets, PDFs, invoices, bank statements)
- A series of interrelated tasks
- Reconciliation: find discrepancies between bank statements and the ledger
- Accruals: record expenses incurred but not yet paid
- Posting: move entries from journals to the general ledger
- Reports: generate trial balances, income statements, balance sheets
- Substituting wrong baselines or authorization logic: e.g., recognizing revenue by payment date instead of invoice date
- Lost intermediate results: a number computed at step 5 is forgotten by step 7, and the chain continues with the wrong value
- Missed contradictions: inconsistent information across documents (invoice amount vs. bank record) goes undetected
- In aggregate: higher budget → higher scores
- Within a fixed budget: tasks where the model spent more tokens scored *lower*
- Full workflows, not isolated questions: SWE-bench fixes a real GitHub issue; APEX runs a real month-end close
- Both show multi-step reasoning stability is the bottleneck: SWE-bench found models locate code but can't fix it correctly; APEX found models locate data but can't push the reasoning through
- Both show tool use is not the main bottleneck
APEX-Accounting, built by Mercor and Ramp, measures exactly this gap: can frontier models do real accounting work?
---
Benchmark Design: 10 "Worlds", 160 Real Tasks
10 Worlds
APEX-Accounting is not a pile of isolated questions. It contains 10 complete "accounting worlds," each with:
The key design point: tasks are coupled. Task 3 may depend on task 1's result; task 7 may require completing task 5 first. Just like real month-end close — a process, not 20 independent questions.
160 Tasks
16 tasks per world, 160 total, covering core accounting work:
Expert-Level Annotation
Every task was authored, solved, and rubric-scored by accounting and bookkeeping experts. Scoring is not binary but multi-dimensional rubric-based, allowing partial credit and distinguishing "right method, wrong arithmetic" from outright failure.
---
Headline Results: Best 56.4%, No Model Reliably Passes
Nine frontier models were tested. Three numbers tell the story:
56.4%
Claude-Fable-5 (Max) ranks first with 56.4% Mean Criteria@3. Second is Muse-Spark-1.1 (xHigh) at 52.6%. Mean Criteria@3 gives each model 3 attempts and averages rubric scores per attempt — a fairly forgiving metric — yet the best model only reaches 56.4%.
2.6%
Pass^8 (at least one fully passing run out of 8 attempts) peaks at just 2.6%, achieved by GPT-5.6-Sol (Max+Pro). Even with 8 chances, almost no task is reliably completed end-to-end.
Pass@8 peaks at 21.5%
Muse-Spark-1.1 (xHigh) achieves the best Pass@8 of 21.5% — meaning even the best model fails 4 out of 5 tasks in all 8 attempts.
Together: frontier models are nowhere near replacing humans on real accounting work. Barely half under a lenient metric, near-total failure under a strict one.
---
Failure Modes: Not Retrieval — Reasoning
The failure analysis is the most valuable part, with a counterintuitive finding:
Models Find the Right Inputs
Top models are quite good at locating the right files and data. They correctly pull the relevant invoices, bank statements, and contracts from piles of PDFs and spreadsheets. Retrieval failure rates are low.
Models Break on Multi-Step Reasoning
The real failures happen in reasoning:
Tool-Use Errors Are Not the Problem
A notable finding: tool-calling errors are not a major failure mode. Models call APIs, read files, and run computations correctly. The bottleneck isn't "can it use tools" but "can the reasoning chain be completed stably."
This contradicts a common intuition — the Agent bottleneck is not tool use, it's the stability of multi-step reasoning.
---
Simpson's Paradox: More Money, Worse Results (Within Budget)
The paper reports an intriguing Simpson's paradox. When token budgets were raised from $1 to $50:
The plausible explanation: hard tasks make the model spend more tokens (retries, more file lookups), but hard tasks are simply harder to get right. Both "more tokens" and "low scores" are effects of task difficulty, not cause and effect.
Deployment implication: you can't solve hard tasks by throwing more budget at them — the bottleneck is reasoning capability, not resources.
---
Why This Benchmark Matters
1. It measures real work, not exam questions. Unlike most LLM benchmarks, it tests a profession's full daily workflow — the same design philosophy as SWE-bench. It directly ties "model capability" to "professional replaceability": 56.4% doesn't mean the model fails a test, it means the model can only assist, not own the job. 2. Expert-level rubrics. Multi-dimensional scoring distinguishes partial correctness and "right method, wrong computation" — granular enough to diagnose model bottlenecks. 3. It identifies long-horizon reasoning stability as the real bottleneck. Models aren't incapable of accounting — they can't reliably complete a multi-step accounting reasoning chain. 4. The Simpson's paradox has practical meaning. Agent systems cannot rely on "more tokens / more turns" as a fallback for hard tasks.
---
A Cross-Domain Isomorphism: Comparison with SWE-bench
APEX-Accounting mirrors SWE-bench structurally:
---
Limitations
1. Closed benchmark: the 160 tasks are private, limiting community fine-grained debugging, though a leaderboard eval is open to frontier labs. 2. 10 worlds may lack diversity: accounting varies enormously across industries (manufacturing, retail, finance). 3. No human baseline: without expert performance data, it's hard to judge whether 56.4% is high or low. 4. All 9 models are closed frontier models: no open-source models (Llama, Qwen), so scale-vs-capability analysis isn't possible.
---
My Take
What strikes me most is not the 56.4% figure but the structural gap it reveals: passing exams ≠ being competent at work. Between exam-style "clean prompt, single-step answers" and real work, there is at least an order-of-magnitude difference.
Deeper: the failure analysis points to a conclusion relevant to the entire Agent field — long-horizon reasoning stability is the next core bottleneck. Models find the right files, call the right tools, get individual steps right, but cannot reliably chain 10 steps together. That's not an "accounting AI" problem; it's an all-Agent problem.
Finally, remember the Simpson's paradox finding: more budget can't solve hard tasks. For every Agent design that leans on "more compute as a fallback" — the bottleneck on hard tasks is reasoning, not resources. You can't fix what the model fundamentally can't think through by letting it think a few more times.
---
Paper: https://arxiv.org/abs/2607.27189 HTML: https://arxiv.org/html/2607.27189v1 Code: https://github.com/Mercor-Intelligence/archipelago