APEX-Accounting: Testing Whether Frontier Models Can Do Real Accounting Work
Paper: APEX-Accounting Authors: Benchek Julien, Bennett Austin, Kern Jasmin, Stevens Ryan, Sultan Rene, et al. Institutions: Mercor × Ramp arXiv: 2607.27189 Link: https://arxiv.org/abs/2607.27189 Code: https://github.com/Mercor-Intelligence/archipelago
---
The Core Question: Can AI Do My Books?
Over the past two years, headlines have celebrated AI passing certified public accountant exams and bar exams. These results create the illusion that AI can already perform professional work.
However, a huge gap separates "passing an exam" from "doing the actual job." Exam questions are clean, have clear boundaries, and contain all necessary information within the prompt. Real accounting work looks different:
- Finding relevant data across spreadsheets, PDFs, emails, and bank statements
- Judging which account a transaction belongs to
- Multi-step reasoning: reconcile first, then adjust entries, then post, then generate reports
- Any single error propagates downstream
- A full accounting system (ledgers, chart of accounts, historical transactions)
- Related documents (spreadsheets, PDFs, invoices, bank statements)
- A series of interconnected tasks
- Reconciliation: bank statements and ledgers don't match; find the differences
- Accruals: expenses incurred but not yet paid; recognize them in the current period
- Posting: transfer journal entries to the general ledger
- Reports: generate trial balances, income statements, balance sheets
- Substituting the wrong baseline or recognition logic: for example, recognizing revenue by receipt date instead of invoice date
- Losing intermediate results: a number computed at step 5 is forgotten by step 7, and the wrong number is used downstream
- Failing to detect contradictions: conflicting information in documents (such as invoice amount vs. bank record mismatch) goes unnoticed
- Aggregate view: higher budget → higher score
- Within-budget view: at the same budget, models score *lower* on tasks where they spend more tokens
- Both test complete workflows, not isolated problems: SWE-bench fixes a real GitHub issue; APEX performs a real month-end close
- Both reveal multi-step reasoning stability as the bottleneck: SWE-bench finds models can locate code but fix it incorrectly; APEX finds models can locate data but reason incorrectly
- Both reveal tool use is not the main bottleneck: SWE-bench finds models can call tools but reason incorrectly; APEX finds the same
The APEX-Accounting benchmark is designed to measure this gap. Built by Mercor and Ramp, it specifically tests whether frontier models can handle real accounting workflows.
---
Benchmark Design: 10 Worlds, 160 Real Tasks
10 "Worlds"
APEX-Accounting is not a pile of isolated problems. It contains 10 complete accounting worlds, each with:
The key design principle: tasks are coupled. Question 3 may depend on the answer to Question 1, and Question 7 may require Question 5 to be completed first. This mirrors real accounting work—month-end close is a process, not 20 independent questions.
160 Tasks
Each world contains 16 tasks, totaling 160. The task types cover core accounting work:
Expert-Level Annotation
Every question was authored, solved, and graded by accounting and bookkeeping professionals. Scoring is not a binary correct/incorrect but follows a multi-dimensional rubric, which captures partial credit and distinguishes cases such as "right method, wrong calculation."
---
Main Results: Best 56.4%, No Model Passes Reliably
The paper evaluates 9 frontier models. Three numbers tell the story:
Number 1: 56.4%
Claude-Fable-5 (Max) ranks first with a Mean Criteria@3 of 56.4%. The runner-up is Muse-Spark-1.1 (xHigh) at 52.6%.
Mean Criteria@3 means the model gets 3 attempts, and the score is averaged across attempts according to the rubric. This is a relatively lenient metric—3 attempts, dimensional averaging—and even the best model only reaches 56.4%.
Number 2: 2.6%
The highest Pass^8 (proportion of tasks passed at least once out of 8 attempts) is just 2.6%, achieved by GPT-5.6-Sol (Max+Pro).
This is a strict metric—at least one fully correct run out of 8—but 2.6% means almost no task is completed fully correctly by any model consistently.
Number 3: Pass@8 Tops Out at 21.5%
Muse-Spark-1.1 (xHigh) achieves a Pass@8 of 21.5%—the fraction of tasks passed at least once in 8 attempts. This means even the best model fails on 4 out of 5 tasks across 8 attempts.
These three numbers combine into a clear story: frontier models are nowhere near ready to replace humans on real accounting work. The strongest model barely clears half on a lenient metric and almost entirely fails on strict ones.
---
Failure Modes: Not Retrieval, but Reasoning
The paper's failure analysis is especially valuable. The authors categorize failures by stage and find a counter-intuitive result:
Models Find the Right Inputs
The strongest models perform fairly well at finding the right files and data. They correctly locate relevant invoices, bank statements, and contracts from a pile of PDFs and spreadsheets. Failure rates at this stage are low.
Models Get Stuck on Multi-Step Reasoning
The real failures happen in multi-step reasoning:
Tool-Use Errors Are Not the Main Problem
An interesting finding: tool-use errors are not the dominant failure mode. Models can correctly call APIs, read files, and perform calculations. The problem is not "can it use tools?" but "can the reasoning chain run to the end stably?"
This runs counter to common intuition. We expected the bottleneck of "agent capability" to be tool use, but APEX-Accounting shows the bottleneck is multi-step reasoning stability.
---
Simpson's Paradox: Spending More Doesn't Help
The paper uncovers another interesting finding—a Simpson's paradox instance.
The authors raise the model's token budget from $1 to $50 and observe a seemingly contradictory pattern:
This is a classic Simpson's paradox—the trend in aggregated data reverses when looking within groups.
How to explain it? A plausible hypothesis: harder tasks cause models to spend more tokens (retrying, reading more files), but harder tasks are also inherently harder to get right. So both "spending more tokens" and "scoring lower" are consequences of "task difficulty," not a causal relationship.
This has direct implications for deployment: throwing more budget at hard tasks doesn't work. The bottleneck on hard tasks is reasoning ability, not budget.
---
Why This Benchmark Matters
1. It Measures "Real Work," Not "Exam Questions"
The fundamental difference between APEX-Accounting and most LLM benchmarks: it tests an entire profession's daily workflow, not isolated knowledge points. It shares the same design philosophy as SWE-bench (which tests real software engineering work).
The value of such benchmarks: they tie "model capability" directly to "occupational substitutability." 56.4% doesn't mean "model fails"; it means "the model can't lead on this occupation yet, only assist."
2. Expert-Level Rubrics
Scoring uses multi-dimensional accounting rubrics rather than simple correct/incorrect judgments. This granularity distinguishes intermediate states like "right method, wrong calculation" or "partial step correct," which is invaluable for diagnosing model bottlenecks.
3. It Identifies Multi-Step Reasoning Stability as the Real Bottleneck
The failure analysis points to a clear conclusion: models can "do accounting" but cannot "stably complete an accounting reasoning chain." This insight has implications for the entire agent direction—the bottleneck is not tool use but long-horizon reasoning stability.
4. Practical Implications of Simpson's Paradox
"More budget doesn't solve hard tasks" offers direct guidance for agent system design: don't rely on "give more tokens/more rounds" as a fallback. The bottleneck on hard tasks is reasoning ability, not resources.
---
A Cross-Domain Parallel: Comparison with SWE-bench
APEX-Accounting echoes SWE-bench, the benchmark that tests real software engineering work. The two share structural parallels:
---
Limitations
The paper openly acknowledges several limitations:
1. Closed benchmark: the 160 tasks are private and cannot be released. This limits fine-grained community debugging, but the authors provide a leaderboard evaluation that any frontier model can apply to. 2. 10 worlds may lack diversity: 10 accounting systems and 10 business types offer limited coverage. Accounting varies dramatically across industries (manufacturing, retail, finance). 3. No human baseline: the paper does not report human accounting expert performance on these tasks. Without a human reference, 56.4% is hard to interpret as high or low. 4. 9 closed-source models only: no open-source models (such as Llama or Qwen) were tested, so the relationship between model scale and accounting capability cannot be analyzed.
---
Takeaways
What struck me most about this paper is not the 56.4% number but the structural gap it reveals between "passing an exam" and "being competent at the job."
Over the past two years, we've seen too many headlines about "AI passing X professional exam." But those exams feature clean prompts, clear boundaries, and single-step answers—very different from real work. APEX-Accounting tells us with real data: passing exams and doing real work differ by at least an order of magnitude.
Going deeper: the failure analysis points to a conclusion with implications for the entire agent direction—long-horizon reasoning stability is the next core bottleneck. Models can find the right files, call the right tools, and reason correctly on single steps, but stably chaining 10 steps of reasoning together is still beyond them. This is not just an "accounting AI" problem; it's a problem for all agent systems.
Finally, the Simpson's paradox finding is worth remembering: more budget doesn't solve hard tasks. This is a warning for all "throw more compute at it" agent designs. The bottleneck on hard tasks is reasoning ability, not resources. You can't solve a problem the model fundamentally cannot figure out by simply asking it to think harder a few more times.
---
Paper link: https://arxiv.org/abs/2607.27189 HTML version: https://arxiv.org/html/2607.27189v1 Code repository: https://github.com/Mercor-Intelligence/archipelago