APEX-Accounting: A Benchmark for Testing Frontier AI Models on Real Accounting Work

Forum topic · 小凯 · 2026-07-31

Summary

APEX-Accounting is a benchmark built by Mercor in partnership with Ramp to assess whether frontier language models can perform real accounting tasks, such as reconciling accounts, accruing expenses, posting transactions, and producing financial reports. The private evaluation set contains 160 tasks split across 10 simulated worlds, each consisting of an accounting system plus spreadsheets, PDFs, and other supporting files. Every task was authored, solved, and graded using rubrics written by accounting and bookkeeping experts. Across nine frontier models, Claude-Fable-5 (Max) leads with 56.4% Mean Criteria@3, followed by Muse-Spark-1.1 (xHigh) at 52.6%. Reliability remains a major weakness: no model exceeds 2.6% Pass^8, and the best Pass@8 is 21.5%. The authors also experiment with scaling the token budget from $1 to $50, observing a Simpson's paradox: aggregate scores rise with larger budgets, yet within a fixed budget framework, tasks where models spend more tokens score lower. The paper is available as arXiv 2607.27189.

Paper Overview

Field: NLP / AI Evaluation
Authors: Julien Benchek, Austin Bennett, Jasmin Kern, Ryan Stevens, Rene Sultan, Charis Ching, Hayley Popiel, Vaibhav Mittal, Felix Mercier, Brendan Foody, Bertie Vidgen
Published: 2026-07-29
arXiv: 2607.27189

What APEX-Accounting Is

APEX-Accounting is a benchmark built by Mercor in partnership with Ramp to assess whether frontier models can do the real work of accountants. Tasks include:

Reconciling accounts
Accruing expenses
Posting transactions
Producing reports

Benchmark Design

The private eval set comprises 160 tasks, split across 10 worlds.
Each world contains an accounting system, as well as spreadsheets, PDFs, and other files.
Every task was authored and solved by experts in accounting and bookkeeping, who also wrote grading rubrics.

Results

Across nine frontier models, Claude-Fable-5 (Max) leads with 56.4% Mean Criteria@3, ahead of Muse-Spark-1.1 (xHigh) at 52.6%.
No model scores more than 2.6% Pass^8; the highest Pass@8 is 21.5%.

Token Budget Experiments

The authors experiment with increasing the token budget from $1 to $50 and observe an instance of Simpson's paradox: total scores increase with the token budget, but within a given budget-limited framework, tasks where models spend more tokens actually score lower.

Original Abstract (excerpt)

> We introduce APEX-Accounting, a benchmark built by Mercor in partnership with Ramp, to assess whether frontier models can do the real work of accountants... Across nine frontier models, Claude-Fable-5 (Max) leads with 56.4% Mean Criteria@3, ahead of Muse-Spark-1.1 (xHigh) at 52.6%. No model scores more than 2.6% Pass^8 and the highest Pass@8 is 21.5%.

*Auto-collected on 2026-07-31.*