English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

APEX-Accounting: Frontier Models Still Fail at Real Accounting Work, Best Score Only 56.4%

Forum topic · ✨步子哥 · 2026-07-30

Summary

APEX-Accounting is a benchmark built by Mercor and Ramp to test whether frontier AI models can perform real accounting work rather than pass simplified exams. It contains 10 complete accounting "worlds," each with a full accounting system, related documents (spreadsheets, PDFs, bank statements), and 16 interdependent tasks — 160 tasks total covering reconciliation, accruals, posting, and financial report generation, all expert-annotated with multi-dimensional rubrics. Nine frontier models were evaluated: the best, Claude-Fable-5 (Max), scored only 56.4% Mean Criteria@3, while the highest Pass^8 (fully correct in at least one of 8 attempts) was just 2.6%, and the best Pass@8 was 21.5%. Failure analysis shows models locate correct inputs and call tools well, but break down on multi-step reasoning: wrong baseline logic, lost intermediate results, and missed contradictions. The paper also documents a Simpson's paradox — higher token budgets correlate with better aggregate scores, yet within a fixed budget, tasks consuming more tokens score lower — implying harder tasks cannot be solved by simply increasing budgets. The benchmark suggests long-horizon reasoning stability, not tool use, is the core bottleneck for AI agents in professional work.

Paper: APEX-Accounting Authors: Benchek Julien, Bennett Austin, Kern Jasmin, Stevens Ryan, Sultan Rene, et al. Institutions: Mercor × Ramp arXiv: 2607.27189 Link: https://arxiv.org/abs/2607.27189 Code: https://github.com/Mercor-Intelligence/archipelago

---

A Simple Question: Can AI Actually Do the Books?

Over the past two years we've seen headlines like "AI passes the CPA exam." These scores create the illusion that AI can already do professional work. But there is a huge gap between passing an exam and doing actual work. Exam questions are clean and self-contained. Real accounting work means:

  • Finding relevant data across spreadsheets, PDFs, emails, and bank statements
  • Judging which account a transaction belongs to
  • Multi-step reasoning: reconcile, adjust entries, post, generate statements
  • Any error in one step cascades into everything after it
  • APEX-Accounting, built by Mercor and Ramp, measures exactly this gap: can frontier models do real accounting work?

    ---

    Benchmark Design: 10 "Worlds", 160 Real Tasks

    10 Worlds

    APEX-Accounting is not a pile of isolated questions. It contains 10 complete "accounting worlds," each with:

  • A full accounting system (ledger, chart of accounts, historical transactions)
  • A set of related documents (spreadsheets, PDFs, invoices, bank statements)
  • A series of interrelated tasks
  • The key design point: tasks are coupled. Task 3 may depend on task 1's result; task 7 may require completing task 5 first. Just like real month-end close — a process, not 20 independent questions.

    160 Tasks

    16 tasks per world, 160 total, covering core accounting work:

  • Reconciliation: find discrepancies between bank statements and the ledger
  • Accruals: record expenses incurred but not yet paid
  • Posting: move entries from journals to the general ledger
  • Reports: generate trial balances, income statements, balance sheets
  • Expert-Level Annotation

    Every task was authored, solved, and rubric-scored by accounting and bookkeeping experts. Scoring is not binary but multi-dimensional rubric-based, allowing partial credit and distinguishing "right method, wrong arithmetic" from outright failure.

    ---

    Headline Results: Best 56.4%, No Model Reliably Passes

    Nine frontier models were tested. Three numbers tell the story:

    56.4%

    Claude-Fable-5 (Max) ranks first with 56.4% Mean Criteria@3. Second is Muse-Spark-1.1 (xHigh) at 52.6%. Mean Criteria@3 gives each model 3 attempts and averages rubric scores per attempt — a fairly forgiving metric — yet the best model only reaches 56.4%.

    2.6%

    Pass^8 (at least one fully passing run out of 8 attempts) peaks at just 2.6%, achieved by GPT-5.6-Sol (Max+Pro). Even with 8 chances, almost no task is reliably completed end-to-end.

    Pass@8 peaks at 21.5%

    Muse-Spark-1.1 (xHigh) achieves the best Pass@8 of 21.5% — meaning even the best model fails 4 out of 5 tasks in all 8 attempts.

    Together: frontier models are nowhere near replacing humans on real accounting work. Barely half under a lenient metric, near-total failure under a strict one.

    ---

    Failure Modes: Not Retrieval — Reasoning

    The failure analysis is the most valuable part, with a counterintuitive finding:

    Models Find the Right Inputs

    Top models are quite good at locating the right files and data. They correctly pull the relevant invoices, bank statements, and contracts from piles of PDFs and spreadsheets. Retrieval failure rates are low.

    Models Break on Multi-Step Reasoning

    The real failures happen in reasoning:

  • Substituting wrong baselines or authorization logic: e.g., recognizing revenue by payment date instead of invoice date
  • Lost intermediate results: a number computed at step 5 is forgotten by step 7, and the chain continues with the wrong value
  • Missed contradictions: inconsistent information across documents (invoice amount vs. bank record) goes undetected
  • Tool-Use Errors Are Not the Problem

    A notable finding: tool-calling errors are not a major failure mode. Models call APIs, read files, and run computations correctly. The bottleneck isn't "can it use tools" but "can the reasoning chain be completed stably."

    This contradicts a common intuition — the Agent bottleneck is not tool use, it's the stability of multi-step reasoning.

    ---

    Simpson's Paradox: More Money, Worse Results (Within Budget)

    The paper reports an intriguing Simpson's paradox. When token budgets were raised from $1 to $50:

  • In aggregate: higher budget → higher scores
  • Within a fixed budget: tasks where the model spent more tokens scored *lower*
  • The plausible explanation: hard tasks make the model spend more tokens (retries, more file lookups), but hard tasks are simply harder to get right. Both "more tokens" and "low scores" are effects of task difficulty, not cause and effect.

    Deployment implication: you can't solve hard tasks by throwing more budget at them — the bottleneck is reasoning capability, not resources.

    ---

    Why This Benchmark Matters

    1. It measures real work, not exam questions. Unlike most LLM benchmarks, it tests a profession's full daily workflow — the same design philosophy as SWE-bench. It directly ties "model capability" to "professional replaceability": 56.4% doesn't mean the model fails a test, it means the model can only assist, not own the job. 2. Expert-level rubrics. Multi-dimensional scoring distinguishes partial correctness and "right method, wrong computation" — granular enough to diagnose model bottlenecks. 3. It identifies long-horizon reasoning stability as the real bottleneck. Models aren't incapable of accounting — they can't reliably complete a multi-step accounting reasoning chain. 4. The Simpson's paradox has practical meaning. Agent systems cannot rely on "more tokens / more turns" as a fallback for hard tasks.

    ---

    A Cross-Domain Isomorphism: Comparison with SWE-bench

    APEX-Accounting mirrors SWE-bench structurally:

  • Full workflows, not isolated questions: SWE-bench fixes a real GitHub issue; APEX runs a real month-end close
  • Both show multi-step reasoning stability is the bottleneck: SWE-bench found models locate code but can't fix it correctly; APEX found models locate data but can't push the reasoning through
  • Both show tool use is not the main bottleneck
Together they point to one design principle: the next breakthrough lies not in stronger tool use, but in more stable long-horizon reasoning.

---

Limitations

1. Closed benchmark: the 160 tasks are private, limiting community fine-grained debugging, though a leaderboard eval is open to frontier labs. 2. 10 worlds may lack diversity: accounting varies enormously across industries (manufacturing, retail, finance). 3. No human baseline: without expert performance data, it's hard to judge whether 56.4% is high or low. 4. All 9 models are closed frontier models: no open-source models (Llama, Qwen), so scale-vs-capability analysis isn't possible.

---

My Take

What strikes me most is not the 56.4% figure but the structural gap it reveals: passing exams ≠ being competent at work. Between exam-style "clean prompt, single-step answers" and real work, there is at least an order-of-magnitude difference.

Deeper: the failure analysis points to a conclusion relevant to the entire Agent field — long-horizon reasoning stability is the next core bottleneck. Models find the right files, call the right tools, get individual steps right, but cannot reliably chain 10 steps together. That's not an "accounting AI" problem; it's an all-Agent problem.

Finally, remember the Simpson's paradox finding: more budget can't solve hard tasks. For every Agent design that leans on "more compute as a fallback" — the bottleneck on hard tasks is reasoning, not resources. You can't fix what the model fundamentally can't think through by letting it think a few more times.

---

Paper: https://arxiv.org/abs/2607.27189 HTML: https://arxiv.org/html/2607.27189v1 Code: https://github.com/Mercor-Intelligence/archipelago

Tags

#ai-benchmarks#accounting#apex-accounting#llm-evaluation#agents#multi-step-reasoning#simpsons-paradox#frontier-models

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178503813