> Original repo: harveyai/harvey-labs > Trending stats: +87 stars today | Language: Python | Category: AI legal benchmark
The Blind Spot in AI Benchmarking
AI benchmarks are everywhere. MMLU measures knowledge breadth, SWE-bench measures coding ability, MATH measures mathematical reasoning. But one field has long lacked a solid benchmark: legal work.
The reason is simple. Legal work is not "answering questions" or "writing code" — it is finding issues in piles of real documents, cross-referencing them, and producing structured deliverables according to rules. The complexity lies not in knowledge itself, but in process and environment.
Harvey AI has just open-sourced the Legal Agent Benchmark (LAB), built specifically to measure how well LLM agents handle real legal work.
What LAB Is
LAB consists of two components:
1. Task dataset: agent instructions, relevant documents, and grading rubrics 2. Execution framework: tools to run agents, evaluate results, and generate reports
The example task is an M&A data-room assignment, and its design is strikingly realistic:
- The agent receives a data room (a pile of M&A-related documents)
- The agent must complete specific tasks per instructions (e.g., reviewing contract clauses, identifying risks, generating reports)
- The agent can use tools (retrieval, computation, file operations)
- Evaluation is scored against rubrics — not simple right/wrong
- Multiple model adapters: plug in different LLMs (GPT-4, Claude, Gemini, etc.)
- Tool calls: agents can retrieve documents, manipulate files, and compute
- Report generation: each run produces a detailed report, including per-rubric-item results
- Batch sweeps: run many tasks across many models at once and generate comparison reports
Why Legal Work Is Hard to Evaluate
Legal work differs from programming in three fundamental ways, which is why the SWE-bench approach does not transfer:
First, inputs are documents, not code. Programming tasks come with function signatures and test cases — highly structured. Legal tasks come with dozens of pages of contracts, due-diligence reports, and regulatory filings — low structure. The agent must understand the documents before it can even begin.
Second, outputs are a quality spectrum, not binary. Programming tasks have test suites: pass or fail. Legal outputs are reports and documents whose quality lies on a continuous spectrum. LAB quantifies this with rubrics: each rubric item is pass/fail, but overall quality is a weighted combination of many rubric items.
Third, the process is not linear. Coding tasks are typically "read the problem → write code → run tests." Legal tasks may require repeated retrieval, cross-referencing, and revising earlier judgments. An agent's trajectory is far more complex than in coding.
LAB's Evaluation Method
LAB uses all-pass rubric scoring: a task has N rubric items, and the agent must pass all of them to count as complete. This is stricter than proportional scoring — a 90% pass rate in a legal context might mean 10% of clauses went unreviewed, which is unacceptable.
LAB also uses an LLM judge to evaluate agent outputs. This raises a question: how reliable is the LLM judge itself? LAB's documentation includes a dedicated behavioral analysis of the LLM judge, covering its consistency and biases across different rubric items.
Design of the Execution Harness
LAB's execution harness supports:
What This Means for Legal AI
The open-sourcing of LAB sends several important signals:
First, legal AI has moved from "can it work" to "how well does it work." In 2023 people asked whether AI can do legal work; in 2025 they ask how well it does it. LAB gives "how well" a quantifiable answer.
Second, agent capability matters more than model capability. LAB measures not a single LLM's legal knowledge, but how an LLM agent performs within a legal workflow. Gaps in agent architecture (tool calling, document retrieval, multi-step reasoning) may matter more than gaps in the underlying model.
Third, realistic environments are key. LAB's tasks are based on real legal scenarios (the M&A data room), not artificial law-exam questions. Benchmark results therefore approximate real-world effectiveness rather than academic test scores.
A Question Worth Pondering
LAB's rubrics were designed by Harvey's legal experts. That means benchmark quality depends on the designers' domain expertise.
If another team wants to extend LAB — say, adding litigation or compliance tasks — they need the corresponding legal experts. This differs from SWE-bench, whose test cases come from real GitHub PRs that any programmer can contribute to.
The bar for legal benchmarks is inherently higher. Will this limit the pace of community contributions? Or does the specificity of the legal domain require exactly this high bar to guarantee quality?
Harvey's choice: open-source the framework and task set, but retain expert review. A design that balances openness with quality.
---
*Project: github.com/harveyai/harvey-labs* *Announcement: harvey.ai/blog/introducing-harveys-legal-agent-benchmark*