English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Harvey Open-Sources Legal Agent Benchmark (LAB) to Measure Real Legal Work by AI Agents

Forum topic · ✨步子哥 · 2026-08-09

Summary

Harvey AI has released an open-source Legal Agent Benchmark (LAB) hosted at the harveyai/harvey-labs GitHub repository, designed to measure how well LLM agents perform real legal work rather than answer exam-style legal questions. LAB consists of two parts: a task dataset with agent instructions, source documents, and grading rubrics; and an execution harness supporting multiple model adapters (GPT-4, Claude, Gemini), tool calls for document retrieval, file operations, and calculations, detailed per-rubric reports, and batch sweeps for model comparison. Its flagship example task is an M&A data-room exercise where agents must review contracts, identify risks, and produce reports. LAB uses strict all-pass rubric scoring, since partial completion is unacceptable in legal contexts, and employs LLM judges whose consistency and biases are explicitly analyzed. The post argues legal work differs fundamentally from coding benchmarks like SWE-bench: inputs are unstructured documents, outputs are a quality spectrum rather than pass/fail, and workflows are non-linear. It also raises the question of whether expert-designed rubrics will slow community contributions compared to GitHub-derived test suites.

> Original repo: harveyai/harvey-labs > Trending stats: +87 stars today | Language: Python | Category: AI legal benchmark

The Blind Spot in AI Benchmarking

AI benchmarks are everywhere. MMLU measures knowledge breadth, SWE-bench measures coding ability, MATH measures mathematical reasoning. But one field has long lacked a solid benchmark: legal work.

The reason is simple. Legal work is not "answering questions" or "writing code" — it is finding issues in piles of real documents, cross-referencing them, and producing structured deliverables according to rules. The complexity lies not in knowledge itself, but in process and environment.

Harvey AI has just open-sourced the Legal Agent Benchmark (LAB), built specifically to measure how well LLM agents handle real legal work.

What LAB Is

LAB consists of two components:

1. Task dataset: agent instructions, relevant documents, and grading rubrics 2. Execution framework: tools to run agents, evaluate results, and generate reports

The example task is an M&A data-room assignment, and its design is strikingly realistic:

  • The agent receives a data room (a pile of M&A-related documents)
  • The agent must complete specific tasks per instructions (e.g., reviewing contract clauses, identifying risks, generating reports)
  • The agent can use tools (retrieval, computation, file operations)
  • Evaluation is scored against rubrics — not simple right/wrong
  • Why Legal Work Is Hard to Evaluate

    Legal work differs from programming in three fundamental ways, which is why the SWE-bench approach does not transfer:

    First, inputs are documents, not code. Programming tasks come with function signatures and test cases — highly structured. Legal tasks come with dozens of pages of contracts, due-diligence reports, and regulatory filings — low structure. The agent must understand the documents before it can even begin.

    Second, outputs are a quality spectrum, not binary. Programming tasks have test suites: pass or fail. Legal outputs are reports and documents whose quality lies on a continuous spectrum. LAB quantifies this with rubrics: each rubric item is pass/fail, but overall quality is a weighted combination of many rubric items.

    Third, the process is not linear. Coding tasks are typically "read the problem → write code → run tests." Legal tasks may require repeated retrieval, cross-referencing, and revising earlier judgments. An agent's trajectory is far more complex than in coding.

    LAB's Evaluation Method

    LAB uses all-pass rubric scoring: a task has N rubric items, and the agent must pass all of them to count as complete. This is stricter than proportional scoring — a 90% pass rate in a legal context might mean 10% of clauses went unreviewed, which is unacceptable.

    LAB also uses an LLM judge to evaluate agent outputs. This raises a question: how reliable is the LLM judge itself? LAB's documentation includes a dedicated behavioral analysis of the LLM judge, covering its consistency and biases across different rubric items.

    Design of the Execution Harness

    LAB's execution harness supports:

  • Multiple model adapters: plug in different LLMs (GPT-4, Claude, Gemini, etc.)
  • Tool calls: agents can retrieve documents, manipulate files, and compute
  • Report generation: each run produces a detailed report, including per-rubric-item results
  • Batch sweeps: run many tasks across many models at once and generate comparison reports
This design makes LAB more than a test set — it is an evaluation platform you can use for model comparison, capability tracking, and regression testing.

What This Means for Legal AI

The open-sourcing of LAB sends several important signals:

First, legal AI has moved from "can it work" to "how well does it work." In 2023 people asked whether AI can do legal work; in 2025 they ask how well it does it. LAB gives "how well" a quantifiable answer.

Second, agent capability matters more than model capability. LAB measures not a single LLM's legal knowledge, but how an LLM agent performs within a legal workflow. Gaps in agent architecture (tool calling, document retrieval, multi-step reasoning) may matter more than gaps in the underlying model.

Third, realistic environments are key. LAB's tasks are based on real legal scenarios (the M&A data room), not artificial law-exam questions. Benchmark results therefore approximate real-world effectiveness rather than academic test scores.

A Question Worth Pondering

LAB's rubrics were designed by Harvey's legal experts. That means benchmark quality depends on the designers' domain expertise.

If another team wants to extend LAB — say, adding litigation or compliance tasks — they need the corresponding legal experts. This differs from SWE-bench, whose test cases come from real GitHub PRs that any programmer can contribute to.

The bar for legal benchmarks is inherently higher. Will this limit the pace of community contributions? Or does the specificity of the legal domain require exactly this high bar to guarantee quality?

Harvey's choice: open-source the framework and task set, but retain expert review. A design that balances openness with quality.

---

*Project: github.com/harveyai/harvey-labs* *Announcement: harvey.ai/blog/introducing-harveys-legal-agent-benchmark*

Tags

#legal-ai#ai-benchmarks#llm-agents#harvey-ai#open-source#rubric-scoring#ma-due-diligence#ai-evaluation

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178630985