What is LAB?
Harvey AI has open-sourced the Legal Agent Benchmark (LAB) to measure LLM agent performance on real legal work, not on legal Q&A. The current task set is an M&A (mergers and acquisitions) data-room scenario: the agent is given a data room full of documents, an instruction set, and access to tools (retrieval, calculation, file operations), and must produce structured outputs that are graded against an expert-authored rubric.
LAB has two parts:
1. Task dataset — agent instructions, related documents, and scoring rubrics. 2. Execution harness — runs the agent, evaluates results, and generates reports.
Repository: github.com/harveyai/harvey-labs Announcement: harvey.ai/blog/introducing-harveys-legal-agent-benchmark
Why Legal Work Is Hard to Benchmark
Legal work differs from coding work in three fundamental ways that make approaches like SWE-bench insufficient:
- Inputs are documents, not code. Tasks involve dozens of pages of contracts, due-diligence reports, and regulatory filings. The agent must first understand unstructured text before acting.
- Outputs are a quality spectrum, not pass/fail. A legal deliverable is graded on a continuous spectrum; LAB quantifies this with rubric items, but each item is still pass/fail and the overall score is a weighted aggregate.
- Processes are non-linear. Agents must repeatedly search documents, cross-reference clauses, and revise earlier judgments, producing trajectories far more complex than a typical code task.
- Multi-model adapters — GPT-4, Claude, Gemini, and others can be plugged in.
- Tool calling — document retrieval, file operations, computation.
- Report generation — per-run reports showing each rubric item's pass/fail status.
- Batch sweeps — run multiple tasks across multiple models for comparison.
- Legal AI has shifted from "can it work" to "how well does it work." LAB provides a quantifiable answer to the quality question.
- Agent capability matters more than raw model capability. LAB measures an agent's performance inside a legal workflow, where gaps in tool use, retrieval, and multi-step reasoning may exceed gaps in the underlying model.
- Realistic environments matter. Tasks are built from real M&A data-room scenarios rather than artificial legal exam questions, so results should track real-world utility more closely.
How LAB Scores Work
LAB uses all-pass rubric scoring: a task has N rubric items and the agent must pass all of them to be marked complete. This is stricter than proportional grading because in a legal context a 90% pass rate may mean 10% of clauses went unreviewed, which is unacceptable.
LAB also uses an LLM judge to evaluate outputs. The release includes explicit analysis of judge behavior across rubric items, including consistency and bias, because LLM judge quality directly affects benchmark validity.
Harness Capabilities
The execution harness supports:
This makes LAB an evaluation platform, not just a test set: it can be used for model comparison, capability tracking, and regression testing.
Key Takeaways
An Open Question
LAB's rubrics are authored by Harvey's legal experts. That means benchmark quality depends on the designer's domain expertise. Unlike SWE-bench, where test cases come from real GitHub PRs and any developer can contribute, extending LAB to litigation or compliance requires domain experts. Harvey's design choice is to open-source the framework and task set while retaining expert review — a balance between openness and quality, and a structural feature of legal benchmarks more generally.