ClassEval-Pro: A Cross-Domain Benchmark for Class-Level Code Generation

Forum topic · 小凯 · 2026-05-01

Summary

ClassEval-Pro (arXiv:2504.20837) is a new benchmark targeting class-level code generation, a capability between function-level synthesis and repository-level modification that existing evaluations underserve. It contains 300 class-level tasks across 11 domains, built via an automated three-stage pipeline combining complexity enhancement, cross-domain class composition, and real-world GitHub code contributed after January 2025 to reduce data contamination. Every task is validated by an LLM review panel and must pass test suites with over 90% coverage. Evaluation of five frontier LLMs under five generation strategies shows the best model achieves only 45.6% class-level Pass@1, with a 17.7-point gap between strongest and weakest models. Strategy choice strongly interacts with model capability: bottom-up structured generation can boost weaker models by up to 9.4 points, while compositional generation can collapse to 1.3%. Manual analysis of 500 failures attributes 56.2% to logic errors and 38.0% to dependency errors, identifying cross-method coordination as the core bottleneck.

Paper Overview

Field: NLP Authors: Yeheng Chen, Chaoxiang Xie, Yuling Shi Published: 2025-04-30 arXiv: 2504.20837

Abstract (translated)

LLMs have achieved strong results on both function-level code synthesis and repository-level code modification, yet a capability that falls between these two extremes — compositional code creation, i.e., building a complete, internally structured class from a specification — remains underserved. Current evaluations are either confined to isolated functions or rely on manually curated class-level tasks that are expensive to scale and increasingly susceptible to data contamination.

The authors introduce ClassEval-Pro, a benchmark of 300 class-level tasks spanning 11 domains, constructed through an automated three-stage pipeline that combines:

1. Complexity enhancement 2. Cross-domain class composition 3. Integration of real-world GitHub code contributed after January 2025

Every task is validated by an LLM review panel and must pass a test suite with coverage exceeding 90%.

Key Findings

Five frontier LLMs were evaluated under five generation strategies.
The best model achieves only 45.6% class-level Pass@1.
A 17.7-point gap between the strongest and weakest models confirms the benchmark's discriminative power.
Strategy selection strongly interacts with model capability: structured approaches such as bottom-up generation can improve weaker models by up to 9.4 percentage points, while compositional generation can collapse to as low as 1.3%.
Error analysis of 500 manually annotated failures shows logic errors (56.2%) and dependency errors (38.0%) dominate, with cross-method coordination identified as the core bottleneck.