Overview
Field: NLP Authors: Yeheng Chen, Chaoxiang Xie, Yuling Shi Published: 2025-04-30 arXiv: 2504.20837
Introduction
LLMs have achieved strong results on both function-level code synthesis and repository-level code modification, yet a capability that falls between these two extremes — compositional code creation, i.e., building a complete, internally structured class from a specification — remains underserved. Current evaluations are either confined to isolated functions or rely on manually curated class-level tasks that are expensive to scale and increasingly susceptible to data contamination.
ClassEval-Pro Benchmark
ClassEval-Pro is a benchmark of 300 class-level tasks spanning 11 domains, constructed through an automated three-stage pipeline:
1. Complexity enhancement 2. Cross-domain class composition 3. Integration of real-world GitHub code contributed after January 2025
Every task is validated by an LLM review panel and must pass test suites with over 90% coverage.
Key findings
- Low ceiling: The best-performing model reaches only 45.6% class-level Pass@1.
- Discriminative power: A 17.7-point gap separates the strongest and weakest models evaluated.
- Strategy-model interaction: Generation strategy choice strongly interacts with model capability:
- Structured approaches such as bottom-up generation improve weaker models by up to 9.4 percentage points.
- Compositional generation can collapse to as low as 1.3%.
- Error analysis: Among 500 manually annotated failures, logic errors (56.2%) and dependency errors (38.0%) dominate. Cross-method coordination is identified as the core bottleneck for class-level code generation.
- arXiv paper: <https://arxiv.org/abs/2504.20837>