ClassEval-Pro: A Cross-Domain Class-Level Benchmark Exposes AI Coding Limits

Forum topic · QianXun · 2026-05-01

Summary

ClassEval-Pro, a benchmark from Shanghai Jiao Tong University and Fudan University (FSE 2026), tests large language models on class-level code generation rather than simple function-level tasks like HumanEval. The benchmark draws post-January-2025 GitHub code across 11 domains to avoid data contamination, requires test coverage above 90%, and adds an LLM-based review panel. Results show even top models such as GPT-5.1 and Gemini-2.5-Pro achieve only around 45.6% Pass@1 on class-level tasks. The most frequent failures are logic coordination errors (56.2%) and dependency invocation errors (38.0%), indicating that the main weakness of current AI coding is not writing individual functions but managing shared state, inter-method calls, and overall engineering coherence.

Overview

While LLMs excel at writing standalone, function-level code (as measured by benchmarks like HumanEval), real software engineering requires generating entire classes—multiple methods that share state, call each other, and stay logically consistent. ClassEval-Pro, a 2026 study by Shanghai Jiao Tong University and Fudan University, introduces a class-level, cross-domain benchmark to test this capability.

Key Benchmark Features

Cross-domain composition: Tasks span 11 different domains (e.g., finance, healthcare, gaming), requiring combined, realistic class implementations.
Contamination-free: Code is sourced from real GitHub repositories published after January 2025, then anonymized and refactored—models cannot pass by memorizing training data.
High difficulty bar: Solutions must pass test suites with over 90% coverage, plus review by a panel of multiple top LLMs.

Results

Top models struggle: Even GPT-5.1 and Gemini-2.5-Pro reach only about 45.6% Pass@1 on class-level generation.
Dominant failure modes:
Logic coordination errors: 56.2% — methods within a class contradict each other or reference wrong state.
Dependency invocation errors: 38.0% — incorrect calls between methods.

Takeaway

ClassEval-Pro redefines the bar for AI coding: writing elegant individual functions is only the surface skill; building robust, internally consistent classes is the real challenge. As code generation evolves from assisted completion toward autonomous development, models need architectural awareness and the ability to handle complex class-level dependencies.

*Source: FSE 2026 paper “ClassEval-Pro”.*

Discussion: What class-level logic does your AI assistant struggle with most? Share your experience in the comments.