If you were hiring a programmer, you would not test them solely on how to write bubble sort or reverse a string. You would want to see whether they can build a complete, internally consistent system—say, an order management class.
Yet AI research has long relied on function-level benchmarks like HumanEval to measure the coding ability of large language models. This has made many models look like perfect test-takers while falling apart in real projects. ClassEval-Pro, a 2026 study from Shanghai Jiao Tong University and Fudan University, finally gives LLMs a true "engineering exam."
Why Function-Level Aces Struggle With Engineering
LLMs are already fluent at writing small, standalone functions. But class-level code generation is a different challenge entirely.
Writing a class means the model must produce multiple methods that share state, call each other, and remain logically aligned. It is like moving from "manufacturing parts" to "assembling a precision clock." Many models forget what a private variable was called when writing the second method, or produce contradictory logic.
What Makes ClassEval-Pro Different
This "class-level cross-domain benchmark" has three notable features:
- Cross-domain composition: Tasks are no longer single-domain. Models must perform compositional creation across 11 different domains, such as finance, healthcare, and gaming.
- Contamination-resistant: Problems were built by sanitizing and refactoring real GitHub code released after January 2025, so models cannot pass by memorizing training data.
- Rigorous difficulty: Solutions must pass test suites with over 90% coverage, plus review by a "jury" of multiple top-tier LLMs.
- Failing grades across the board: Even top performers like GPT-5.1 and Gemini-2.5-Pro reach only about 45.6% class-level Pass@1 (writing correct code in one shot).
- Fatal weaknesses: The two most common errors are logic coordination errors (56.2%) and dependency invocation errors (38.0%)—proof that AI's biggest shortcoming is not writing code but understanding engineering.
Results: AI's Engineering Ceiling
The results are sobering:
Commentary
ClassEval-Pro redefines the "passing line" for AI coding. Writing an elegant function is a technique; building robust software is the real discipline. As code generation evolves from "assisted search" toward "autonomous development," we need AI with global vision and the ability to handle complex class dependencies—architect-level AI, not just models good at homework exercises.
*This article is based on the FSE 2026 paper "ClassEval-Pro."*