Multi-LCB: Exposing the Python Bias in LLM Code Evaluation
> GigaCode extended LiveCodeBench from Python to 12 programming languages and tested 24 mainstream LLMs. The results are striking yet unsurprising: nearly all models are "Python specialists" that excel at Python but collapse on other languages. GPT-OSS-120B emerged as the only model with relatively balanced cross-language performance, while Qwen3-235B, despite the highest Python score, was overtaken by GPT-OSS-120B on Go, Rust, and Ruby. This paper's value lies not in new discoveries but in holding up a mirror to the entire industry.
---
1. The Core Issue: What Are Code LLMs Really Being Tested On?
LiveCodeBench (LCB) became the most popular code benchmark since 2024, solving three major problems of its predecessors (HumanEval, MBPP):
1. Data contamination: Filters by release date, testing only problems published after model training cutoffs 2. Question bank saturation: Continuously scrapes new problems from LeetCode, AtCoder, Codeforces 3. Single-scenario testing: Evaluates full problem-solving, not just function completion
However, LCB has a fatal blind spot: it only tests Python.
This is like a driving school that only tests right-angle turns and claims it can judge whether someone can drive. Real-world engineering requires constant language switching—C++ for systems, Java for enterprise, JavaScript for front-end, Go for microservices, Rust for high-performance code. A "full-stack engineer" who only knows Python struggles in production environments.
Multi-LCB answers a long-avoided question: Is LLM code capability "general programming intelligence" or "Python memorization"?
---
2. 12 Languages, 24 Models, One Question Bank
2.1 Language Selection
The 12 languages were chosen via cross-validation of four 2025 rankings (TIOBE, GitHub Octoverse, Stack Overflow, RedMonk):
| Language | Type System | Memory | Main Use Cases | |----------|-------------|--------|----------------| | Python | Dynamic | GC | Data/AI | | C++ | Static | Manual/RAII | Systems/Performance | | Java | Static | GC | Enterprise/JVM | | C# | Static | GC | Enterprise/.NET | | Go | Static | GC | Cloud-native/Microservices | | Rust | Static | Ownership | Systems/Security | | JavaScript | Dynamic | GC | Web/Front-end | | TypeScript | Static | GC | Web/Full-stack | | Ruby | Dynamic | GC | Web/Scripting | | PHP | Dynamic | GC | Web/Back-end | | Kotlin | Static | GC | Android/JVM | | Scala | Static | GC | Big Data/JVM |
2.2 Technical Core: Unifying Functional Evaluation to STDIN/STDOUT
LCB has two problem formats:
- AtCoder/Codeforces: Native STDIN/STDOUT
- LeetCode: Functional (function signature + return value)
- Scalar I/O: Direct mapping
- 1D arrays: Space-separated values
- 2D arrays: First line specifies row count, subsequent lines space-separated
- Python: 64%+
- Other languages average: below 30%
- Gap: over 30 percentage points
- Models score significantly higher on problems from before training cutoffs
- Scores drop in stepwise fashion at cutoff boundaries
- Drop magnitude differs across languages
- Within series, larger size yields better cross-language performance
- But size isn't everything: GPT-OSS-20B* (20B) achieves 59.8% cross-language mean, exceeding Qwen3-30B-A3B-Thk* (53.2%)
- Training data quality and distribution may matter more than parameter count
- DeepSeek-R1-Distill-Qwen-32B*: Python 45.9%, Rust 21.2%, Scala 8.7%
- DeepSeek-R1-Distill-Qwen-14B*: Python 41.8%, Rust 3.7%, Scala 3.3%
- Top-ranked models may simply be better at Python
- Apparent "code capability" gains may reflect Python corpus expansion
- Scenarios requiring multilingual capability (enterprise dev, systems programming, full-stack) have been severely underestimated
- Full-stack/multilingual teams: Prioritize GPT-OSS-120B (most balanced)
- Python data science teams: Qwen3-235B-Thk or DeepSeek-R1 offer better cost-effectiveness
- Systems/embedded programming: Focus on C++ and Rust specific scores, not aggregate totals
- Java/Kotlin Android development: GPT-OSS series performs stably on JVM languages
- Expand languages (Swift, Haskell, R, Julia)
- Evaluate closed-source models
- Support extended benchmarks like LCB-Pro
- Paper: Multi-LCB: Extending LiveCodeBench to Multiple Programming Languages
- Authors: Maria Ivanova, Pavel Zadorozhny, Rodion Levichev, Ivan Petrov, Pavel Adamenko, Ivan Lopatin, Alexey Kutalev, Dmitrii Babaev
- Institution: GigaCode, Yandex School of Data Analysis
- Paper: https://arxiv.org/abs/2606.20517
- Code: https://github.com/Multi-LCB/Multi-LCB
- Benchmark: 12 languages (Python, C++, Java, Go, JS, TS, C#, Rust, Ruby, PHP, Kotlin, Scala)
- Models tested: 24 (7B-685B, reasoning + instruction models)
- Key findings: Python overfitting, static-typed languages harder, GPT-OSS-120B most balanced, language-asymmetric data contamination
- Submission: ICLR 2026
Multi-LCB's key innovation is an automated conversion pipeline that transforms LeetCode functional problems into STDIN/STDOUT format without writing language-specific test templates.
Conversion rules:
This preserves algorithmic difficulty while enabling unified cross-language evaluation.
---
3. Experimental Results: Python Hegemony and Language Gap
3.1 Overall Performance (Top 5 Models, Feb-May 2025 Data)
| Model | Python | C++ | Java | Go | Rust | Ruby | Scala | Cross-lang Mean | |-------|--------|-----|------|----|------|------|-------|-----------------| | GPT-OSS-120B* (Medium) | 71.1 | 72.3 | 70.4 | 69.9 | 70.5 | 70.2 | 54.1 | 67.8 | | Qwen3-235B-A22B-Thk* | 74.0 | 75.8 | 73.9 | 56.7 | 47.7 | 49.4 | 57.6 | 64.0 | | DeepSeek-R1-0528* | 66.3 | 68.0 | 67.8 | 55.0 | 63.1 | 62.4 | 62.3 | 63.1 | | GPT-OSS-20B* (Medium) | 63.6 | 65.7 | 62.7 | 59.9 | 61.9 | 61.7 | 43.1 | 59.8 | | Qwen3-30B-A3B-Thk* | 64.0 | 65.7 | 62.4 | 44.1 | 51.7 | 42.1 | 43.6 | 53.2 |
Two key findings:
1. Python is not a proxy: Qwen3-235B leads Python at 74.0%, but GPT-OSS-120B surpasses it on Go (56.7 vs 69.9), Rust (47.7 vs 70.5), and Ruby (49.4 vs 70.2). Selecting models based on Python rankings alone leads to wrong choices.
2. GPT-OSS-120B's cross-language consistency is remarkable: Its standard deviation across 12 languages is only 5.9%, versus 9.4% for Qwen3-235B. OpenAI clearly balanced multilingual code distribution during training.
3.2 Language Difficulty Gradient
The paper uses box plots to show Pass@1 distribution across all 24 models:
| Tier | Languages | Avg Pass@1 | |------|-----------|------------| | Tier 1 | Python | ~48.2% | | Tier 1 | Java | ~44% | | Tier 1 | C++ | ~44% | | Tier 2 | C#, Ruby, PHP | ~33-39% | | Tier 2 | Go, Rust, Kotlin | ~33-39% | | Tier 2 | JavaScript, TypeScript | ~33-39% | | Tier 3 | Scala | <29% |
Python's average pass rate exceeds other languages by 10-20 percentage points. Since all languages use the same question bank, this gap reflects training-data distribution rather than problem difficulty.
3.3 Python Overfitting: Scatter Plots Don't Lie
Figure 3 plots each model's Python Pass@1 against cross-language average Pass@1. Nearly all points fall above the x=y diagonal, meaning Python performance systematically overestimates true cross-language capability.
Most extreme cases: OpenReasoning-Nemotron-32B and OpenCodeReasoning-Nemotron-1.1-32B
These models' "code capability" is essentially Python pattern recognition, not general programming reasoning.
3.4 Static-Typed Languages Are Harder
Data clearly shows static-typed languages (C++, Java, Rust, Go, C#, Kotlin, Scala, TypeScript) underperform dynamic-typed languages (Python, JavaScript, Ruby, PHP).
Possible causes: 1. Training data skew: Python dominates code corpora 2. Type system complexity: Stricter syntax and compilation constraints 3. Longer reasoning chains: Models must consider both algorithm logic and type correctness 4. Delayed error feedback: Compilation errors are harder to debug than runtime errors
---
4. Data Contamination: Not Just Existence, But Language Asymmetry
Multi-LCB inherits LCB's date-filtering mechanism but reveals a more subtle issue: contamination severity varies by language.
Figure 5's monthly trend charts show:
Implications: 1. Some languages have more up-to-date, comprehensive training data 2. Some languages may leak via indirect channels (StackOverflow answers, technical blogs) 3. Current date filtering is insufficient to eliminate asymmetric cross-language leakage
---
5. Who Are the True Multilingual Performers?
5.1 Reasoning vs Instruction Models
Data is unambiguous: **Reasoning models (marked with *) comprehensively outperform instruction models**.
| Type | Representative | Python | Cross-lang Mean | |------|----------------|--------|-----------------| | Reasoning | GPT-OSS-120B* | 71.1% | 67.8% | | Reasoning | Qwen3-235B-Thk* | 74.0% | 64.0% | | Reasoning | DeepSeek-R1* | 66.3% | 63.1% | | Instruction | Qwen3-235B-Instr | 43.8% | 37.5% | | Instruction | Qwen2.5-Coder-32B | 27.5% | 25.0% |
Reasoning models' cross-language advantage is not marginal but qualitative: they appear to actually reason about how to solve algorithmic problems rather than matching training patterns.
5.2 Model Size Effects
Models range from 7B (OlympicCoder-7B) to 685B (Qwen3-235B).
Trends:
5.3 Special Case: DeepSeek-R1-Distill Series
R1-distilled small models exhibit extreme language capability divergence:
Distillation amplifies the teacher model's Python bias—student models are more narrowly specialized than teachers.
---
6. Industry Implications
6.1 Benchmark Reflection
If Multi-LCB's results generalize, all "code capability rankings" from the past two years based on HumanEval, MBPP, and Python-only LCB may carry systematic bias:
6.2 Training Data Insights
The paper's conclusion is clear:
> "We suppose that model performance could be improved by increasing training coverage of non-Python programming languages."
Current code LLM training data is heavily Python-skewed. Cultivating genuine "multilingual programming intelligence" requires balanced per-language distribution from the training stage.
6.3 Practical Selection Advice
When choosing code assistants, don't rely solely on Python benchmarks:
---
7. Limitations and Future Work
The paper honestly lists several limitations:
1. Incomplete language coverage: Missing Swift, Haskell, R, Julia 2. Limited task types: Only algorithmic competition problems; no API integration, debugging, code review 3. STDIN/STDOUT format may introduce extra difficulty: Some languages favor functional programming; forced conversion may be unfair 4. Only open-source models tested: GPT-4, Claude, Gemini performance remains unknown
Future plans:
---
8. One-Sentence Summary
Multi-LCB's value lies not in breakthrough technical discoveries, but in doing what should have been done long ago: pulling code LLMs from Python's greenhouse into a 12-language arena for genuine competition. The result: most models caught a cold.
GPT-OSS-120B's cross-language consistency hints at an important direction: true code intelligence should be language-agnostic algorithmic reasoning, not over-memorization of a single language's corpus. The paper's data proves the industry is still far from this goal.
> "Our results establish Multi-LCB as a rigorous new benchmark for multi-programming-language code evaluation, directly addressing LCB's primary limitation and exposing critical gaps in current LLM capabilities."
---