Python Is Not a Proxy: Multi-LCB Exposes LLM Code Capabilities Across 12 Languages
> The GigaCode team extended LiveCodeBench from Python-only to 12 programming languages and tested 24 mainstream LLMs. The result: nearly all models are "Python specialists." GPT-OSS-120B is the only model with relatively balanced cross-language performance, while Qwen3-235B, strongest in Python, is overtaken by GPT-OSS-120B on Go, Rust, and Ruby.
Why the Benchmark Matters
LiveCodeBench (LCB) solved three problems of earlier benchmarks (HumanEval, MBPP): data contamination (date-based filtering of problems released after training cutoffs), benchmark saturation (continuously sourcing new problems from LeetCode, AtCoder, and Codeforces), and narrow scenarios (full problem solving, not just function completion). But LCB had one critical blind spot: it only tested Python.
Multi-LCB answers a long-avoided question: is LLM coding ability "general programming intelligence" or just "Python memorization"?
Methodology: 12 Languages, 24 Models, One Problem Set
The 12 languages were selected via cross-referencing four 2025 rankings (TIOBE, GitHub Octoverse, Stack Overflow, RedMonk): Python, C++, Java, C#, Go, Rust, JavaScript, TypeScript, Ruby, PHP, Kotlin, and Scala.
The core technical innovation is an automated conversion pipeline that transforms LeetCode's function-style problems into STDIN/STDOUT format losslessly:
- Scalar input/output: direct mapping
- 1D arrays: space-separated values
- 2D arrays: first line gives row count, subsequent lines are space-separated
- Missing languages: Swift, Haskell, R, Julia
- Only competitive-programming tasks; no API integration, debugging, or code review
- Forced STDIN/STDOUT conversion may disadvantage function-style-idiomatic languages
- Closed-source models (GPT-4, Claude, Gemini) untested
- Paper: Multi-LCB: Extending LiveCodeBench to Multiple Programming Languages
- Authors: Maria Ivanova, Pavel Zadorozhny, Rodion Levichev, Ivan Petrov, Pavel Adamenko, Ivan Lopatin, Alexey Kutalev, Dmitrii Babaev
- Institutions: GigaCode, Yandex School of Data Analysis
- Paper: https://arxiv.org/abs/2606.20517
- Code: https://github.com/Multi-LCB/Multi-LCB
- Submitted to ICLR 2026
This preserves the original algorithmic difficulty while enabling unified cross-language evaluation.
Key Results
Top models (Feb-May 2025 data, Pass@1 %)
| Model | Python | C++ | Java | Go | Rust | Ruby | Scala | Cross-lang avg | |-------|--------|-----|------|----|------|------|-------|----------------| | GPT-OSS-120B* (Medium) | 71.1 | 72.3 | 70.4 | 69.9 | 70.5 | 70.2 | 54.1 | 67.8 | | Qwen3-235B-A22B-Thk* | 74.0 | 75.8 | 73.9 | 56.7 | 47.7 | 49.4 | 57.6 | 64.0 | | DeepSeek-R1-0528* | 66.3 | 68.0 | 67.8 | 55.0 | 63.1 | 62.4 | 62.3 | 63.1 | | GPT-OSS-20B* (Medium) | 63.6 | 65.7 | 62.7 | 59.9 | 61.9 | 61.7 | 43.1 | 59.8 | | Qwen3-30B-A3B-Thk* | 64.0 | 65.7 | 62.4 | 44.1 | 51.7 | 42.1 | 43.6 | 53.2 |
*\* denotes reasoning models.*
Python is not a proxy
Qwen3-235B leads in Python (74.0%) but falls behind GPT-OSS-120B on Go (56.7% vs 69.9%), Rust (47.7% vs 70.5%), and Ruby (49.4% vs 70.2%). Choosing models by Python-only rankings would produce wrong selections.
Cross-language consistency
GPT-OSS-120B shows a standard deviation of only 5.9% across 12 languages, versus 9.4% for Qwen3-235B.
Language difficulty gradient
| Tier | Languages | Avg Pass@1 | |------|-----------|------------| | Tier 1 | Python | ~48.2% | | Tier 1 | Java, C++ | ~44% | | Tier 2 | C#, Ruby, PHP, Go, Rust, Kotlin, JS, TS | ~33-39% | | Tier 3 | Scala | <29% |
Python's average pass rate is 10-20 percentage points above other languages on the same problem set — evidence of training-data imbalance, not problem difficulty.
Python overfitting in scatter plots
Almost every model's Python score sits above the x=y diagonal against its cross-language average. OpenReasoning-Nemotron-32B and OpenCodeReasoning-Nemotron-1.1-32B score 64%+ in Python but under 30% elsewhere — a gap exceeding 30 percentage points.
Static-typed languages are harder
Statically-typed languages (C++, Java, Rust, Go, C#, Kotlin, Scala, TypeScript) consistently underperform dynamic ones (Python, JavaScript, Ruby, PHP). Possible causes: skewed training data, stricter compile-time constraints, longer reasoning chains, and delayed error feedback.
Data Leakage Varies by Language
Multi-LCB inherits LCB's date-filtering, but monthly trends show score drops crossing the training cutoff differ across languages — suggesting cross-language asymmetric leakage that date filtering alone cannot eliminate.
Reasoning vs Instruction Models
Reasoning models dominate:
| Type | Model | Python | Cross-lang avg | |------|-------|--------|----------------| | Reasoning | GPT-OSS-120B* | 71.1% | 67.8% | | Reasoning | Qwen3-235B-Thk* | 74.0% | 64.0% | | Reasoning | DeepSeek-R1* | 66.3% | 63.1% | | Instruction | Qwen3-235B-Instr | 43.8% | 37.5% | | Instruction | Qwen2.5-Coder-32B | 27.5% | 25.0% |
Model scale
Within families, larger models do better cross-lingually, but scale isn't everything: GPT-OSS-20B* (59.8%) beats Qwen3-30B-A3B-Thk* (53.2%). Data quality and distribution may matter more than parameter count.
Distillation amplifies bias
DeepSeek-R1-Distill models show extreme language divergence: R1-Distill-Qwen-14B* scores 41.8% in Python but only 3.7% in Rust and 3.3% in Scala. Distillation magnifies the teacher's Python bias.
Industry Implications
1. Benchmark bias: All recent coding rankings based on Python-only benchmarks may be systematically skewed. 2. Training data: As the paper states, "We suppose that model performance could be improved by increasing training coverage of non-Python programming languages." 3. Model selection advice: Multilingual/full-stack teams should favor GPT-OSS-120B; Python-centric teams may prefer Qwen3-235B-Thk or DeepSeek-R1; systems/embedded teams should check C++/Rust-specific scores.
Limitations
Takeaway
Multi-LCB's value is doing what should have been done long ago: pulling code LLMs out of the Python greenhouse into a 12-language arena. GPT-OSS-120B's consistency points to an important direction: true coding intelligence should be language-agnostic algorithmic reasoning, not over-memorization of one language's corpus. The industry remains far from that goal.
> "Our results establish Multi-LCB as a rigorous new benchmark for multi-programming-language code evaluation, directly addressing LCB's primary limitation and exposing critical gaps in current LLM capabilities."