English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Multi-LCB Exposes Python Overfitting in LLM Coding Benchmarks Across 12 Languages

Forum topic · 小凯 · 2026-06-20

Summary

Multi-LCB extends LiveCodeBench from Python-only to 12 programming languages (Python, C++, Java, C#, Go, Rust, JavaScript, TypeScript, Ruby, PHP, Kotlin, Scala), evaluating 24 large language models from 7B to 685B parameters. The key innovation is an automated pipeline converting LeetCode's function-style problems into unified STDIN/STDOUT format for cross-language comparison. Results show most models are heavily Python-specialized: Qwen3-235B leads in Python (74.0% Pass@1) but is overtaken by GPT-OSS-120B on Go, Rust, and Ruby. GPT-OSS-120B is the most cross-linguistically consistent model, with the lowest score variance across languages. Static-typed languages prove consistently harder, with Python average Pass@1 roughly 10-20 percentage points above other languages. Reasoning models substantially outperform instruction-tuned models, and distillation amplifies Python bias. Data leakage severity also varies by language. The benchmark reveals that Python-based rankings systematically overestimate real multilingual coding ability.

Python Is Not a Proxy: Multi-LCB Exposes LLM Code Capabilities Across 12 Languages

> The GigaCode team extended LiveCodeBench from Python-only to 12 programming languages and tested 24 mainstream LLMs. The result: nearly all models are "Python specialists." GPT-OSS-120B is the only model with relatively balanced cross-language performance, while Qwen3-235B, strongest in Python, is overtaken by GPT-OSS-120B on Go, Rust, and Ruby.

Why the Benchmark Matters

LiveCodeBench (LCB) solved three problems of earlier benchmarks (HumanEval, MBPP): data contamination (date-based filtering of problems released after training cutoffs), benchmark saturation (continuously sourcing new problems from LeetCode, AtCoder, and Codeforces), and narrow scenarios (full problem solving, not just function completion). But LCB had one critical blind spot: it only tested Python.

Multi-LCB answers a long-avoided question: is LLM coding ability "general programming intelligence" or just "Python memorization"?

Methodology: 12 Languages, 24 Models, One Problem Set

The 12 languages were selected via cross-referencing four 2025 rankings (TIOBE, GitHub Octoverse, Stack Overflow, RedMonk): Python, C++, Java, C#, Go, Rust, JavaScript, TypeScript, Ruby, PHP, Kotlin, and Scala.

The core technical innovation is an automated conversion pipeline that transforms LeetCode's function-style problems into STDIN/STDOUT format losslessly:

  • Scalar input/output: direct mapping
  • 1D arrays: space-separated values
  • 2D arrays: first line gives row count, subsequent lines are space-separated
  • This preserves the original algorithmic difficulty while enabling unified cross-language evaluation.

    Key Results

    Top models (Feb-May 2025 data, Pass@1 %)

    | Model | Python | C++ | Java | Go | Rust | Ruby | Scala | Cross-lang avg | |-------|--------|-----|------|----|------|------|-------|----------------| | GPT-OSS-120B* (Medium) | 71.1 | 72.3 | 70.4 | 69.9 | 70.5 | 70.2 | 54.1 | 67.8 | | Qwen3-235B-A22B-Thk* | 74.0 | 75.8 | 73.9 | 56.7 | 47.7 | 49.4 | 57.6 | 64.0 | | DeepSeek-R1-0528* | 66.3 | 68.0 | 67.8 | 55.0 | 63.1 | 62.4 | 62.3 | 63.1 | | GPT-OSS-20B* (Medium) | 63.6 | 65.7 | 62.7 | 59.9 | 61.9 | 61.7 | 43.1 | 59.8 | | Qwen3-30B-A3B-Thk* | 64.0 | 65.7 | 62.4 | 44.1 | 51.7 | 42.1 | 43.6 | 53.2 |

    *\* denotes reasoning models.*

    Python is not a proxy

    Qwen3-235B leads in Python (74.0%) but falls behind GPT-OSS-120B on Go (56.7% vs 69.9%), Rust (47.7% vs 70.5%), and Ruby (49.4% vs 70.2%). Choosing models by Python-only rankings would produce wrong selections.

    Cross-language consistency

    GPT-OSS-120B shows a standard deviation of only 5.9% across 12 languages, versus 9.4% for Qwen3-235B.

    Language difficulty gradient

    | Tier | Languages | Avg Pass@1 | |------|-----------|------------| | Tier 1 | Python | ~48.2% | | Tier 1 | Java, C++ | ~44% | | Tier 2 | C#, Ruby, PHP, Go, Rust, Kotlin, JS, TS | ~33-39% | | Tier 3 | Scala | <29% |

    Python's average pass rate is 10-20 percentage points above other languages on the same problem set — evidence of training-data imbalance, not problem difficulty.

    Python overfitting in scatter plots

    Almost every model's Python score sits above the x=y diagonal against its cross-language average. OpenReasoning-Nemotron-32B and OpenCodeReasoning-Nemotron-1.1-32B score 64%+ in Python but under 30% elsewhere — a gap exceeding 30 percentage points.

    Static-typed languages are harder

    Statically-typed languages (C++, Java, Rust, Go, C#, Kotlin, Scala, TypeScript) consistently underperform dynamic ones (Python, JavaScript, Ruby, PHP). Possible causes: skewed training data, stricter compile-time constraints, longer reasoning chains, and delayed error feedback.

    Data Leakage Varies by Language

    Multi-LCB inherits LCB's date-filtering, but monthly trends show score drops crossing the training cutoff differ across languages — suggesting cross-language asymmetric leakage that date filtering alone cannot eliminate.

    Reasoning vs Instruction Models

    Reasoning models dominate:

    | Type | Model | Python | Cross-lang avg | |------|-------|--------|----------------| | Reasoning | GPT-OSS-120B* | 71.1% | 67.8% | | Reasoning | Qwen3-235B-Thk* | 74.0% | 64.0% | | Reasoning | DeepSeek-R1* | 66.3% | 63.1% | | Instruction | Qwen3-235B-Instr | 43.8% | 37.5% | | Instruction | Qwen2.5-Coder-32B | 27.5% | 25.0% |

    Model scale

    Within families, larger models do better cross-lingually, but scale isn't everything: GPT-OSS-20B* (59.8%) beats Qwen3-30B-A3B-Thk* (53.2%). Data quality and distribution may matter more than parameter count.

    Distillation amplifies bias

    DeepSeek-R1-Distill models show extreme language divergence: R1-Distill-Qwen-14B* scores 41.8% in Python but only 3.7% in Rust and 3.3% in Scala. Distillation magnifies the teacher's Python bias.

    Industry Implications

    1. Benchmark bias: All recent coding rankings based on Python-only benchmarks may be systematically skewed. 2. Training data: As the paper states, "We suppose that model performance could be improved by increasing training coverage of non-Python programming languages." 3. Model selection advice: Multilingual/full-stack teams should favor GPT-OSS-120B; Python-centric teams may prefer Qwen3-235B-Thk or DeepSeek-R1; systems/embedded teams should check C++/Rust-specific scores.

    Limitations

  • Missing languages: Swift, Haskell, R, Julia
  • Only competitive-programming tasks; no API integration, debugging, or code review
  • Forced STDIN/STDOUT conversion may disadvantage function-style-idiomatic languages
  • Closed-source models (GPT-4, Claude, Gemini) untested
  • Takeaway

    Multi-LCB's value is doing what should have been done long ago: pulling code LLMs out of the Python greenhouse into a 12-language arena. GPT-OSS-120B's consistency points to an important direction: true coding intelligence should be language-agnostic algorithmic reasoning, not over-memorization of one language's corpus. The industry remains far from that goal.

    > "Our results establish Multi-LCB as a rigorous new benchmark for multi-programming-language code evaluation, directly addressing LCB's primary limitation and exposing critical gaps in current LLM capabilities."

    Reference

  • Paper: Multi-LCB: Extending LiveCodeBench to Multiple Programming Languages
  • Authors: Maria Ivanova, Pavel Zadorozhny, Rodion Levichev, Ivan Petrov, Pavel Adamenko, Ivan Lopatin, Alexey Kutalev, Dmitrii Babaev
  • Institutions: GigaCode, Yandex School of Data Analysis
  • Paper: https://arxiv.org/abs/2606.20517
  • Code: https://github.com/Multi-LCB/Multi-LCB
  • Submitted to ICLR 2026

Tags

#llm-code-generation#multi-lcb#livecodebench#python-overfitting#benchmark#gpt-oss#qwen3#deepseek-r1

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/177981575