Maria Ivanova, Pavel Zadorozhny, Rodion Levichev, Ivan Petrov, Pavel Adamenko, Ivan Lopatin, Alexey Kutalev, Dmitrii Babaev
5 min
Large language models have become essential tools for software engineering, yet their evaluation remains heavily skewed toward Python. While benchmarks like LiveCodeBench (LCB) have improved evaluation rigor by using contamination-aware, continuously updated competitive programming problems, they have historically ignored the multilingual reality of real-world development. This paper introduces Multi-LCB to address this gap, providing a systematic way to evaluate LLM performance across twelve major programming languages, including C++, Java, Rust, and TypeScript.
Multi-LCB leverages the existing LCB infrastructure, which sources problems from platforms like LeetCode, AtCoder, and Codeforces. The authors developed an automated pipeline to convert functional-format tasks (typically found in LeetCode) into a unified STDIN/STDOUT format. This allows for a consistent evaluation harness across all supported languages without requiring language-specific rewriting. The benchmark maintains LCB's core strengths: it is contamination-aware, supports continuous updates, and uses hidden official test cases to ensure functional correctness. The authors evaluated 24 diverse LLMs—ranging from 7B to 685B parameters—using a zero-shot prompting strategy to measure Pass@1 performance.
The evaluation reveals that Python performance is not a reliable proxy for a model's overall coding capability. The researchers identified three critical trends:
Multi-LCB provides a necessary, more rigorous standard for assessing the true coding competence of LLMs. By exposing the limitations of Python-centric evaluation, this benchmark forces developers to account for cross-language generalization, which is essential for building robust, language-agnostic AI coding assistants. The release of the full benchmark suite, including prompts and evaluation configurations, facilitates further research into improving multilingual code generation.
LiveCodeBench (LCB) has recently become a widely adopted benchmark for evaluating large language models (LLMs) on code-generation tasks. By curating competitive programming problems, constantly adding fresh problems to the set, and filtering them by release dates, LCB provides contamination-aware evaluation and offers a holistic view of coding capability. However, LCB remains restricted to Python, leaving open the question of whether LLMs can generalize across the diverse programming languages required in real-world software engineering. We introduce Multi-LCB, a benchmark for evaluating LLMs across twelve programming languages, including Python. Multi-LCB transforms Python tasks from the LCB dataset into equivalent tasks in other languages while preserving LCB's contamination controls and evaluation protocol. Because it is fully compatible with the original LCB format, Multi-LCB will automatically track future LCB updates, enabling systematic assessment of cross-language code generation competence and requiring models to sustain performance well beyond Python. We evaluated 24 LLMs for instruction and reasoning on Multi-LCB, uncovering evidence of Python overfitting, language-specific contamination, and substantial disparities in multilingual performance. Our results establish Multi-LCB as a rigorous new benchmark for multi-programming-language code evaluation, directly addressing LCB's primary limitation and exposing critical gaps in current LLM capabilities.
Sam: That does make sense. But I want to push on something. Is there a risk that the models have simply seen these specific problems before, during training? If the benchmark problems were already floating around the internet, the model might be recalling answers rather than solving them fresh.
Alex: That's a serious concern, and the researchers address it directly. It's called data contamination — the idea that a model's training data might already contain the answers to the test questions. Their solution is to filter problems strictly by release date. If a problem was published after the model finished training, there's no way it could have encountered it. So the benchmark specifically targets those newer problems.
Sam: So they're not just testing the models — they're testing them on problems the AI genuinely couldn't have memorized. That makes the findings about the Python gap considerably more credible.
Alex: Exactly. And that's what gives the study its weight. It's not just showing that models perform differently across languages — it's showing that this gap persists even when you rule out the possibility that the model is simply recalling familiar material.
Sam: The authors must have thought about the limits of their own approach, though. Are there ways the benchmark itself could produce misleading results?
Alex: They're candid about this. The strict input-output format that makes the benchmark fair can also introduce a specific kind of error. A model might have the correct logical solution to a problem, but if it formats its output slightly wrong — adds an extra space, uses the wrong line ending — the automated checker marks it as incorrect. The logic was sound; the presentation wasn't.
Sam: So it's like a student who works out the right answer on a maths exam but loses marks because they didn't show their working in the required format.
Alex: Exactly that. It's a real limitation, and the authors flag it honestly. It means the benchmark might slightly undercount a model's true ability in some cases. That's worth keeping in mind when interpreting the results.
Sam: So what's the practical takeaway here? If I'm a developer choosing an AI coding tool, what should I actually pay attention to?
Alex: Don't rely on a model's Python score as a proxy for general coding ability. The study suggests that's a genuinely unreliable shortcut. If your project uses Go, C#, Rust, or anything other than Python, you should look for evaluations that test the model in those languages specifically. A high Python benchmark score tells you the model is good at Python. It doesn't tell you much beyond that.
Sam: It's a useful reminder that benchmarks are only as good as what they actually measure. Testing one language and assuming the rest follow is a significant leap.
Alex: And that's precisely why work like this matters. Building better evaluation tools is how the field develops a clearer, more honest picture of what these models can and can't do. Thanks for listening to ResearchPod.