Alessandro Giagnorio, Alberto Martin-Lopez, Gabriele Bavota
4 min
This paper investigates the performance gap of Large Language Models (LLMs) when applied to no-resource programming languages—languages for which virtually no training data exists, such as proprietary or newly released domain-specific languages. The authors seek to answer two primary questions: how much does language popularity affect code generation performance, and what are the most effective, cost-efficient strategies for adapting LLMs to these languages?
To address these questions, the authors created three new code generation benchmarks (HumanEval, MBPP, and a custom McEval-Hard set) translated into Gleam and MoonBit. They evaluated six state-of-the-art LLMs, ranging from commercial models like GPT-4o to open-source models like Qwen 2.5 Coder, across high-, low-, and no-resource language categories.
The study confirms a severe performance disparity: while top-tier models achieve high pass@1 rates on high-resource languages (59-89%), their performance on no-resource languages is near zero (0-1%). The authors tested several adaptation techniques, including few-shot prompting, Retrieval-Augmented Generation (RAG), and further pre-training. While further pre-training on base models yielded the best performance gains, it typically destroys the instruction-following capabilities of instruct-tuned models.
To solve this, the authors implemented a weight-diff transfer approach. They performed further pre-training on a base model to teach it the target language, then injected instruction-following capabilities by calculating the weight difference between the original base model and its corresponding instruct-tuned version. This method successfully combined language specialization with instruction-following, achieving pass@1 scores exceeding 25% on the challenging McEval-Hard benchmark.
As organizations increasingly develop proprietary or domain-specific languages, they face a "no-resource" problem where standard AI coding assistants fail. This research provides a practical, computationally efficient roadmap for companies to deploy specialized, high-performing code recommenders without the prohibitive costs of full-scale instruction fine-tuning or the need for massive, pre-existing training corpora.
Large Language Models (LLMs) have significantly advanced the automation of software engineering tasks. One prominent example is code generation, where an LLM produces code in a specified programming language based on a natural language description. Most research in this area has focused on high-resource languages, such as Python or Java, which benefit from abundant training data. A smaller body of work has explored low-resource languages, which are underrepresented in training corpora. In contrast, no-resource languages for which LLMs have seen virtually no training data remain largely unstudied. These languages often emerge in industry, where organizations develop proprietary or domain-specific languages unsupported by commercial tools like GitHub Copilot. This results in the need for companies to deploy their own in-house code recommenders. To investigate possible solutions in this context, we build and release three code generation benchmarks for no-resource languages, based on two recently proposed programming languages for which very little training data is available. Using these benchmarks, we experiment several solutions to teach LLMs about no-resource languages, including prompt-based techniques as well as pre-training and fine-tuning exploiting the little data available. While further pre-training gives the largest performance gains for no-resource languages, applying it directly to instruction-tuned models harms their ability to follow instructions. To address this, we start from a base model, further pre-training it on the target language, and then inject instruction-following capabilities via weight diff transfer from an instruction model. Such an approach significantly improves code generation capabilities in no-resource settings, allowing companies to cheaply deploy a specialized instruct model without dealing with the computational cost of instruction fine-tuning.
Alex: Okay, so the weights are basically what the model has learned.
Sam: Exactly. Now, here's the clever part. You take two versions of the same model: one is a raw base model, and one has been tuned to follow instructions. You calculate the difference between their weights—the gap between the dial settings. That difference, in isolation, represents the "instruction-following" personality. It's the thing that was added during the tuning process.
Alex: So you can extract that personality as a kind of recipe.
Sam: Precisely. Then you take your base model, train it on the new language so it absorbs the syntax and structure, and then you apply that recipe on top. You're adding the instruction-following capability back in, without having to redo the expensive training that created it in the first place. It's a surgical patch rather than a full rebuild.
Alex: And it actually works?
Sam: The results pushed performance well above the baseline. What's particularly notable is the efficiency. Because this approach is so much cheaper than full retraining, smaller models—around eight billion parameters—could actually outperform much larger models of around thirty-two billion parameters that had only been fine-tuned in the conventional way. A parameter, by the way, is roughly one of those weight settings we mentioned—so a larger model has more dials, and is generally more capable. But here, the smaller model with better training beat the bigger model with worse training.
Alex: So size isn't everything. It's how you train it.
Sam: That's the central finding. And the team was careful about data quality too—they manually reviewed hundreds of examples, checking for accuracy, clarity, and fluency, to make sure the performance gains were real and not just an artifact of noisy data.
Alex: That's a meaningful result for any organization that's built something internal and proprietary. You don't need a massive budget to get a capable coding assistant—you just need the right method for combining what already exists.
Sam: That's a fair summary. The research suggests there's a viable path for teams working with specialized or low-resource languages to build their own tools, without starting from zero. Thanks for listening to ResearchPod.