Chris Han, Pengzhi Gao, Pei Fu, Jian Luan
6 min
Abstract
We study reference-free post-training for multilingual machine translation with open large language models. Starting from the supervised-finetuned MiLMMT-46-v0.1 models, we apply Group Relative Policy Optimization (GRPO) with a reward that averages two reference-free quality estimation models and is gated by language identification. We then linearly interpolate the supervised fine-tuning (SFT) and reinforcement learning (RL) model checkpoints to obtain MiLMMT-46-v1.0. Across 46 languages, the resulting models consistently improve translation quality over their SFT counterparts, outperform strong recent open baselines, including Seed-X, HY-MT2, and TranslateGemma, and achieve leading reference-free scores against evaluated proprietary systems such as Google Translate, Gemini 3 Pro, and GPT-5. We further investigate on-policy distillation and find that it reaches, but does not surpass, the quality frontier achieved by RL with checkpoint interpolation. We release the models and code to facilitate future research.
Alex: So rather than just swapping in the new model and hoping for the best, they're carefully blending the old and new together to stay stable?
Sam: Precisely. And that stability matters. Without it, the model might become fluent in ways that drift from what it was originally good at. The blend keeps both qualities intact.
Alex: Let's go a bit deeper on the ranking side. How exactly does the model decide which of its own translations is "better"?
Sam: The technique is called Group Relative Policy Optimization—GRPO for short. Here's a useful way to picture it. Imagine a student who writes ten different versions of the same essay. Instead of grading each one against a single answer key, the teacher lines them all up and asks: which of these is most logical, most coherent, most well-argued? The student learns not from a fixed standard, but from comparing their own attempts against each other. The model works the same way—it doesn't need a perfect reference translation, just a reliable way to rank its own outputs and learn from the differences.
Alex: That makes sense. But does this approach work for smaller models, or does it only work if you have enormous computing power?
Sam: That's where on-policy distillation comes in. Think of it like a master craftsperson training an apprentice. The large model does the hard work of figuring out which translations are better through the ranking process. The smaller model then watches those decisions and learns to mimic them on new sentences—without having to run the full ranking process itself every time. The improvements get passed down without the computational cost.
Alex: So the large model does the difficult reasoning, and the smaller model inherits the results of that reasoning without having to repeat the process from scratch?
Sam: Exactly. It's a practical way to make these gains accessible without requiring massive infrastructure.
Alex: It sounds like the whole system is designed to be efficient—better translations, without needing human-labeled data at every step, and without requiring enormous computers to run it.
Sam: That's a fair summary. There is one honest trade-off worth mentioning, though. While these models improve in naturalness and fluency, the paper notes they sometimes show a modest decrease in what's called "lexical overlap"—that's the degree to which the translation matches the source word-for-word, as measured by traditional scoring tools. So the translations may read more naturally to a human, but score slightly lower on older automated metrics that prioritize exact word matches. It's a known tension in the field.
Alex: So the model is getting better at the thing humans actually care about—does this sound like a real translation?—but the older measuring tools don't fully capture that?
Sam: That's the suggestion, yes. It points to a broader question in machine translation research: whether the tools we use to measure quality are keeping pace with the models themselves. This paper doesn't resolve that debate, but it does add to it in a useful way.
Alex: A self-improving system that learns without a fixed answer key, stays honest through a language gate, and can pass its gains down to smaller models. There's a lot of careful engineering behind what looks like a simple idea. Thanks for walking us through it, Sam.
Sam: Thanks, Alex. And thanks to everyone listening to ResearchPod.