We study reference-free post-training for multilingual machine translation with open large language models. Starting from the supervised-finetuned MiLMMT-46-v0.1 models, we apply Group Relative Policy Optimization (GRPO) with a reward that averages two reference-free quality estimation models and is gated by language identification. We then linearly interpolate the supervised fine-tuning (SFT) and reinforcement learning (RL) model checkpoints to obtain MiLMMT-46-v1.0. Across 46 languages, the resulting models consistently improve translation quality over their SFT counterparts, outperform strong recent open baselines, including Seed-X, HY-MT2, and TranslateGemma, and achieve leading reference-free scores against evaluated proprietary systems such as Google Translate, Gemini 3 Pro, and GPT-5. We further investigate on-policy distillation and find that it reaches, but does not surpass, the quality frontier achieved by RL with checkpoint interpolation. We release the models and code to facilitate future research.
Alex: Welcome to another episode of ResearchPod. Today, we're looking at a paper about machine translation—specifically, how to make AI translation models better even when you don't have a perfect "answer key" to learn from.
Sam: That's right. The study explores what the researchers call "reference-free post-training." The core puzzle is this: normally, to train a translation model, you need thousands of matched sentence pairs—one sentence in English, the same sentence in French, and so on. But what if you don't have that? Can you still improve the model?
Alex: So the question is whether we can teach a model to become a better translator just by showing it raw text, without a human ever providing the "correct" translation for each example?
Sam: Exactly. Here's how it works. The model generates several different translations of the same sentence—say, five or six attempts. Then a separate scoring program evaluates each one and identifies which sounds most natural and accurate. By rewarding the model every time it picks one of those higher-quality versions, it gradually learns to produce better translations on its own.
Alex: That's a clever workaround. But it raises an obvious concern—if the model is just guessing and being scored by another program, what stops it from cheating? Like, finding a way to score well without actually translating properly?
Sam: That's exactly the right concern, and the researchers take it seriously. This kind of cheating is called "reward hacking"—where a model learns to game the scoring system rather than do the actual job. For instance, it might produce very fluent, natural-sounding text that just isn't a translation at all, or it might drift into a different language that the scorer happens to rate highly. To prevent this, the authors built in what they call a "language gate." Think of it as a strict supervisor standing at the door. Before any reward is given, it checks: is the output actually in the right language? If not, the reward drops to zero, full stop.
Alex: So the gate keeps it honest, and the scoring system keeps it accurate. Between those two, the model can't really cut corners. Does this actually hold up when you compare it to the big commercial systems?
Sam: It does, according to the paper. Their open-source models—the kind anyone can download and use—outperformed some large proprietary systems on standard translation quality tests. That's a meaningful result, because proprietary systems typically have far more resources behind them.
Alex: They also mention something called "checkpoint interpolation." What does that actually do?
Sam: Good question. Think of it as blending two versions of the same model. You have the original version, which is cautious and follows strict patterns it learned during initial training. Then you have the newer version, which has been through this reward-based training and has become more fluent—but potentially a bit unpredictable. Checkpoint interpolation mixes the internal settings of both versions together. The result is a model that keeps the reliability of the original while also benefiting from the improvements the new training produced.
Alex: So rather than just swapping in the new model and hoping for the best, they're carefully blending the old and new together to stay stable?
Sam: Precisely. And that stability matters. Without it, the model might become fluent in ways that drift from what it was originally good at. The blend keeps both qualities intact.
Alex: Let's go a bit deeper on the ranking side. How exactly does the model decide which of its own translations is "better"?
Sam: The technique is called Group Relative Policy Optimization—GRPO for short. Here's a useful way to picture it. Imagine a student who writes ten different versions of the same essay. Instead of grading each one against a single answer key, the teacher lines them all up and asks: which of these is most logical, most coherent, most well-argued? The student learns not from a fixed standard, but from comparing their own attempts against each other. The model works the same way—it doesn't need a perfect reference translation, just a reliable way to rank its own outputs and learn from the differences.
Alex: That makes sense. But does this approach work for smaller models, or does it only work if you have enormous computing power?
Sam: That's where on-policy distillation comes in. Think of it like a master craftsperson training an apprentice. The large model does the hard work of figuring out which translations are better through the ranking process. The smaller model then watches those decisions and learns to mimic them on new sentences—without having to run the full ranking process itself every time. The improvements get passed down without the computational cost.
Alex: So the large model does the difficult reasoning, and the smaller model inherits the results of that reasoning without having to repeat the process from scratch?
Sam: Exactly. It's a practical way to make these gains accessible without requiring massive infrastructure.
Alex: It sounds like the whole system is designed to be efficient—better translations, without needing human-labeled data at every step, and without requiring enormous computers to run it.
Sam: That's a fair summary. There is one honest trade-off worth mentioning, though. While these models improve in naturalness and fluency, the paper notes they sometimes show a modest decrease in what's called "lexical overlap"—that's the degree to which the translation matches the source word-for-word, as measured by traditional scoring tools. So the translations may read more naturally to a human, but score slightly lower on older automated metrics that prioritize exact word matches. It's a known tension in the field.
Alex: So the model is getting better at the thing humans actually care about—does this sound like a real translation?—but the older measuring tools don't fully capture that?
Sam: That's the suggestion, yes. It points to a broader question in machine translation research: whether the tools we use to measure quality are keeping pace with the models themselves. This paper doesn't resolve that debate, but it does add to it in a useful way.
Alex: A self-improving system that learns without a fixed answer key, stays honest through a language gate, and can pass its gains down to smaller models. There's a lot of careful engineering behind what looks like a simple idea. Thanks for walking us through it, Sam.
Sam: Thanks, Alex. And thanks to everyone listening to ResearchPod.