arxiv

Meta-Learning Preferences for Multilingual LLM Alignment

Jiaying Lin, Seongho Son, Nam Phuong Tran, Long Tran-thanh et al.Jul 14, 2026OriginalPDF

ResearchPod Summary

Research Question

Aligning Large Language Models (LLMs) to human preferences typically requires large, high-quality datasets. This creates a significant barrier for low-resource languages where such data is scarce. The authors investigate whether a meta-learning framework can leverage preference data from high-resource languages to learn a transferable initialization, allowing for efficient alignment in target languages with minimal data.

Approach

The researchers formulate multilingual preference alignment as a task-level adaptation problem, where each language represents a distinct task. They adapt the Model-Agnostic Meta-Learning (MAML) framework to two popular alignment methods: Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO).

MAML-RLHF: Meta-learns a reward function initialization across multiple languages, which is then fine-tuned on the target language to guide policy optimization.
MAML-DPO: Meta-learns a policy initialization directly, bypassing the separate reward-modeling step.

The authors provide theoretical convergence guarantees for both methods, establishing the conditions under which meta-trained initializations provably accelerate adaptation compared to standard multitask or target-only baselines.

Main Findings

Empirical evaluations on multilingual benchmarks (using models ranging from 270M to 7B parameters) demonstrate that the meta-learning approach significantly outperforms standard baselines. In extremely low-resource settings (100 preference samples), the meta-DPO approach achieves up to a 28% improvement in win rates. The performance gains are robust across varying linguistic distances between meta-training and target languages, and the method maintains its advantage even as the amount of adaptation data increases.

Why It Matters

This work provides a scalable solution for multilingual LLM alignment, addressing the "data inequality" problem in AI development. By demonstrating that models can learn to learn preferences across languages, the framework enables more equitable performance for speakers of low-resource languages without requiring massive, language-specific datasets.

Paper Details

arXiv: 2607.13315

Topics

Meta-LearningMultilingual LLM AlignmentReinforcement Learning from Human Feedback (RLHF)Direct Preference Optimization (DPO)Low-resource Language AdaptationMAML-DPO/MAML-RLHFMultilingual Preference AlignmentSample EfficiencyBradley-Terry ModelGradient-based Meta-learningRLHF (Reinforcement Learning from Human Feedback)DPO (Direct Preference Optimization)