Transferability for General Reasoning: An Automated Curriculum for Multi-Domain RLVR

In multi-domain reinforcement learning with verifiable rewards (RLVR), reasoning skills often transfer unevenly across different tasks. While existing curricula prioritize domains where a model is currently improving (learnability), they are often blind to whether a gradient update on one domain actually helps other domains. This paper asks: can we design an automated curriculum that selects training domains based on their cross-domain transferability, and can this be done efficiently without expensive extra computations?

The authors propose the Transfer-Aware Curriculum (TAC), a bandit-style online curriculum. TAC selects domains based on a feedback signal that combines two components:

1. Local Learnability: A z-score of the mean absolute advantages from the Group Relative Policy Optimization (GRPO) process.
2. Cross-Domain Transferability: A novel estimator that uses random projections to sketch gradients into a low-dimensional space. By maintaining an exponential moving average of these projected gradients, TAC calculates the pairwise cosine similarity between domains. This signal identifies which domains produce updates that align with the broader training suite.

These two signals are combined into a single score that guides the bandit's sampling distribution. The entire process is integrated into the standard RL training loop with less than 1% wall-clock overhead.

TAC was evaluated on a six-domain reasoning suite (math, programming, logic, simulation, tables, and science) using Qwen3-1.7B and Llama3.2-3B models. The results show that TAC consistently outperforms proportional random sampling, hand-designed schedules, and learnability-only bandit approaches. Specifically, TAC improved macro-averaged accuracy by up to 2.8 points (a 10% relative improvement). Ablation studies confirmed that the transferability term is critical to this performance, and the method remains robust even when training data mixtures are imbalanced.