Guobin Shen, Chenxiao Zhao, Xiang Cheng, Lei Huang, Xing Yu
5 min
Abstract
Training stability remains a central challenge in reinforcement learning (RL) for large language models (LLMs). Policy staleness, asynchronous training, and mismatches between training and inference engines all cause the behavior policy to diverge from the current policy, risking training collapse. Importance sampling provides a principled correction for this distribution shift but suffers from high variance; existing remedies such as token-level clipping and sequence-level normalization lack a unified theoretical foundation. We propose Variational sEquence-level Soft Policy Optimization (VESPO). By incorporating variance reduction into a variational formulation over proposal distributions, VESPO derives a closed-form reshaping kernel that operates directly on sequence-level importance weights without length normalization. Experiments on mathematical reasoning benchmarks show that VESPO maintains stable training under staleness ratios up to 64x and fully asynchronous execution, and delivers consistent gains across both dense and Mixture-of-Experts models. Code is available at https://github.com/FloyedShen/VESPO
Alex: That balance makes sense for stability. But what stops the middle ground from getting too wild and causing those explosive swings anyway?
Sam: They add a firm limit on variance, like a guardrail capping how much "traffic"—big weight swings—can pile up under this new map. Solving this constrained math puzzle gives a clean formula: the reshaping becomes the weight raised to a power times an exponential drop for extremes, softly damping outliers without hard cuts.
Alex: Oh— so it's not just clipping wildly; it's a principled curve that hugs both sides while keeping things from blowing up. Like tuning a bridge to handle rush hour without collapsing.
Sam: Precisely. This closed-form kernel stays fair across sequence lengths, no sneaky biases, and holds steady even under heavy staleness or full async runs.
Alex: That smoothing sounds key to avoiding crashes. Does it treat good and bad examples the same way?
Sam: No—they use separate settings for sequences better than average and those worse. Better ones get a milder curve to keep useful signals strong; worse ones face stronger damping to avoid over-penalizing what the AI already avoids. Researchers call this *asymmetric soft clipping*, and it mirrors how some training methods handle ups and downs differently, computed safely by working in log-space—adding logs instead of multiplying huge numbers directly to prevent computer glitches.
Alex: Log-space for safety, got it. Those curve settings sound tunable, but what do the actual tests show?
Sam: The researchers trained versions of these math-solving AIs on clusters of powerful computer chips, feeding them thousands of math problems with rewards based on correct answers verified by another checker tool. They tested on tough high-school level contests, like AIME and AMC, where problems require step-by-step logic much like the training. VESPO reached the highest average scores across three different AI sizes.
Alex: Steady progress. But why does it shine more on the largest model?
Sam: That one uses MoE, where the AI splits work across specialized sub-teams of its brain, like dividing a school project among expert groups—but tiny routing choices differ between training and testing, worsening the old-data mismatch. This method's smooth weight handling cuts those amplified swings better than token-by-token clips, keeping training on track even fully async, with rollout machines generating data separately from the trainer.
Alex: So the math-derived smoothing stabilizes and scales to messy real setups without tweaks per model. Worth watching for broader use.
Sam: The paper suggests as much for math reasoners, though hyperparameter choices like the curve strengths need per-task care, and checks on other skills like chat alignment are next. It's a notable advance for handling async clusters reliably.
Alex: A meaningful step for making these trainings practical at scale, grounded in careful analysis. Thanks for joining us on ResearchPod.