Daijiao Liu, Huiyi Wang, Lina Yao, Dong Gong
9 min
Reinforcement learning with verifiable rewards (RLVR) typically relies on sampling small groups of trajectories from a policy to calculate group-relative advantages. However, this approach often leads to rollout homogeneity, where repeated sampling yields highly similar outputs, collapsing the advantage signal and limiting the diversity of discovered solutions. The authors investigate whether intervening directly at the rollout source through minimal input perturbations can broaden trajectory exploration and enhance reasoning capabilities.
The authors propose EDEN (Embedding-space Directional Exploration via anchored Noise). Instead of changing the policy architecture or the training objective, EDEN intervenes at the input layer. For a subset of rollouts in each group, the system appends a single, perturbed embedding vector to the prompt. This perturbation combines a shared, fixed anchor embedding—which provides a consistent conditioning direction—with isotropic Gaussian noise, which introduces local variation. The system uses an advantage-gated filter to retain only the most informative perturbed rollouts, which are then integrated into the training updates with scaled advantages.
EDEN consistently outperforms standard Group Relative Policy Optimization (GRPO) and common exploration baselines (such as higher sampling temperatures or asymmetric clipping) across various model scales, including Qwen2.5-0.5B, 1.5B, and 7B. Experiments across a range of mathematical benchmarks, from grade-school arithmetic (GSM8K) to challenging competition-level tasks (AIME), show that EDEN achieves significantly higher solution coverage. The authors demonstrate that the combination of noise and anchoring is complementary: the anchor provides a directed exploration bias, while the noise ensures sufficient within-group variance. Training dynamics reveal that EDEN maintains higher policy entropy and produces more non-zero group-relative advantage signals compared to the baseline.
As RLVR becomes the standard for post-training reasoning LLMs, the "rollout ceiling"—where increasing the number of rollouts yields diminishing returns due to lack of diversity—becomes a major bottleneck. EDEN demonstrates that the locus of exploration matters; by changing the conditions under which trajectories are generated, researchers can extract more useful signal from the same rollout budget. Because it requires no changes to model architecture, test-time inference, or decoding, it offers a highly compatible and efficient lever for improving the reasoning performance of existing RLVR pipelines.
Reinforcement learning with verifiable rewards (RLVR) typically uses small rollout groups per prompt, where repeated sampling from the same policy can yield highly similar trajectories, reducing within-group reward variation and weakening group-relative advantage signals. Existing remedies such as higher sampling temperature, entropy bonuses, asymmetric clipping, and larger rollout budgets broaden sampling or modify policy updates but leave rollout conditioning unchanged. In this paper, we propose EDEN (Embedding-space Directional Exploration via anchored Noise), which intervenes at the rollout source through anchored input perturbations. EDEN explores beyond the policy's own rollouts. Within the same rollout budget, it replaces a small fraction of standard rollouts in each group with rollouts that the same policy generates from the prompt plus a single appended, perturbed embedding position, rather than from the prompt alone. Gaussian noise drives variation, while a shared anchor provides a common conditioning direction within each prompt group. Selected perturbed rollouts contribute to GRPO updates with downweighted advantages, and the model architecture, discrete decoding, and test-time inference remain unchanged. Across model scales and mathematical reasoning benchmarks, EDEN outperforms GRPO and the exploration controls evaluated in the same pipeline, with the largest gains in sampled solution coverage. Analyses show that more prompt groups provide non-zero group-relative learning signals and that noise and anchoring play complementary roles. Intervening at the rollout source is thus an effective lever for exploration in RLVR.
Sam: Make that input change concrete. What actually reaches the model?
Alex: An embedding is a numerical vector representing an input token. EDEN appends one extra embedding immediately after the prompt, before the answer begins. It takes that vector from a vocabulary token and adds Gaussian noise: random variation spread equally across directions.
Sam: If the noise has no preferred direction, why do the authors call this directional exploration?
Alex: Because the noisy answers for a given problem share an anchor: the same starting vector. Their random variations differ, but they’re centered on that common anchor. Across problems, the method selects different anchors from a fixed, screened bank.
Sam: So the anchor supplies common conditioning, and the noise lets answers vary around it. How much of a training group gets this treatment?
Alex: The default budget is sixteen answers per problem, matching the baseline. Four get the extra noisy embedding; the rest use the ordinary prompt. The model receives no further perturbations while generating those answers.
Sam: Those answers were produced under different conditions. Can the training procedure simply treat them as ordinary answers?
Alex: It doesn’t treat them identically. First, rewards and advantages are calculated across the mixed group. Then, across the training batch, EDEN keeps only the noisy answers with the largest absolute advantages. Their contribution to the update is also reduced.
Sam: Absolute advantage includes negative values. Why keep a strongly unsuccessful answer?
Alex: Because informative doesn’t mean successful. With correct-or-incorrect rewards, the minority outcome within a mixed group has the larger absolute advantage. A rare success can be encouraged; a rare failure can be discouraged. The selection favors deviations from the group’s dominant outcome.
Sam: And does the model learn those answers with the noisy input still attached?
Alex: No. During the update, that extra position is removed, and answer probabilities are evaluated under the clean prompt. That’s central to the design: explore under altered conditions, then incorporate selected trajectories into the ordinary model. Deployment uses ordinary prompts too.
Sam: That seems like both the attraction and a potential weakness. Does the paper correct for the difference between generation and training conditions?
Alex: It explicitly does not use an importance-sampling correction for that mismatch. Filtering and reduced advantage weights limit the perturbed answers’ influence instead. My reading is that this is a practical training recipe with empirical support, not a corrected estimate of clean-prompt learning.
Sam: Let’s turn to that support. What is the clearest performance result?
Alex: On the smaller math-specialized model, EDEN improves average Pass-at-eight by about three percentage points over GRPO. Pass-at-eight asks whether at least one answer in that many samples is correct. The average here covers the main comparison’s mathematics benchmarks.
Sam: Does it mainly improve the chance of finding one good answer, or does the ordinary single answer improve too?
Alex: Both averages improve, but the sampled-coverage gain is larger. Greedy single-answer accuracy improves by about two and a half percentage points in that comparison. The largest tested model shows the same pattern: a smaller single-answer gain and a larger coverage gain.
Sam: If coverage is the main benefit, does it survive when the baseline gets many chances?
Alex: On the difficult AIME competition problems, they evaluate with hundreds of answers per problem. EDEN’s average coverage advantage over GRPO is about seven percentage points across the math-specialized model sizes. That suggests the benefit isn’t confined to a small evaluation budget.
Sam: Could this just be another way to spend more computation, or to keep the model’s output more random?
Alex: The generation budget is matched, though the anchor bank requires screening before training. Giving GRPO larger answer groups did not reproduce EDEN’s gains in the small-model comparison. Stronger entropy bonuses raised randomness without improving validation accuracy, and higher temperatures gave inconsistent gains.
Sam: The component tests seem especially useful here. Does either the anchor or the noise work alone?
Alex: Neither reproduces the full improvement. Removing noise brings performance near GRPO, while unanchored noise also falls short. A fixed random anchor recovers part of the gain, but choosing unrelated random anchors for individual answers performs worse.
Sam: That supports a shared center rather than arbitrary disturbance. Do they see the learning signal itself change?
Alex: Later in training, EDEN has fewer groups with identical rewards and therefore zero advantages. The authors also find broader variation in correct answer traces. But their diversity measures use text patterns and predefined reasoning markers, so I wouldn’t equate them with independently verified new reasoning strategies.
Sam: How far would you trust this outside the experiments?
Alex: The evidence is promising within mathematics, with controlled comparisons and component tests. But all tested models belong to the same family, and repeated training-seed results are reported for the smallest model. Stability also matters: removing the temperature warmup caused training to crash in an ablation.
Sam: So who should read the full document, and where should they start?
Alex: Researchers building verifiable-reward training pipelines should read the noise-aware policy update section first, especially the generation-to-update mismatch. Then examine the anchored-noise ablations and the comparison with larger rollout budgets. Those show both what the intervention adds and which safeguards it needs.
Sam: And what should everyone else carry away?
Alex: Better exploration can mean changing the conditions that produce answers, not just asking for more answers. Here, that improved mathematical solution coverage without changing how the trained model is used.
Sam: Keep the generation conditions in view.