PCSD: Persistent Consistency for Self-Distillation in Agentic Reinforcement Learning | Chunji Lv et al. | ResearchPod

Transcript: PCSD: Persistent Consistency for Self-Distillation in Agentic Reinforcement Learning

Alex: Welcome to another episode of ResearchPod. Today, we're looking at a paper that tackles a tricky problem in AI: how to help a student learn from a more capable teacher without picking up the teacher's mistakes.

Sam: That's the challenge. We're discussing a technique called Persistent Consistency Self-Distillation, or PCSD. Imagine a student who copies everything their tutor does—even the tutor's bad habits and slip-ups. If an AI student tries to copy a more powerful "teacher" model blindly, it often learns those same errors, or gets thrown off by random noise in the teacher's output.

Alex: So this paper is basically asking how to filter out the bad advice while keeping the good stuff?

Sam: Precisely. The core insight is this: if a piece of advice is genuinely useful, it shouldn't just appear once and vanish. It should show up repeatedly and consistently. The researchers call this a persistent signal. If the teacher's confidence in a particular action holds steady over several steps, that's a sign it reflects a real skill—not just a random blip.

Alex: So they're looking for patterns in the teacher's behavior to decide what to trust. Like, one data point isn't enough—you need to see it hold up over time?

Sam: Exactly. Think of it like getting directions from someone who seems confident. If they say "turn left" once and then go quiet, you're not sure. But if they keep saying "yes, left, definitely left" for the next few steps, you start to trust them. The system works the same way. It tracks how much more confident the teacher is compared to the student at each step, and if that confidence gap stays positive and steady, it gives that guidance more weight.

Alex: That's a useful way to think about it. So consistency is the signal that separates real knowledge from noise.

Sam: Right. And the system is also built to handle the fact that not all moments are equally reliable. If the teacher's guidance is stable, the system uses a tight, focused window to measure it. If things are noisy and inconsistent, it widens that window to smooth out the confusion—like squinting to see something more clearly when the light is bad.

Alex: But what happens when the teacher starts losing confidence mid-way through? What if the guidance is initially strong but then starts to fade?

Sam: That's where the second key mechanism comes in. The system doesn't just measure the level of the teacher's confidence—it also tracks the direction. Is that confidence growing, holding steady, or declining? If it's declining, the system automatically turns down the volume on that advice. It's a bit like a navigator who was certain about the route ten minutes ago but is now hesitating—you'd start second-guessing them too.

PCSD: Persistent Consistency for Self-Distillation in Agentic Reinforcement Learning

Chunji Lv, Yangguang Wei, Junlin Liu, Yang Gao, Ming Liu, Xinming Wang, Jinyang Wu, Guoren Wang, Changsheng Li

Aug 3, 2026·4:41·arxiv·Original Paper

Persistent Consistency Self-Distillation (PCSD)On-policy self-distillation (OPSD)Reward SparsityGroup Relative Policy Optimization (GRPO)PCSD (Persistent Credibility-based Self-Distillation)GRPO (Group Relative Policy Optimization)

Listen

5 min

Briefing

Research Question

In agentic reinforcement learning, agents often face sparse, delayed rewards that make credit assignment difficult. While on-policy self-distillation (OPSD) can provide dense token-level supervision using a privileged teacher, existing methods struggle to determine which teacher signals are reliable. The authors investigate how to effectively filter noisy teacher guidance to improve agent performance without relying on inference-time skills.

Approach

The authors propose Persistent Consistency Self-Distillation (PCSD), a framework that estimates teacher credibility by analyzing the local persistence of teacher-favoring signals. Instead of treating each token in isolation or applying uniform weights to entire steps, PCSD uses three mechanisms:

Adaptive Aggregation: It dynamically adjusts the window size for aggregating teacher-student probability gaps based on local variance, using shorter windows for stable regions and longer windows for noisy ones.
Trend-Aware Modulation: It identifies and attenuates regions where teacher support is declining, preventing the model from relying on transient or unreliable signals.
Continuous Gating: It maps these persistence estimates to smooth, continuous weights via a sigmoid function, which are then used to weight the distillation objective alongside standard GRPO.

Main Findings

PCSD consistently outperforms baseline methods, including standard GRPO and existing self-distillation weighting schemes, across ALFWorld and WebShop benchmarks. Specifically, on ALFWorld, PCSD achieved significant gains over GRPO (15.6 and 13.3 points on different backbones) and outperformed the SDAR baseline. The results indicate that by focusing on persistent local evidence, the agent can effectively filter out noise and leverage high-quality teacher guidance to improve fine-grained credit assignment.

Why It Matters

This work addresses the fundamental trade-off between token-level precision and robustness in self-distillation. By providing a principled way to weigh teacher supervision, PCSD enables more effective learning in long-horizon agentic tasks where environmental feedback is sparse. It demonstrates that incorporating local context into the distillation process is a powerful strategy for improving agent training without increasing inference-time computational costs.

Key Terms and Definitions

On-Policy Self-Distillation (OPSD) — A technique where a student model learns from a frozen, privileged-context-augmented version of itself to receive dense supervision.
Persistent Consistency — The core metric in this paper, defined as the sustained presence of teacher-favoring signals over a local neighborhood of tokens.
GRPO (Group Relative Policy Optimization) — An RL algorithm that optimizes agent behavior using group-relative rewards to avoid the need for a separate value function.
Privileged Context — Additional information provided to the teacher model during training that is not available to the student, allowing the teacher to act as a stronger supervisor.
Trend-Aware Modulation — A mechanism that reduces the distillation weight for tokens where the teacher's relative support is locally declining.

Abstract

Large language model agents have shown strong potential in complex interactive tasks, yet their reinforcement learning (RL) is often hindered by sparse rewards, as a long multi-turn trajectory may receive only a single outcome-level signal. On-policy self-distillation (OPSD) provides dense token-level supervision from a privileged teacher, but the teacher may not be reliable at every position. Existing methods commonly rely on isolated token-level discrepancies, which can be sensitive to noise, or assign a shared step-level weight that may overlook positional variation. We propose Persistent Consistency Self-Distillation (PCSD), which derives token-level distillation weights from the local persistence of teacher-favoring signals. PCSD combines adaptive windows with exponentially decayed aggregation to capture persistent relative teacher support, applies trend-aware modulation to attenuate locally declining support, and produces continuous weights through sigmoid gating. The resulting objective is jointly optimized with GRPO, combining dense teacher guidance with sparse environmental feedback. Without inference-time skills, PCSD achieves the best ALFWorld Overall results among all baselines on both backbones, exceeding GRPO by 15.6 and 13.3 points and SDAR by 6.2 and 5.5 points, while remaining competitive on WebShop and gaining 15.8 points over GRPO on unseen ALFWorld split.

PCSD: Persistent Consistency for Self-Distillation in Agentic Reinforcement Learning: Research Paper Podcast