Learning from the Self-future: On-policy Self-distillation for dLLMs | Yifu Luo et al. | ResearchPod

Transcript: Learning from the Self-future: On-policy Self-distillation for dLLMs

Alex: Welcome to another episode of ResearchPod.

Sam: Today we're looking at a new training framework called d-OPSD — a method for helping text-generating AI improve its own reasoning without needing a separate, more powerful model to guide it.

Alex: So it's about getting these models to teach themselves more effectively?

Sam: Exactly. But to understand why that's tricky, you need to know that not all AI text generators work the same way. Most current training methods were designed for models that write one word at a time — like reading a sentence left to right. Diffusion models work differently. Instead of building text word by word, they start with something like a blurry, noisy draft and gradually clean it up into a finished answer. Existing training methods just don't fit that process.

Alex: That's like trying to teach someone who thinks in pictures using a manual written for someone who thinks in lists.

Sam: A good way to put it. Think of a sculptor. A traditional model carves a statue line by line, left to right. A diffusion model starts with a rough, noisy block and chips away from all directions at once until the final form emerges. That process is called "denoising." Because it doesn't work in a straight line, it can consider the whole shape at once rather than just the part it's currently working on.

Alex: And if it's working on the whole thing at once, how does it learn from its own mistakes?

Sam: That's the heart of the challenge. In older word-by-word models, the system looks at everything it's already written to guess the next word. But because a diffusion model sees the whole sequence, researchers found they could flip that around. They take a partial, self-generated answer and use it as a hint for the earlier reasoning steps — like a student glancing at the answer key halfway through a test, then working backwards to understand why that answer is correct.

Alex: So the model is using its own future output to guide its current reasoning?

Sam: Precisely. The researchers call this "On-policy Self-distillation," or OPSD. The model generates a response, then uses its own later steps as a guide to refine the earlier ones. It acts as its own teacher.

Alex: That raises an obvious question though — if the model is just copying its own future output, doesn't it risk getting stuck in a loop of its own bad habits?

Sam: That's a valid concern, and the researchers addressed it directly. The system only reveals a small, random fraction of the final answer at any one time — it never sees the whole answer at once. That forces the model to actually learn the underlying logic rather than just memorize a shortcut. It's like practicing a difficult piano piece, getting one passage right, and then carefully studying exactly what your hands did in that moment — rather than playing the whole thing over and over and hoping it sticks.

Learning from the Self-future: On-policy Self-distillation for dLLMs

Yifu Luo, Zeyu Chen, Haoyu Wang, Xinhao Hu, Yuxuan Zhang, Zhizhou Sha, Shiwei Liu

Jun 16, 2026·4:18·arxiv·Original Paper

On-policy self-distillation (OPSD)Diffusion LLMs (dLLMs)d-OPSDSuffix conditioningStep-level supervisionDiffusion Large Language Models (dLLMs)

Listen

4 min

Briefing

Research Context and Motivation

On-policy self-distillation (OPSD) has become a powerful paradigm for improving Large Language Models (LLMs) by allowing a model to learn from its own generated trajectories. However, existing OPSD methods are designed for autoregressive (AR) models, which rely on left-to-right token generation. Diffusion LLMs (dLLMs) operate differently, generating sequences through an iterative, non-autoregressive denoising process. This fundamental mismatch makes standard AR-centric distillation techniques—which rely on prefix-based conditioning and token-level supervision—incompatible with the architecture of dLLMs.

The d-OPSD Framework

The authors propose d-OPSD, the first OPSD framework specifically tailored for dLLMs. The approach introduces two key innovations to bridge the gap between AR-based distillation and diffusion-based generation:

Suffix-Conditioned Self-Teacher: Instead of using static reference solutions or prefix-based prompts, the framework uses the model's own previously generated answers as 'future-experience.' By revealing parts of a self-generated trajectory to the teacher as a suffix, the model can learn from its own successful reasoning patterns.
Step-Level Supervision: Rather than enforcing token-level divergence, which is ill-suited for the simultaneous token prediction of dLLMs, the authors implement step-level divergence. This aligns the training objective with the iterative denoising steps, where the model transitions from a fully masked sequence to a clean output.

Performance and Efficiency

Experiments across four reasoning benchmarks (GSM8K, MATH500, Countdown, and Sudoku) demonstrate that d-OPSD consistently outperforms traditional RLVR and SFT baselines. Beyond raw performance, the framework exhibits remarkable sample efficiency, achieving superior results with roughly 10% of the optimization steps required by the diffu-GRPO baseline. This suggests that leveraging 'self-future' information is a highly effective way to drive self-improvement in non-autoregressive language models.

Key Terms and Definitions

On-policy Self-distillation (OPSD) — A training paradigm where a model acts as both the student and the teacher, using its own generated outputs to provide dense supervision for further learning.
Diffusion LLM (dLLM) — A type of language model that generates text through an iterative denoising process, predicting masked tokens simultaneously rather than in a strict left-to-right order.
Suffix Conditioning — A technique where privileged information is provided to the model by appending it to the end of the input, which is made possible by the bidirectional nature of dLLMs.
Step-level Divergence — A supervision method that calculates the difference between teacher and student distributions at each denoising step of the diffusion process, rather than at the individual token level.
Privileged Information — Extra context or ground-truth data provided to the teacher model during training to guide the student model toward better reasoning paths.

Abstract

On-policy self-distillation (OPSD) has proven effective for post-training large language models (LLMs), yet its application to diffusion LLMs (dLLMs) remains unexplored. Existing OPSD methods are inherently autoregressive-centric. They inject privileged information via left-to-right prefix conditioning with token-level divergence supervision, a design that fundamentally conflicts with the arbitraryorder generation of dLLMs. We introduce d-OPSD, the first OPSD framework tailored for dLLMs. Our approach makes two core contributions. First, we reframe self-teacher construction by using self-generated answers as suffix conditioning, enabling the student model to learn from "self future-experience" rather than privileged prefixes. Second, we shift supervision from token-level to step-level, aligning training with the iterative denoising process of dLLMs. Experiments across four reasoning benchmarks show that d-OPSD consistently outperforms RLVR and SFT baselines with superior sample efficiency, requiring only around 10% of the optimization steps by RLVR and opening a promising pathway for dLLM posttraining. The code is available at https://github.com/xingzhejun/d-OPSD.

Learning from the Self-future: On-policy Self-distillation for dLLMs: Research Paper Podcast