Privileged Likelihood Is Not Automatically Value: Three Checks for Token Credit in On-Policy Self-Distillation

Outcome verifiers provide a binary signal (correct or incorrect) for a completed reasoning trace but fail to assign credit to individual intermediate tokens. Privileged self-distillation attempts to bridge this gap by using training-only information—such as reference solutions or hindsight critiques—to rescore a model's own rollouts. This paper investigates whether these "privileged likelihood scores" actually function as meaningful credit for task success, or if they merely reflect stylistic preferences or artifacts of the feedback generation process.

The authors formalize the problem by separating three distinct questions:

1. Does the score track task success (i.e., does it prefer actions that lead to better outcomes)?
2. Does the feedback construction introduce bias (e.g., does the score depend on the specific rollout it is evaluating)?
3. What behavior does the training loss actually reinforce (i.e., how do the scores translate into parameter updates)?

They test these questions using a 20B parameter model on the AIME 2025 dataset. They compare various token-score variants against an outcome-only control, measuring performance through AUC and training outcomes, while formally analyzing the conditions under which likelihood ratios can represent valid action values.

The experiments reveal that privileged likelihood scores are not inherently useful. In matched experiments, the additive scores performed near chance (AUC=0.505) and, after adjusting for response length, actually favored incorrect traces. Across all training runs, the outcome-only control significantly outperformed the token-score variants (64.2% vs. 24.2%–33.9%). The authors demonstrate that even if a score is structurally well-formed, the training loss can reverse the sign of the update or reinforce irrelevant features, meaning that likelihood signals cannot be treated as synonymous with outcome credit without rigorous validation.