Finding the Evidence: Discovering Decision-Supporting Tokens for On-Policy Reasoning Distillation

On-policy distillation (OPD) is the standard for teaching small models to reason, but current methods struggle to identify which tokens actually contain transferable reasoning knowledge. While entropy-based selection successfully identifies "decision" tokens (where the model branches), it fails to capture the "evidence" tokens—the intermediate steps where a model might be confident but wrong. This paper investigates how to identify and distill these missing evidence tokens to improve reasoning performance.

The authors propose DEAR (Decision-Evidence Aware Reasoning Distillation). The method operates in two stages:

1. Decision Identification: It uses student entropy to identify high-uncertainty tokens, which represent the "reasoning skeleton" or branching points.
2. Evidence Discovery: It identifies evidence tokens by measuring the hidden-state cosine similarity between non-decision tokens and the identified decision anchors. This is further boosted by teacher-student divergence to prioritize tokens where the teacher possesses knowledge the student lacks.

By combining these two sets, DEAR creates a comprehensive training signal that covers both the "where to branch" and the "how to justify" aspects of reasoning.

DEAR consistently outperforms standard OPD and entropy-only baselines across various math and code generation benchmarks. Specifically, it achieves up to +2.5pp on competition math and +5.7pp on code generation tasks. The authors demonstrate that while entropy-only selection captures only about 39% of the total gradient mass, DEAR captures approximately 76%, effectively closing the gap in reasoning knowledge transfer. The gains are most pronounced on harder problems where long, multi-step derivations are required.

This work highlights a fundamental limitation in current distillation paradigms: treating all tokens equally or focusing solely on high-entropy tokens ignores the substantive, low-entropy intermediate steps that constitute the bulk of reasoning. By providing a mechanism to discover these "hidden" evidence tokens, DEAR offers a more efficient and effective way to distill reasoning capabilities into smaller models without requiring additional forward passes or complex architectural changes.