Which Tokens Should SFT Actually Learn? A Token-Trimming Perspective on Mathematical Reasoning

Standard supervised fine-tuning (SFT) treats all tokens in a training sequence with equal importance, applying a uniform cross-entropy loss. However, in complex tasks like mathematical reasoning, some tokens are already well-mastered (leading to over-sharpening), while others are highly uncertain or noisy (dominating the loss). This paper investigates whether selectively reweighting tokens based on their internal model confidence—specifically their logit gap—can improve training dynamics and reasoning performance.

The authors propose Trimmed Logit-Gap SFT (TrimSFT). This method calculates the logit gap for each token, defined as the difference between the logit of the gold token and that of its strongest competitor. Using a Gaussian weighting function centered at a margin (m) with a specific bandwidth (τ), the method assigns lower weights to tokens at both extremes of the logit-gap spectrum. This effectively 'trims' the loss for tokens that are either too easy (already mastered) or too hard (weakly supported), concentrating the model's learning signal on an intermediate range of difficulty. The approach is computationally efficient, requiring no reference models or additional forward passes.

TrimSFT consistently outperforms standard SFT across six base models (including Llama, Qwen, and DeepSeekMath families) and five mathematical benchmarks. Notably, it achieved gains of up to +26.9 points on the MATH500 benchmark. Ablation studies reveal that the bandwidth (τ) is more critical than the exact margin (m), and that the two-sided trimming profile is essential; half-trim variants that only remove supervision from one side perform significantly worse. Distributional analysis shows that TrimSFT reshapes model confidence more effectively than uniform SFT or monotonic reweighting methods, leading to better self-consistency and capability coverage.