P-MTP: Efficient Document Parsing via Multi-Token Prediction with Progressive Depth Scaling

Vision-Language Models (VLMs) are highly effective at parsing document images into structured formats like JSON or LaTeX, but their autoregressive, token-by-token generation process creates significant latency, especially for dense documents. The authors investigate whether Multi-Token Prediction (MTP)—a technique that predicts multiple future tokens in a single forward pass—can be scaled to deeper look-ahead levels to accelerate document parsing without sacrificing accuracy.

The authors identify that existing MTP methods suffer from training instability when scaling to deeper look-ahead levels, often due to noisy gradients from distal tokens. They propose P-MTP, which introduces two key innovations:

1. Progressive Curriculum Loss: Instead of using fixed weights for all look-ahead depths, this mechanism adaptively weights losses based on the cumulative probability of preceding tokens and the consistency of predictions across different look-ahead distances. This creates an easy-to-hard training curriculum that suppresses gradient noise from unreliable, distant predictions.
2. Confidence-Gated Dynamic Drafting: Rather than using a fixed look-ahead depth during inference, this method dynamically adjusts the number of tokens to predict based on the model's real-time confidence. If the cumulative probability of a sequence remains high, the model continues to draft; if uncertainty increases, it truncates the sequence to avoid errors.

Experimental results demonstrate that P-MTP significantly outperforms traditional MTP approaches in document parsing tasks. By enabling deeper look-ahead depths through stable training and adaptive inference, the framework achieves up to a 5x speedup in inference latency. The authors show that this speedup is achieved with negligible impact on parsing accuracy, marking the first successful application of extensive look-ahead MTP in the document parsing domain.