D-cut: Adaptive Verification Depth Pruning for Batched Speculative Decoding

In batched speculative decoding, long draft sequences often lead to a "verification cost explosion" where the time spent verifying rejected tokens outweighs the speedup gained from accepted ones. The authors investigate how to optimize this trade-off by moving away from fixed-depth verification toward a strategy that adapts to both the varying confidence of individual requests and the specific hardware constraints of the deployment environment.

D-cut introduces a two-stage adaptive pruning mechanism. First, it performs cross-request pruning by ranking all draft tokens in a batch based on their predicted acceptance probability (derived from the drafter's confidence scores). Second, it incorporates a runtime-aware cost model that profiles the latency of different verification depths on the specific target hardware (e.g., H20 vs. H800 GPUs). By dynamically selecting a global verification budget that maximizes the projected speedup, D-cut ensures that the system only verifies tokens that provide a net positive contribution to throughput.

The study demonstrates that fixed-depth speculative decoding (like DFlash) often performs worse than standard autoregressive decoding at high concurrency due to wasted computation. D-cut effectively mitigates this by pruning low-utility draft tokens. Across various dense and Mixture-of-Experts (MoE) models, D-cut improves average speedups from 1.26x to 1.65x under high concurrency. In many configurations where baseline speculative decoding was slower than autoregressive methods, D-cut successfully restores acceleration, achieving up to 3.0x speedup on MoE models.

As LLM serving moves toward high-concurrency environments, the efficiency of speculative decoding is increasingly limited by the overhead of verifying long, low-confidence draft sequences. D-cut provides a training-free, hardware-aware solution that allows practitioners to maintain the benefits of speculative decoding without the performance degradation typically seen in heavily batched settings.