AngelSpec: Towards Real-World High Performance Inference with Speculative Decoding

Speculative decoding is a powerful technique to speed up LLM inference, but its effectiveness is often limited by the choice of the drafting mechanism. Standard approaches typically use a single, universal drafter for all workloads. This paper investigates whether tailoring the drafting structure and training data to specific task distributions—such as open-ended chat versus structured code generation—can significantly improve inference throughput.

AngelSpec introduces a unified training framework that addresses workload heterogeneity at three levels:

1. Training: Instead of a one-size-fits-all model, the authors co-specialize drafters. They train an MTP-based drafter on diverse conversational data for high-entropy tasks and a block-diffusion drafter on code and mathematics data for predictable continuations.
2. Architecture: They propose DFly, a block-diffusion framework that improves upon existing methods by using a hybrid target-conditioning backbone and a predecessor-conditioned autoregressive head. This allows for better target-feature utilization and improved intra-block dependency modeling.
3. Inference: They implement a dynamic verification strategy called D-cut. This treats verification compute as a shared resource, reallocating it toward high-confidence prefixes across concurrent requests based on a profiled cost model, allowing the system to adapt to varying hardware and load conditions.

By specializing the drafter to the workload, the authors achieve substantial performance improvements. On the Hy3-A21B model, the DFly architecture increased the average accepted token length by approximately 30%. In practical serving scenarios, this resulted in a 1.98–2.40x speedup over standard autoregressive decoding and outperformed the baseline DFlash method by 10.5–11.8% in throughput across various concurrency levels.