Chung-En Ho, Weiyu Sun, Cheng-Jhih Shih, He Li, Yong Liu, Yingyan (Celine) Lin
9 min
Diffusion large language models (DLLMs) generate text through iterative denoising. While multi-branch speculative decoding can speed up this process by verifying multiple draft branches in a single pass, it creates a new bottleneck: evaluating all branches together increases the computational cost of each forward pass. This paper introduces SpecFold, an algorithm-system co-design that targets this inefficiency by exploiting multi-branch computational redundancy. The authors observe that draft branches are often highly similar to their parents, inheriting most token representations. SpecFold identifies these similarities through a token-level residual gate and selectively reuses parent computation, only recomputing divergent positions.
At the algorithmic level, SpecFold performs per-token relative residual checks between child and parent drafts. If the residual is below a threshold, the model folds the child's attention and feed-forward network (FFN) computation into the parent's, essentially skipping redundant work. Critically, SpecFold preserves hidden-state diversity across branches by maintaining individual residual streams and correcting attention outputs. At the system level, the authors implement this logic using optimized Triton kernels, which compact and pack recomputed positions across branches, allowing for efficient sparse execution on GPUs without materializing redundant branch tensors.
SpecFold is orthogonal to existing temporal caching and compatible with standard speculation strategies. Across two DLLM families (Fast-dLLM-v2 and Nemotron-Labs-Diffusion), five models, and five benchmarks, SpecFold achieves up to 1.64x throughput over the Spiffy baseline and up to 1.99x over vanilla decoding, all while maintaining task performance comparable to dense verification methods.
Multi-branch speculative decoding is limited by the overhead of verifying many draft branches simultaneously. By converting this dense evaluation into a sparse one, SpecFold allows researchers to utilize larger draft budgets (i.e., more potential look-ahead) without the proportional latency penalties that previously made high-budget speculation impractical.
Diffusion large language models (DLLMs) generate text through iterative block denoising, and multi-branch speculative decoding accelerates this process by verifying a main branch together with multiple draft branches in a single forward pass. While prior DLLM acceleration methods primarily exploit temporal redundancy across denoising steps, we identify a complementary redundancy axis within each speculative verification step: multi-branch computational redundancy. During speculative verification, draft branches inherit most tokens from their parents while unmasking a small set of additional positions, causing large portions of hidden states to remain highly similar across branches. We propose SpecFold, an algorithm-system co-design that exploits this multi-branch redundancy to reduce the cost of multi-branch speculative verification. Algorithmically, SpecFold performs token-level residual gating and selectively reuses parent computation through folded attention and FFN while preserving residual hidden states. Systemically, a Triton kernel implementation translates this fine-grained reuse into end-to-end throughput gains through efficient sparse multi-branch execution. SpecFold is orthogonal to temporal caching and compatible with existing DLLM speculation strategies. Across two DLLM families, five models, and five standard benchmarks, SpecFold achieves up to 1.64x throughput over Spiffy and up to 1.99x over vanilla decoding, while maintaining comparable task performance.
Alex: Spiffy changes how many model passes you need. SpecFold reduces the cost of checking the branches together. It uses the same draft construction, but avoids recalculating internal representations that are already very similar in a draft’s parent branch.
Sam: Why should different drafts have similar internal representations? Different text can change the interpretation of a whole sentence.
Alex: Each child draft inherits almost all its parent’s tokens and reveals only a small additional set. The authors measured the resulting hidden states: each token’s internal numerical representation. In their profile of one Fast-dLLM model, seventy-two percent of measured parent–child comparisons met their strict similarity threshold.
Sam: Similar is not identical, though. How does the method decide which differences it can afford to ignore?
Alex: At every layer, it measures the parent–child difference at each position, relative to the size of the parent representation. Small differences trigger reuse, which the authors call folding. Larger differences trigger fresh computation. The main branch is always computed afresh.
Sam: Give us a concrete case. Where does a newly revealed token actually change the calculations?
Alex: The authors compare two versions of a thirty-two-position block. The parent has just over half its positions revealed; the child reveals one more. That edited position stays different through the layers, while differences elsewhere concentrate in masked and recently revealed positions.
Sam: So the change creates a ripple, but not an equally strong ripple everywhere. That argues for decisions position by position, rather than treating the whole draft as redundant.
Alex: That is what their analysis supports. Already committed positions tend to remain similar. The gate can therefore reuse calculations at some positions while recomputing others, and reassess those choices at the next layer.
Sam: There is an obvious obstacle here. If every position can influence every other position, how can copying parent calculations still respect the child’s changed text?
Alex: That interaction happens through attention, the calculation that lets positions influence one another. At folded positions, SpecFold reuses the parent’s attention state, then corrects it for the child’s newly computed contributions. It does not simply copy the parent’s final attention output unchanged.
Sam: Does that make the whole operation exact, or is the correction exact only within the reused representation?
Alex: The narrower claim. The correction respects the child’s keys and values, the representations attention reads, given the folded inputs. But folding already approximates some projections and reuses other layer outputs. SpecFold is not a guarantee of identical predictions to dense verification.
Sam: Then what prevents small differences from disappearing as the layers accumulate?
Alex: Each branch keeps its own residual stream, its running representation through the model. Reused layer outputs are added to that branch’s own state, rather than replacing it with the parent’s. If accumulated differences cross the threshold later, that position resumes fresh computation.
Sam: Skipping calculations on paper is one thing. Does the implementation actually turn that into less time on the hardware?
Alex: The authors wrote specialized GPU routines using Triton, a language for implementing GPU calculations. Those routines pack positions needing recomputation together and track where folded positions inherit their data. This lets the expensive position-wise operations run on the smaller packed workload.
Sam: How broad is the test? I would be cautious if the gain came from one model on one task.
Alex: They evaluated five models across five benchmarks. The models came from two diffusion-model families, and the tasks covered mathematics, code generation, and instruction following. Most experiments used a single NVIDIA H100 GPU, with additional hardware checks described in the appendix.
Sam: Were the two accelerators given comparable drafts, or could better guessing explain the speed difference?
Alex: They used the same calibrated draft graph for each model. That makes the main comparison about execution, not a better drafting policy. They measured both the number of model evaluations and end-to-end tokens per second, alongside task accuracy.
Sam: Now the qualification on the headline: does the gain hold equally well as the models get larger?
Alex: No. The largest gains over Spiffy occurred in the smaller Fast-dLLM model. In the largest Nemotron model tested, the maximum improvement over Spiffy was about seven percent. So the practical benefit depends substantially on the model and comparison you care about.
Sam: And “comparable accuracy” can conceal meaningful changes. What should a reader check before using this?
Alex: Check the individual task scores, not just the authors’ overall characterization. Scores move in both directions, and some code-generation results fall below ordinary decoding. The method also has a reuse threshold: more aggressive folding eventually reduces accuracy.
Sam: So this is an adjustable approximation, rather than a free speedup with unchanged outputs.
Alex: Yes. The authors use a conservative threshold of zero point one across the main evaluation, without tuning it for each task. Their threshold sweep finds a region of broadly stable accuracy, followed by deterioration. My reading is credible engineering evidence within these settings, not universal performance preservation.
Sam: Who should read the full document, and where would you send them first?
Alex: Researchers building diffusion-model inference systems should start with the folding method and Triton implementation. Then read the main results by model scale and the recompute-threshold sweep. Those sections show both the execution opportunity and the accuracy trade-off you would need to validate.
Sam: And for researchers outside that implementation work, what is the line to remember?
Alex: Skipping model calls helps only if checking the alternatives does not eat the savings. SpecFold’s contribution is to share work across those alternatives, not just predict more of them.
Sam: Keep the cost of the check in view.