ResearchPod Summary
Discrete diffusion models, such as those used for text or image generation, often attempt to speed up sampling by writing multiple token positions simultaneously (parallel decoding). This paper investigates whether these parallel steps can accurately match the training data distribution, or if the choice of which tokens to write together introduces an unavoidable bias.
Using ScanAndAdd—a synthetic task where the joint distribution, entropies, and conditional probabilities are all computable—the authors analyze the behavior of common discrete diffusion families: absorbing-state masked diffusion, remasking, and uniform-state samplers. They derive theorems showing that any parallel step (writing two or more positions) that involves dependent tokens will necessarily deviate from the true data distribution. They then compare these theoretical limits against a trained Transformer model to measure the actual distortion, using total variation distance (TVD) against a known sampling-noise floor.
This work provides a fundamental information-theoretic limit for parallel decoding in discrete diffusion. It suggests that the common practice of using model confidence to order parallel writes is inherently flawed because confidence scores do not capture the dependency structure of the data. Researchers should be cautious when using parallel decoding, as high-quality samples may mask significant underlying distributional shifts that are not captured by standard metrics like accuracy or perplexity.
AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.