How Transparent is DiffusionGemma?

As AI models move toward non-autoregressive architectures like text diffusion, a critical question arises: does the shift from sequential token generation to continuous latent space refinement obscure the model's reasoning process? This paper investigates the transparency of DiffusionGemma, a text diffusion model, by evaluating its "opaque serial depth"—the amount of computation performed without passing through an interpretable state. The authors decompose transparency into variable transparency (understanding intermediate computational snapshots) and algorithmic transparency (reconstructing the reasoning process).

Initially, DiffusionGemma appears significantly less transparent than autoregressive models like Gemma 4, with an opaque serial depth 28.6 times higher. However, the authors demonstrate that the information flowing between denoising steps can be mapped through an interpretable token bottleneck. By restricting this flow to a small set of high-probability tokens, they maintain downstream performance while reducing the opaque serial depth to just 1.1 times that of Gemma 4. This suggests that the model's intermediate states are largely interpretable, functioning similarly to a chain of thought.

Beyond variable transparency, the authors explore algorithmic transparency, noting that diffusion models can perform distributed reasoning where tokens influence each other non-chronologically. Case studies reveal novel phenomena such as token smearing and intermediate-context reasoning, which differ from standard autoregressive behavior. Despite these architectural differences, the authors find that DiffusionGemma remains as monitorable as autoregressive models, meaning external monitors can effectively extract information from its outputs to assess safety and alignment.