Adversarial Diffusion Across Modalities: A Fusion Survey of Attacks, Defenses, and Evaluation for Text, Vision, and Vision-Language Models

Research Context and Motivation

Adversarial evaluation, or red teaming, has become a critical component of AI safety. While traditional methods like gradient-based suffix optimization or genetic algorithms have dominated, denoising diffusion models—which generate data by iteratively removing noise—are emerging as a powerful new tool for crafting adversarial inputs. Despite their potential, research in this area is currently siloed across four disconnected tracks: text/LLM attacks, image classifier attacks, vision-language model (VLM) jailbreaking, and diffusion-based input purification defenses.

This paper performs a meta-research fusion, integrating these disparate fields into a single conceptual framework. The authors argue that diffusion models offer unique advantages for attackers, such as bidirectional conditioning and principled sampling diversity, which traditional autoregressive generators lack. By unifying the vocabulary and evaluation criteria, the authors aim to reduce duplicated effort and provide a roadmap for future research.

Taxonomy and Evaluation Framework

The authors propose a six-class taxonomy to categorize how diffusion models function within an adversarial pipeline. These roles range from using diffusion as a trained generator or a frozen model with latent perturbation to employing it as a black-box renderer or a victim model itself. To standardize how these attacks are measured, the paper introduces a five-dimension evaluation framework: attack success rate, transferability, query budget, perplexity, and defense-evasion. This framework is applied uniformly across modalities to highlight the current gaps in the literature, particularly the difficulty diffusion-based attacks face when targeting closed-frontier models.

Critical Analysis and Research Agenda

By cataloging fifty relevant papers, the authors identify five recurring weaknesses in the current text-side adversarial literature, such as inconsistent baseline comparisons and insufficient target-model coverage. The review concludes with a forward-looking research agenda, offering concrete experimental designs to address open questions. The authors emphasize that as LLMs are deployed in high-stakes environments, understanding these diffusion-based mechanisms is essential for building robust defenses.