Evaluating the Diversity of AI-Generated Content with Diversity Profiles

Diversity is a fundamental criterion for evaluating generative artificial intelligence systems across modalities such as images, natural language, graphs, and molecules. Generative models must not only produce high-quality samples, but also avoid mode collapse and ensure broad coverage of the target domain. Historically, evaluation has relied on reducing diversity to a single scalar score by embedding generated items, computing pairwise distances, and aggregating them into one number. However, this scalar reduction is inherently ambiguous and under-specified.

Different scalar metrics encode distinct inductive biases and can lead to contradictory rankings of the same sample sets. This happens because widely used diversity metrics satisfy different subsets of theoretical axioms and operate under different parameter choices, such as thresholds, scales, exponents, or orders. Furthermore, high-dimensional representation spaces common in modern AI suffer from distance concentration, where pairwise distances cluster together. Consequently, relying on a single absolute score can hide critical trade-offs and render evaluations highly sensitive to arbitrary hyperparameter choices.

To understand the flaws of scalar metrics, researchers can examine them through an axiomatic lens and through the lens of high-dimensional geometry. An axiomatic analysis evaluates metrics against four key desiderata: size monotonicity, the twin property, distance monotonicity, and continuity. Representative metrics—such as average distance, energy scores, packing metrics, the Vendi score, and metric-space magnitude—fail to satisfy all four properties simultaneously. For instance, some metrics violate the twin property, meaning that adding duplicate samples can improperly distort the score.

In addition, high-dimensional representation spaces introduce significant geometric challenges. As dimensionality increases, the contrast between the maximum and minimum pairwise distances among random samples diminishes. This distance concentration weakens locality and nearest-neighbor assumptions. Because empirical distance distributions vary widely across image, text, and molecular modalities, a fixed numerical threshold or scale has no universal meaning. Large regions of a metric's parameter space can easily become saturated or uninformative, making single-number evaluations fragile and representation-dependent.