Anisotropy Decides Cosine vs. Rank Metrics for Text Embeddings

Researchers have long debated whether cosine similarity is the optimal metric for comparing text embeddings, with various studies suggesting alternatives without establishing a clear rule. This paper investigates why these disagreements exist by testing nineteen parameter-free similarity metrics across nineteen diverse encoders—ranging from small sentence transformers to seven-billion-parameter language models—and seven standard datasets. The study aims to identify the geometric conditions that dictate when a non-cosine metric provides a performance advantage.

The study reveals a clean, geometric divide. When an encoder spreads its variance evenly across the embedding space (isotropic), cosine similarity is the best choice, and no other parameter-free metric offers a meaningful improvement. Conversely, when an encoder is anisotropic—meaning its variance is concentrated into a few dominant directions—rank-based and L1-type metrics (such as Manhattan or Canberra distance) consistently outperform cosine. This effect is not driven by model size, pooling methods, or training objectives, but rather by the shape of the embedding space itself. The authors demonstrate this by showing that even models trained with a cosine objective can be anisotropic, and vice versa.

The authors identify a single diagnostic number, rogue-dimension dominance, which measures the fraction of total variance held by the single most dominant dimension. This metric predicts the performance gain of alternative metrics with high correlation. To confirm that this concentration of variance is the cause of cosine's failure, the researchers projected out the dominant directions from the embedding space. This process allowed cosine similarity to recover its performance, while the advantage of alternative metrics nearly vanished. This confirms that when a few dimensions dominate, they effectively "swamp" the cosine calculation, whereas rank-based and L1-type metrics are more robust to these outliers.