Relation Geometry in Semantic Space of Language Models

This paper investigates how language models (LMs) represent semantic relations—such as hypernymy, synonymy, and antonymy—within their internal vector spaces. The authors move beyond the common focus on hypernymy to examine whether words related to a target word (relata) cluster into distinct, identifiable regions. They test this across three model families: causal, masked, and diffusion LMs. The study evaluates these models by probing for the geometric properties of relations, specifically testing for symmetry, asymmetry, and transitivity, while also analyzing the relative importance of lexical versus contextual information in forming these semantic representations.

The researchers found that asymmetric relations (e.g., hypernymy) are better encoded in the semantic space than symmetric relations (e.g., synonymy), as the former tend to occupy more distinct regions. The study also reveals a clear divergence in how different model architectures process semantic information: causal models lean more on the lexical identity of tokens, whereas masked and diffusion models rely more on the surrounding context. These results challenge the traditional distributional semantic view that word meaning can be learned purely from distribution, suggesting that the geometric representation of relations is inconsistent across different types of semantic connections.

This work provides empirical evidence that language models do not possess a uniform "understanding" of semantic relations. By showing that some relations are geometrically better defined than others, the authors highlight a potential limitation in how current models capture the nuances of language. This suggests that future research into model interpretability and reasoning capabilities must account for the fact that semantic knowledge is not equally accessible or structured across all relation types or model architectures.