THE JEPA PARADOX IN LANGUAGE: THE GEOMETRY OF LINGUISTIC ALTERNATIVES

Joint-Embedding Predictive Architectures (JEPAs) have emerged as a powerful paradigm for self-supervised learning in continuous modalities like images, video, and audio. Unlike masked language models that predict discrete tokens, JEPAs learn by predicting the latent representation of a target signal from a masked context. While this approach is highly effective for visual data, it has not become a standard for text. This paper investigates this discrepancy, arguing that the failure of JEPAs in language is not a matter of model capacity, but a fundamental geometric mismatch between the objective function and the nature of linguistic data.

The authors identify "conditional concentration" as the critical requirement for successful JEPA training. In visual tasks, spatial continuity ensures that a masked image patch is highly constrained by its neighbors, meaning the target representation typically clusters around a single, meaningful point. In contrast, language is inherently ambiguous; a single textual context can support multiple valid, distinct completions. When a JEPA model is forced to predict a single latent vector for these diverse possibilities, it is mathematically incentivized to regress toward the conditional mean—a centroid that may not represent any coherent linguistic output. This process, termed "centroid degeneracy," forces the model to collapse distinct semantic possibilities into a single, low-rank representation space.

To validate this theory, the researchers developed T-JEPA, a text-based analogue to the established I-JEPA visual model. By tracking metrics such as mutual information, effective rank, and pairwise cosine similarity, they observed a consistent failure signature in T-JEPA that does not appear in I-JEPA. As training progresses, T-JEPA exhibits early information saturation followed by a rapid decline in effective rank and a spike in cosine similarity. This indicates that the model is actively discarding information to minimize its squared-error loss, effectively collapsing its internal representation to avoid the penalty of predicting multiple, incompatible targets. These results were consistent across five independent data seeds, confirming that the collapse is a structural consequence of the objective rather than a sampling artifact.