From Self-Attention to Connection Laplacian: A Unified Operator View of Transformers

This paper provides a geometric reinterpretation of the Transformer's self-attention mechanism. By viewing a sequence of tokens as a vector field over a graph, the authors demonstrate that self-attention is not merely a scalar weighting of token similarities, but a connection-walk operator. In this framework, the attention matrix acts as a walk weight, while the value-output projections function as linear transports that map feature vectors between token positions.

The authors prove that single-head attention is a connection propagation step with constant transport. More importantly, they show that multi-head attention (MHA) is a single edge-dependent connection walk. In this view, the effective transport for any given edge is an attention-weighted average of the individual head-wise transport matrices. This allows researchers to analyze MHA as a single operator rather than a collection of independent heads, providing a bridge to classical geometric operators like the connection Laplacian.

The paper introduces operator-level diagnostics to measure the geometric properties of trained Transformers. By analyzing models ranging from 124M to 8B parameters, the authors find that as depth increases, attention graphs converge toward stable geometric operators. Furthermore, they observe that learned transports self-organize into approximate scaled isometries, a phenomenon that becomes more pronounced as model scale increases. These findings suggest that Transformers implicitly learn to preserve geometric structure, providing a new lens for understanding why these models scale effectively.