An Analysis of Residual-Stream Geometry Across Transformer Depth

Research Approach

This paper introduces a transition-centered geometric framework to analyze how transformer residual streams evolve across layers. Rather than treating each layer as a static snapshot, the authors measure the transformation of the token cloud between consecutive layers. They utilize two primary metrics: relative displacement, which quantifies the magnitude of change between layers, and orthogonal Procrustes analysis, which decomposes this change into a rigid rotation and a non-rigid residual. By applying these metrics to six instruction-tuned models (Qwen, Gemma, and StableLM) across code generation and cross-lingual translation tasks, the researchers identify consistent patterns in how information is processed through the depth of the network.

Key Findings

The analysis reveals that the geometry of the residual stream is highly structured by depth. Relative displacement follows a consistent, model-specific pattern: it is typically larger in the early and late layers, with a quieter middle third. This "middle-layer slowdown" is robust across different tasks. Interestingly, while the global rotation magnitude remains nearly constant across all layers, the non-rigid residual mismatch is highly depth-modulated, consistently peaking at the final transition. Furthermore, the researchers find that while early and middle layers are largely invariant to the specific target language, the final layers show increased displacement and residual for non-English targets, suggesting that language-specific adjustments are concentrated at the end of the model stack.

This work shifts the focus from what individual layers represent to how the entire token cloud is geometrically transformed. By demonstrating that depth curves are model-dependent but condition-stable, the authors provide a new way to characterize the "rhythm" of computation in transformers. This descriptive framework helps bridge the gap between high-level performance metrics and low-level mechanistic interpretability, offering a standardized way to compare how different architectures allocate their representational updates across depth.