Infrared Organization and Critical Cognitive Field Formation in Transformer Dynamics

This paper investigates the large-scale collective dynamics of Transformer-based language models through the lens of statistical physics. Specifically, it asks whether the emergent capabilities of these models—such as in-context learning and long-range memory—can be explained by the organization of collective relaxation modes, as predicted by Cognitive Field Theory (CFT).

The author analyzes the Pythia family of autoregressive language models, which provide a controlled environment with consistent training protocols across different scales. By extracting the Jacobian matrices of hidden-state mappings throughout training, the study calculates the relaxation spectra of the models. These spectra are used to derive key collective observables, including the time-scale density of states (TDOS), memory self-energy, the cognitive forgetting gap, and the memory kernel, without introducing additional phenomenological parameters.

The research reveals that Transformer training is characterized by a progressive accumulation of slow relaxation modes in the infrared sector of the spectrum. This reorganization leads to a nearly flat infrared TDOS and a universal, scale-free memory kernel that follows a 1/t decay, independent of network depth. A critical observation is that the memory self-energy does not grow monotonically; instead, it reaches a transient maximum early in training. This peak corresponds to the point where the cognitive forgetting gap is minimized and collective susceptibility is maximized, suggesting that Transformer learning involves a transient critical formation of a macroscopic cognitive field before the model settles into a stable, near-critical operating regime.

By framing Transformers as nonequilibrium many-body systems rather than just static computational architectures, this work provides a physical foundation for understanding how emergent intelligence arises. The reproducibility of these dynamical patterns across different model scales and depths suggests that infrared slow-mode organization is a universal principle of Transformer dynamics, offering a new quantitative framework for evaluating model development and stability.