REASONING AS ATTRACTOR DYNAMICS: LATENT MEMORY RETRIEVAL VIA GIBBS-WEIGHTED ENERGY MINIMIZATION

Traditional LLM inference treats generation as a greedy, autoregressive process. This paper challenges this view, proposing that LLMs function as Dense Associative Memories. The author investigates whether mathematical reasoning can be better modeled as a dynamic retrieval process where the model settles into stable 'attractor basins' rather than simply predicting the next token.

The author models the LLM's energy landscape by treating reasoning trajectories as particles. Correct reasoning chains are hypothesized to reside in 'flat minima'—wide, stable basins—while hallucinations correspond to 'sharp minima' or unstable local traps. To navigate this landscape, the paper introduces a Gibbs-Weighted Retrieval mechanism. This method samples multiple reasoning paths, calculates their 'energy' via spectral entropy (length-normalized negative log-likelihood), and re-weights them using an inverse-square law. This effectively suppresses high-entropy, unstable paths and amplifies robust, low-entropy solutions.

By applying this physics-inspired retrieval operator to the Microsoft Phi-3.5-mini model, the author observed a 5.38% improvement in accuracy on the GSM8K benchmark (from 84.7% to 90.1%). The results suggest that 'System 2' reasoning—often associated with increased test-time compute—is essentially a process of thermodynamic relaxation. The Gibbs-weighted approach acts as a critical filter, allowing the system to undergo a phase transition that discards hallucinatory 'noise' and converges on the dominant attractor basin representing the correct solution.

This work provides a theoretical bridge between modern transformer architectures and energy-based models like Hopfield networks. It suggests that the reasoning capabilities of LLMs are not just emergent properties of scale, but are grounded in the geometric structure of the model's internal energy landscape. This perspective offers a new framework for improving model reliability by focusing on the stability of reasoning paths rather than just the probability of individual tokens.