Scalable Physics-Inspired Transformers for Spin Glasses

How can machine learning models for spin glasses overcome the dual challenges of poor scalability and high computational costs? Existing variational autoregressive networks (VANs) often fail to benefit from increased model size and struggle with the rugged energy landscapes characteristic of frustrated systems like the Sherrington-Kirkpatrick (SK) and Edwards-Anderson (EA) models.

The authors propose FlashVAN, a transformer-based generative model designed specifically for spin systems. The architecture incorporates three key innovations: (1) a physics-inspired sparse attention mechanism that restricts the receptive field to the local interaction topology of the lattice; (2) spin-tailored positional embeddings that prevent the instabilities often seen in standard additive embeddings; and (3) the integration of FlashAttention-2 and key-value (KV) caching to accelerate parallel ancestral sampling. The training process is further enhanced by a temperature-annealing scheme combined with local Monte Carlo refinement and self-distillation to improve exploration of low-energy states.

FlashVAN significantly outperforms existing neural-network methods in both computational efficiency and system size. It achieves up to a 100x speedup compared to vanilla VANs, allowing for the simulation of systems with up to 4,096 spins on a single NVIDIA H100 GPU. The model accurately resolves thermodynamic observables, including free energies, Boltzmann distributions, and overlap statistics, across a wide temperature range. Furthermore, it successfully identifies ground states in models where previous reinforcement-learning approaches have faced difficulties, establishing a new benchmark for scalable neural-network simulations in statistical mechanics.