Energy-Based Transformers as Predictors of Reading Difficulty

Computational psycholinguists have long used transformer language models to predict human reading times, primarily relying on surprisal (the negative log-probability of a token) and attention entropy (a measure of memory access patterns). However, these two metrics often capture complementary aspects of processing load, and neither consistently accounts for all observed human reading behaviors. This paper investigates whether energy-based transformers—a class of models that treat inference as a gradient descent process on an energy landscape—can provide a single, unified predictor for reading difficulty.

The authors utilize NRGPT, a transformer architecture that reformulates the forward pass as a series of descent steps on a scalar energy function. This energy function is theoretically linked to dense associative memory and Hopfield networks. The researchers evaluated this energy measure against three standard reading-time datasets (Natural Stories, UCL eye-tracking, and UCL self-paced reading) and a controlled experiment on relative clause processing. They specifically compared the energy measure's performance against surprisal and attention entropy to see if it could subsume the predictive power of both.

The study demonstrates that energy is a robust predictor of human reading times, consistently providing a better fit than surprisal alone across all tested corpora. In the controlled relative clause experiment, the energy measure successfully captured the well-known object/subject asymmetry at both the embedded noun phrase and the embedded verb. Crucially, while surprisal failed to predict the difficulty at the embedded verb and attention entropy failed at the noun phrase, the energy measure successfully predicted difficulty at both locations. This suggests that energy acts as a unified metric, capturing both predictability-driven and integration-driven processing costs.