Surprisal Theory is Tautological (without Rational Grounding)

Surprisal theory, a cornerstone of modern psycholinguistics, posits that the processing difficulty of a linguistic unit is an affine function of its surprisal—the negative log-probability of that unit given its context. The author demonstrates that this framework is currently tautological. Because the theory does not specify which language model (the probability distribution) should be used to calculate surprisal, a researcher can mathematically construct a language model to fit any arbitrary pattern of processing difficulty. Consequently, the theory as currently practiced makes no falsifiable predictions.

For two decades, the field relied on the implicit 'corpus assumption': that the relevant language model is the one that best approximates the distribution of the training corpus. This provided a falsifiable constraint—the 'scaling implication'—which suggested that as language models improved their fit to a corpus, they should better predict human processing difficulty. However, recent empirical evidence shows that larger, better-fitting models often become worse predictors of human behavior. This breakdown suggests that the corpus assumption is incorrect and that the field lacks a principled way to select a language model that reflects human cognition.

To move beyond this tautology, the author argues for a 'rationalist intervention' analogous to the propensity interpretation of fitness in evolutionary biology. Instead of relying on corpus-derived models, researchers must ground the choice of language model in independent, non-behavioral constraints of the human cognitive system. By defining the language model based on factors like memory limitations, processing goals, or architectural constraints of the brain, the theory could regain its status as a falsifiable scientific framework.