Behaviour Is an Incomplete Measure of Reasoning Development: Cross-surface pre-arrival accessibility and the limits of developmental inference in a recurrent-depth reasoner

How do we determine when a neural network has actually acquired a capability? Researchers commonly rely on behavioural accuracy thresholds, final model checkpoints, or what a linear probe can decode from hidden states. This paper investigates whether these three quantities identify the same underlying developmental event.

To study this, the author examines a 30M-parameter recurrent-depth transformer trained in a closed, oracle-defined relational world. The model is trained on two different surfaces—one symbolic and one verbal—using an identical curriculum and a strict held-out accuracy criterion of 0.95. By holding the training and inference axes separate, the study uses dense behavioural trajectories, preregistered hidden-state probes, and explicit untrained controls to track when and how relational reasoning capabilities develop.

When holding the architecture, world, curriculum, and criterion constant while changing only the bundled training surface, the developmental trajectories differ radically. Under a frozen acquisition criterion, achieving three-hop competence required 70 logical epochs on the symbolic surface compared to 13,055 epochs on the verbal surface—a massive 186.5-fold contrast. Furthermore, the verbal realization underwent a long, near-floor behavioural grind at three-hop depth, where held-out four-hop accuracy remained close to zero before suddenly clearing four-hop competence in just 8 logical epochs.

This demonstrates that a single behavioural threshold collapses rich developmental structure into a single point. Moreover, the absence of high-held-out behaviour during a long grind does not prove that the model's internal representations are static. Behaviour measures expression rather than availability.

To look beyond overt behaviour, the author fits linear probes to hidden states at preregistered positions preceding the answer slot, focusing on items the model has not yet answered correctly. On the verbal surface, a linear probe successfully recovered future-answer identity before behavioural arrival at an accuracy above uniform chance, frequency baselines, and untrained zero-step controls. Crucially, this pre-arrival accessibility also survived a complete surface change to the symbolic representation at both upstream structural positions and readout comparators.

However, richer observation still encounters hard identification limits. The natural attempt to track this accessibility across training failed because probe eligibility is defined by behavioural arrival: items leave the measured population as they improve. Consequently, the measured population changes with the measurand, proving that neither behaviour nor decoder accessibility alone can fully identify the computation training acquired.