The Anatomy of Implicit Bias: Information Allocation in Neural Network Training

Traditional research on implicit bias in neural networks focuses on the final solution—analyzing the geometry of the loss landscape, such as sharpness or flatness, to explain generalization. This paper shifts the focus to the training process itself, proposing an 'information allocation' perspective. The authors argue that training is not merely a process of loss reduction, but a mechanism that writes error signals into the model in a structured way. By tracking how these signals are distributed across parameter paths (e.g., weights vs. biases), coordinate channels, and sample regions, researchers can observe the formation of implicit bias in real-time.

To operationalize this view, the authors introduce a set of observable diagnostics: gradient demand, actual update injection, coordinate gain (via exponential moving averages), channel-level update ratios, and sample-wise loss distributions. A key innovation is the 'collapse-persistence analysis.' By comparing models at the same training loss, the authors can distinguish between factors that merely change the global speed of progress (like the learning rate) and those that fundamentally alter the internal allocation of training signals (like the preconditioning exponent $p$).

Through controlled experiments, the paper maps various training factors to their specific allocation signatures. The learning rate is identified as a 'progress-dominant' source, as its effects largely disappear when training progress is matched. In contrast, the preconditioning exponent $p$ acts as a 'coordinate-gain' source, directly influencing the relative importance of different channels throughout training. Other factors, including batch size, optimizer memory, and model architecture, are shown to impose unique constraints on how information is absorbed. This framework provides a mechanism-level explanation for why different hyperparameter choices lead to divergent generalization behaviors, even when final training losses are identical.