ERROR HIGHWAYS: SCALING PREDICTIVE CODING TO VERY DEEP NETWORKS

Predictive Coding Networks (PCNs) are a biologically plausible alternative to backpropagation, but they struggle to scale to deep architectures. The authors investigate why PCNs fail at depth, identifying that the learning signal (the error) attenuates exponentially as it propagates backward through layers. This study introduces Highway Error Propagation (HEP) to solve this vanishing signal problem without sacrificing the local learning rules that make PCNs biologically attractive.

The authors analyze the depth problem in PCNs through two lenses: steady-state dynamics (where error signals are bounded by the product of layer Jacobians) and transient dynamics (where the signal decays by a factor of the activity step size at each hop). To address this, they propose HEP, which augments the network structure with feedback matrices that couple hidden states directly to the clamped output error. This creates a 'highway' that injects the supervisory signal into hidden layers in a single step, ensuring the correction magnitude is independent of network depth. Crucially, this modification preserves the local synaptic update rule, maintaining the model's biological plausibility.

Experiments on MNIST and Fashion-MNIST demonstrate that HEP effectively trains Multi-Layer Perceptrons (MLPs) with up to 128 layers. While vanilla PCNs and other variants like μPC struggle to propagate error signals to early layers within a reasonable inference budget, HEP provides a robust learning signal across the entire depth of the network. The authors show that this architecture allows for stable training even when the forward path is constrained to pass through every layer, forcing the network to learn meaningful internal representations rather than relying on shortcut pathways.

This work bridges a significant gap between biologically inspired learning models and the deep architectures required for complex machine learning tasks. By providing a mechanism to scale PCNs to 128 layers, the authors demonstrate that local learning rules can be competitive with backpropagation in deep settings, offering a promising path toward more efficient and neurobiologically grounded artificial intelligence.