Identity Mappings in Deep Residual Networks

Deep Residual Networks (ResNets) rely on skip connections to enable the training of very deep architectures. This paper investigates the theoretical properties of these connections to understand why they succeed and how they can be optimized. The authors specifically examine the propagation of signals through the network during both forward and backward passes to determine the optimal configuration for residual building blocks.

The authors analyze the mathematical formulation of residual units, focusing on the conditions required for "direct" signal propagation. By treating the residual unit as a composition of an identity skip connection and an additive residual function, they derive that signal flow is most efficient when both the skip connection and the post-addition activation function are identity mappings. They test this hypothesis by conducting ablation studies on various shortcut modifications—such as scaling, gating, and 1x1 convolutions—and by re-arranging the order of Batch Normalization (BN) and ReLU activations to create a "pre-activation" structure.

The study reveals that any multiplicative manipulation of the skip connection (e.g., gating or scaling) hampers information flow and increases optimization difficulty, even if the model theoretically has higher representational capacity. Furthermore, the authors find that the conventional "post-activation" design (where ReLU follows the addition) is suboptimal. By moving the BN and ReLU layers to a "pre-activation" position—ensuring the path through the skip connection remains an identity mapping—the network becomes significantly easier to train. This design allows for the successful training of a 1001-layer ResNet on CIFAR-10, achieving lower error rates and better generalization than previous architectures.

This work provides a fundamental understanding of why ResNets work, shifting the focus from simply adding layers to ensuring that information can propagate cleanly through the entire depth of the network. The proposed pre-activation unit has become a standard architectural component in deep learning, enabling the development of much deeper and more accurate models across various computer vision tasks.