Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun
5 min
In deep neural networks, increasing the number of layers is generally expected to improve performance by allowing the model to learn more complex, hierarchical features. However, researchers observed a "degradation problem": as networks become significantly deep, accuracy saturates and then rapidly declines. Crucially, this is not caused by overfitting; adding more layers to a sufficiently deep model actually results in higher training error. This indicates that deeper architectures are harder to optimize using standard solvers, even when they should theoretically be able to perform at least as well as their shallower counterparts.
To address this, the authors introduce a "residual learning" framework. Instead of forcing stacked layers to learn an underlying mapping directly, the network is reformulated to learn the difference between the input and the desired output, known as the residual mapping . The original mapping is then reconstructed as .
This is implemented using "shortcut connections" that skip one or more layers. These connections perform identity mapping, meaning they simply add the input to the output of the stacked layers. This approach has two major advantages:
The authors evaluated their residual networks (ResNets) on the ImageNet dataset, successfully training models with up to 152 layers—eight times deeper than previous state-of-the-art VGG nets while maintaining lower complexity.
The results were transformative:
Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers---8x deeper than VGG nets but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.
Sam: So the deeper network can't perform *worse* than a shallower one, because it always has the option to ignore the extra layers entirely.
Alex: That's the logic. And it worked. By learning these residuals — these small corrections — they were able to train networks with over one hundred layers, which had previously been considered out of reach.
Sam: What keeps the signals stable as they travel through all those layers? Doesn't passing information through a hundred stages risk it becoming distorted?
Alex: That's a real concern, and they address it with a technique called Batch Normalization. Think of it like a shock absorber on a car. As information moves through the network, values can drift — getting too large or too small — which makes training unstable. Batch Normalization continuously recalibrates those values to keep them in a useful range.
Sam: So the shortcut connections solve the "getting stuck" problem, and Batch Normalization solves the "signal going haywire" problem.
Alex: Exactly. And for the very deepest versions of the network, they also use what's called a "bottleneck" design. Imagine a three-stage funnel. The first stage compresses the data down to a smaller, more manageable size. The middle stage does the hard analytical work on that compressed version. Then the third stage expands it back out to the original dimensions. It's a way of doing the difficult computation without having to process the full volume of data at every step.
Sam: And putting all of this together — the shortcut connections, the stable training, the bottleneck design — that's what allowed them to win the 2015 ImageNet competition?
Alex: It was a significant result. ImageNet is one of the most widely used benchmarks for image recognition, and the depth their approach unlocked allowed the model to detect features at many different levels of detail simultaneously — fine textures, broad shapes, and everything in between.
Sam: Is this still how modern systems are built?
Alex: The residual connection has become a standard building block in computer vision and well beyond it. The core insight — that it's easier to learn a small correction than a full transformation — turned out to be broadly useful. By solving the degradation problem, the paper showed that depth is a genuine tool, not just a theoretical advantage, provided you structure the learning correctly.
Sam: It's a surprisingly simple idea when you put it that way. The network doesn't need to be forced to do more — it just needs the freedom to do less when less is the right answer.
Alex: That's a good way to put it. Sometimes the most useful thing a system can do is get out of its own way. Thanks for listening to ResearchPod.