gradient vanishing

The vanishing gradient problem is what happens when the error signal used to train a network shrinks exponentially as it propagates backward through many layers, leaving the earliest layers receiving a gradient so close to zero that they effectively stop learning. Sepp Hochreiter identified and analyzed the problem in his 1991 diploma thesis under Jürgen Schmidhuber at the Technical University of Munich, and Yoshua Bengio, Patrice Simard, and Paolo Frasconi formalized it in English in their widely cited 1994 paper "Learning Long-Term Dependencies with Gradient Descent is Difficult." The mechanism is simple arithmetic: the sigmoid activation function's derivative never exceeds 0.25, so backpropagating through 10 sigmoid layers multiplies that many factors of at most 0.25 together, shrinking the gradient by a factor of roughly 0.25 to the 10th power — about one part in a million — before it ever reaches the earliest layers.

Recurrent networks hit this problem even harder than deep feedforward networks do, because the same weight matrix gets multiplied against itself once per time step. Bengio, Simard, and Frasconi's 1994 analysis specifically targeted this recurrent case: an RNN trying to learn a dependency 50 time steps back has to backpropagate through 50 repeated applications of the identical recurrent weight matrix, and if that matrix's dominant eigenvalue is below 1, the gradient contracts at every single step in a way that compounds far faster than it does in a feedforward network with 50 distinct layers — which is exactly why plain RNNs could learn short-range patterns but consistently failed to learn anything depending on information from dozens of steps earlier.

LSTM's gating mechanism was invented specifically to give the gradient an escape route from this multiplicative chain. Hochreiter and Schmidhuber's 1997 LSTM paper added a cell state updated through addition rather than repeated multiplication — the forget gate decides how much of the previous cell state to keep, and that kept portion passes forward additively, creating what their paper calls a "constant error carousel" where the gradient can flow across many time steps without being forced through a chain of shrinking multiplicative factors. This additive path is the specific architectural feature that let LSTMs learn dependencies spanning hundreds of time steps where plain RNNs failed within a few dozen.

ReLU's defining property is that its derivative is exactly 1 for any positive input, never shrinking the signal at all, which is why it replaced sigmoid as the default activation in deep networks. Where sigmoid's derivative maxes out at 0.25 and keeps shrinking the gradient at every layer it passes through, ReLU either passes the gradient through completely unchanged (for positive inputs) or blocks it entirely (for negative inputs) — Xavier Glorot, Antoine Bordes, and Yoshua Bengio's 2011 paper "Deep Sparse Rectifier Neural Networks" demonstrated this directly enabled training substantially deeper networks than sigmoid-based architectures could reach, at the cost of a different failure mode (dead units, where a neuron's input stays permanently negative and its gradient stays permanently zero).

Residual connections solved the problem a third way, by giving the gradient a direct path around the multiplicative chain entirely rather than trying to fix the chain itself. He, Zhang, Ren, and Sun's 2015 ResNet paper added skip connections that add a layer's input directly to its output, which means the gradient has an identity path backward that bypasses the layer's internal transformation completely — the gradient doesn't have to survive being multiplied by that layer's Jacobian at all, it can just flow through the skip connection instead. This architectural trick is specifically what let ResNet scale successfully to 152 layers, dramatically deeper than anything sigmoid-based or even plain ReLU-based networks had trained reliably before it, and residual connections are now standard in virtually every deep architecture, including the transformer blocks underneath modern large language models.

The vanishing gradient problem and its mirror image, the exploding gradient problem (where derivatives compound to enormous values instead of vanishing), are both consequences of the same underlying fact: backpropagation through many layers is repeated multiplication, and repeated multiplication of anything other than exactly 1 either shrinks toward zero or grows without bound — every fix covered here (gating, ReLU, residual connections) works by either keeping that per-layer factor close to 1 or giving the gradient an alternate path that avoids the multiplication altogether.