dropout regularization

Dropout — a regularization technique that randomly deactivates neurons during training, forcing the network to learn redundant representations and reducing overfitting.

How It Works

• During training: Each neuron is set to zero with probability p (typically 0.1–0.5)
• During inference: All neurons are active, but outputs are scaled by (1-p) to compensate
• Effect: The network can't rely on any single neuron — must learn distributed, robust features

Why It Works

• Approximate ensemble: Each training step uses a different sub-network. Dropout is like training 2^n networks simultaneously
• Prevents co-adaptation: Neurons can't learn to depend on specific partners

Variants

• Standard Dropout: Applied to fully connected layers
• Spatial Dropout (Dropout2D): Drops entire feature maps in CNNs (more effective than per-pixel)
• DropConnect: Drops weights instead of activations
• DropPath/Stochastic Depth: Drops entire residual blocks (used in Vision Transformers)

Practical Tips

• Typically p=0.5 for hidden layers, p=0.1–0.2 for input layers
• Don't use with Batch Normalization (they conflict — BN already regularizes)
• Always disable during evaluation: `model.eval()` in PyTorch

Dropout remains one of the most effective and widely-used regularization techniques despite its simplicity.