sharp minima

Sharp Minima are regions of the loss landscape where the loss increases rapidly when parameters are perturbed — characterized by large eigenvalues of the Hessian matrix, and empirically associated with poorer generalization to unseen data.

What Are Sharp Minima?

• Definition: A minimum where even small perturbations cause significant loss increase -> narrow valley.
• Hessian: Large eigenvalues indicate high curvature (sharpness).
• Large Batch Training: Gradient descent with very large batch sizes tends to converge to sharp minima.
• Overfitting: Sharp minima are often associated with overfitting because they represent "brittle" solutions.

Why It Matters

• Generalization Gap: The train-test performance gap is often larger for models converging to sharp minima.
• Batch Size Effect: This explains why large-batch training often degrades test accuracy.
• Mitigation: Learning rate warmup, SAM, and noise injection help steer optimization toward flatter minima.

Sharp Minima are the narrow canyons of the loss landscape — precise solutions that work perfectly on training data but crumble under the slightest perturbation.