fast geometric ensembling (fge)
Fast Geometric Ensembling (FGE) is an efficient ensemble construction technique that exploits the geometric structure of the loss landscape to collect diverse model checkpoints along a single training trajectory, using a cyclical learning rate schedule with carefully chosen cycle length to traverse low-loss paths connecting different local minima. FGE extends the snapshot ensemble concept by leveraging the observation that good minima in deep neural network loss landscapes are connected by low-loss "tunnels."
Why FGE Matters in AI/ML:
FGE provides high-quality ensembles at single-training-run cost by exploiting the connected geometry of the loss landscape, producing models that are diverse yet individually high-performing by traversing the low-loss manifold between minima.
• Loss landscape connectivity — Research shows that independently trained neural networks converge to minima connected by low-loss paths; FGE exploits this by traversing these paths during training, collecting checkpoints at different points along the connected low-loss manifold
• High-frequency cyclical schedule — FGE uses shorter learning rate cycles than standard snapshot ensembles, enabling more frequent checkpoint collection; the shorter cycles keep the model in low-loss regions while providing sufficient perturbation for diversity
• Geometric averaging — Beyond simple prediction averaging, FGE supports weight-space averaging of checkpoints along the trajectory, producing a single model (SWA-style) that approximates the ensemble at no additional inference cost
• Diversity vs. quality tradeoff — FGE carefully balances checkpoint diversity (models should make different predictions) against individual quality (each checkpoint should perform well); the connected loss landscape ensures both conditions hold simultaneously
• Relationship to SWA — Stochastic Weight Averaging (SWA) averages the weights collected by FGE into a single model, while FGE keeps them separate for ensemble prediction; FGE provides better uncertainty estimation while SWA provides better single-model performance
| Property | FGE | Snapshot Ensemble | Independent Ensemble |
|---|---|---|---|
| Training Cost | ~1× | ~1× | N× |
| Cycle Length | Short (2-4 epochs) | Long (epochs/M) | N/A |
| Checkpoint Quality | High (near minima) | Good (at minima) | Highest |
| Diversity | Moderate-High | Moderate | Highest |
| Uncertainty Quality | Good | Moderate | Best |
| Weight Averaging → | SWA | SWAP | N/A |
| Typical Members | 10-20 | 3-8 | 3-10 |
Fast Geometric Ensembling leverages the connected geometry of neural network loss landscapes to efficiently collect diverse, high-quality model checkpoints along low-loss paths, providing ensemble-quality predictions and uncertainty estimates at the computational cost of a single training run—making it the optimal choice when training budget constraints preclude independent ensemble training.