swa-gaussian

SWAG (SWA-Gaussian) is an approximation to Bayesian deep learning that uses the SWA trajectory to fit a Gaussian distribution over weights — capturing both the mean (SWA solution) and the covariance (spread of the SWA trajectory) for uncertainty estimation.

How Does SWAG Work?

• Mean: bar(theta) from SWA (average of collected checkpoints).
• Covariance: Estimate a low-rank + diagonal covariance from the deviations of collected checkpoints from the mean.
• Posterior: q(theta) = N(bar(theta), Sigma_SWAG) (Gaussian approximate posterior).
• Inference: Sample multiple models from the posterior and average predictions (Bayesian model averaging).
• Paper: Maddox et al. (2019).

Why It Matters

• Uncertainty: Provides calibrated uncertainty estimates without the cost of full Bayesian inference.
• Efficient: Only requires the SWA trajectory — no special modifications to training.
• Scalable: Works for modern deep networks (ResNets, etc.) where full Bayesian methods are intractable.

SWAG is SWA with uncertainty — using the natural variation in the SWA trajectory to estimate a Bayesian posterior for calibrated predictions.