adam optimizer
Adam combines momentum and per-parameter adaptive scaling into a single update rule, tracking two running averages for every weight: a first moment (the mean of recent gradients, same idea as plain momentum) and a second moment (the mean of recent squared gradients, which estimates how noisy or large that parameter's updates have been). Diederik Kingma and Jimmy Ba introduced the method in their 2014 paper "Adam: A Method for Stochastic Optimization," published at ICLR 2015, recommending default hyperparameters — learning rate 0.001, beta1 (first-moment decay) 0.9, beta2 (second-moment decay) 0.999, epsilon 1e-8 for numerical stability — that are still the starting point in virtually every deep learning framework a decade later.
Adam's memory cost is the direct, unavoidable price of tracking two numbers per weight instead of zero, and at modern model scale that price is measured in tens of gigabytes. Storing the first and second moment in 32-bit precision for a 7-billion-parameter model costs roughly 2 times 7 billion times 4 bytes, or about 56 gigabytes — on top of the model's own weights and gradients, which is why optimizer state, not the parameters themselves, is frequently the single largest consumer of GPU memory during large-model training, and why an entire line of research exists purely to shrink it.
8-bit Adam and Adafactor solve the memory problem through two genuinely different mechanisms, not just "the same thing but smaller." Tim Dettmers and colleagues' 2021 paper "8-bit Optimizers via Block-wise Quantization" keeps Adam's exact algorithm but quantizes the stored moments down to 8 bits instead of 32, cutting optimizer memory roughly 4x while the paper reports negligible accuracy loss across the benchmarks tested; Noam Shazeer and Mitchell Stern's 2018 Adafactor, used in Google's original T5 training, instead factorizes the second moment into a much smaller low-rank approximation rather than storing the full per-parameter value, trading some precision in the variance estimate for an even larger memory reduction.
The beta2 hyperparameter, left at Adam's 0.999 default, is one of the few knobs large language model training routinely overrides from the paper's original recommendation. GPT-3's training configuration, as described in Brown et al.'s 2020 paper, used beta1 of 0.9 but lowered beta2 to 0.95 — a faster-decaying second-moment estimate that reacts more quickly to recent gradient statistics, which multiple large-scale training runs have found improves stability at the scale and batch sizes typical of LLM pretraining, where the default 0.999 can make the adaptive learning rate adjust too sluggishly to shifting gradient conditions over a training run spanning hundreds of billions of tokens.
AdamW's decoupled weight decay is not a minor variant — it is the version virtually every modern large model actually trains with. Standard Adam applies weight decay by adding it into the gradient before the adaptive scaling divides by the second moment, which means large-gradient parameters get their decay shrunk by that same division — effectively under-regularizing exactly the weights that most need it; AdamW, described by Loshchilov and Hutter in 2017, applies weight decay as a separate, direct step on the weights themselves, outside the adaptive scaling entirely, and this single change measurably improved generalization enough that AdamW rather than plain Adam is the actual default in nearly every transformer training recipe published since.
Adam's lasting contribution was making a single optimizer robust enough to use as a sane default across wildly different architectures and scales without extensive per-model learning-rate tuning, and every major variant since — AdamW, 8-bit Adam, Adafactor, the beta2 tweak LLM training settled on — has been a response to one specific cost of that robustness, whether that cost showed up as memory, generalization, or stability at scale.