learning rate

Gradient descent on a quadratic with condition number 1,000 is fastest at a learning rate that is 99.9 percent of the one that blows the run up. Not 50 percent, not the middle of a plateau — 99.9 percent, and the figure is exact rather than empirical. That single fact sets the character of every learning-rate decision that follows: the quantity being tuned is not a dial with a comfortable middle, it is a distance from a wall, and where the wall stands is decided by the curvature of the loss, the width of the network, the size of the batch and the units the gradient happens to carry. Change any of those without changing the model at all and the right learning rate moves by orders of magnitude. The number is a statement about the whole training configuration, and treating it as a property of the architecture is the source of most of the folklore around it.

The stability bound is a spectral radius, and the optimum sits directly underneath it. For a quadratic loss built from a curvature matrix, gradient descent multiplies the error in each eigendirection by a factor of one minus the learning rate times that direction's curvature, every step, so the iteration converges only if that factor stays below one in magnitude in every direction. The binding direction is the sharpest one, which gives the familiar divergence threshold of two divided by the largest curvature value, and the rate that minimizes the worst-case contraction balances the sharpest direction against the flattest: it equals two divided by the sum of the smallest and largest curvature values. As a fraction of the divergence threshold, that optimum works out to the condition number divided by the condition number plus one — condition number here meaning the ratio of the largest curvature to the smallest. At a condition number of 10 the optimum is 90.9 percent of the divergence threshold, at 100 it is 99.01 percent, at 1,000 it is 99.90 percent and at 10,000 it is 99.99 percent. Better conditioning buys headroom and no real problem has any. Measuring the cost of standing back from the wall — steps to cut the loss a millionfold at condition number 100 — gives 3,101 steps at 5 percent of the threshold, 619 at 25 percent, 309 at 50 percent, 171 at 90 percent and 268 at 99 percent, where the curvature of the second-slowest direction starts to bite. Half the optimal rate costs 1.2 times the steps and a twentieth of it costs 18 times. One percent over the threshold costs the run. The asymmetry is the whole reason learning-rate tuning feels like a search rather than an optimization: the penalty for being low is polynomial and the penalty for being high is total.

The curvature does not constrain the learning rate so much as move to meet it. Cohen and colleagues observed that a network trained by full-batch gradient descent does not keep the sharpness that initialization gave it. The sharpness rises until it reaches two divided by the learning rate, then hovers there, oscillating across the stability boundary while the loss continues to fall — the regime now called the edge of stability. Reproducing it on a small tanh network, identical data and identical initialization at three learning rates: at a rate of 0.25 the sharpness starts at 5.31 and ends at 7.998 against a threshold of 8.000, a ratio of 1.000; at a rate of 0.50 it ends at 4.03 against 4.00, a ratio of 1.008. The same network on the same data ends up with twice the curvature when trained at half the learning rate. At a rate of 0.10, where the threshold of 20.0 is far above where the network naturally sits, sharpness climbs from 5.31 to 10.84 over 24,000 steps and is still climbing — it has not reached the boundary yet, and the ratio is 0.54. This inverts the usual mental model. Sharpness is not a fixed obstacle that the learning rate must respect; it is a quantity the learning rate selects, which is why measuring the curvature of a trained network tells you about the schedule that produced it as much as about the loss landscape it lives on.

Under plain SGD the learning rate carries the units of the gradient; under Adam it carries none. Multiply the loss by a constant and every gradient scales by that same constant, so SGD needs the rate divided by that constant to take the same step. Adam instead forms a ratio of a running average of the gradient to the square root of a running average of the squared gradient, and both the numerator and denominator scale by the same constant when the loss does, so the ratio — and the optimal rate — barely moves at all. Sweeping this rather than asserting it: across a millionfold rescaling of the loss the best SGD rate moved by exactly 1,000,000 times, and the best Adam rate moved by 1.58 times, which is one cell of a search grid whose spacing is 1.58. The invariance is exact and not statistical — running Adam from the same seed at the same rate with the loss multiplied by 1,000 gives weight trajectories that differ by 4.6 parts in a hundred thousand after 200 steps, which is floating-point noise in Adam's stabilizing constant. This is why 3e-4 functions as a usable default across models three orders of magnitude apart in size while no single SGD rate does, and it is also why a loss-scaling change, a switch from sum reduction to mean reduction, or a change in gradient-accumulation arithmetic silently retunes an SGD run and leaves an Adam run alone.

Warmup exists because the first Adam step is the one step that ignores the loss surface entirely. With bias correction applied, on the very first step the running averages reduce exactly to the raw gradient and its square, so the ratio Adam uses collapses to just the sign of the gradient, and every parameter moves by exactly the learning rate regardless of how large or small its gradient actually was. Checked across gradients spanning eight orders of magnitude, the ratio of update size to learning rate is 1.000000 in every case on that first step. A weight initialized at roughly one over the square root of its fan-in therefore moves a fixed fraction of its own size on step one, set by the learning rate and the fan-in and by nothing else: at a rate of 3e-4 that is 0.96 percent of a weight at fan-in 1,024, 1.92 percent at 4,096 and 3.84 percent at 16,384, so a wide model displaces its weights four times as far on its first step as a narrow one at the same rate. Measuring how erratic that step is, using 4,000 independent coordinates drawn from a fixed gradient distribution, the first update is 3.0 times the magnitude of the same optimizer's settled update and 4.4 times as variable, and the magnitude comes within 5 percent of its settled value only around step 25. Warmup is the ramp that covers that interval, and its natural length is set by the optimizer's own moment-averaging windows — roughly 10 steps for the first moment and roughly 1,000 steps for the second — rather than by anything about the model.

Widening a network moves its optimal learning rate, unless the parameterization is built so that it does not. Sweeping sixteen learning rates at five widths from 64 to 1,024 under the textbook initialization scheme with one global Adam rate — which is what every framework does by default — the optimum drifts from 6.31e-2 down to 1.00e-2, a factor of 6.3 across a 16-fold change in width. Under maximal update parameterization, which scales the output multiplier and the per-layer Adam step inversely with width relative to a fixed base width, the same sweep moves the optimum by a factor of 0.6, which is one grid cell in the other direction, so it has not meaningfully moved at all. The consequence is what makes this worth the trouble: tune at width 64 and transfer that number to width 1,024, and the standard parameterization lands at a loss of 3.56e-3 where 3.91e-6 was attainable, 910 times worse, while maximal update lands 7.35 times off. Yang and Hu's argument is that this is not a tuning trick but a statement about which infinite-width limit a parameterization converges to — the standard one drives feature learning to zero as width grows, so the best rate must shrink to compensate, and the maximal-update scaling is the one under which features keep moving. A learning-rate sweep at a small width is only informative about a large model if the parameterization was chosen to make it so.

Batch size buys steps until it does not, and the learning rate stops rising at exactly the same place. On a noisy quadratic where the gradient variance falls in inverse proportion to batch size, searching for the rate that reaches a fixed target in the fewest steps, the optimal rate climbs from 0.0196 at a batch of 1 to 1.5115 at a batch of 512 — a factor of 77 — and then stops, returning the identical value at 1,024 and at 4,096. The step count falls from 765 to 8 over the same range, a factor of 109, and then floors at 7. Fitting the standard noisy-quadratic model across the sweep gives a median implied critical batch size of 165, and both quantities saturate together because they saturate for the same reason: below the critical batch the run is noise-limited and a bigger batch is a better gradient that supports a bigger step, and above it the run is curvature-limited and the ceiling is fixed by the sharpest direction, which no amount of averaging raises. This is the honest version of the linear-scaling rule that Goyal and colleagues used to train ImageNet in an hour. Scaling the rate with the batch works while the noise dominates and silently stops working at a threshold that is a property of the task and the point in training, not of the hardware, which is why the rule needs warmup to survive the early steps and why McCandlish and colleagues framed the whole question as a measurement to make rather than a formula to apply.

A cosine schedule is a commitment to a step count made before step one, and the step count is an untuned hyperparameter. The declared horizon determines the entire shape of the decay, it is never trained on, never validated and rarely reported, and it changes the outcome substantially. Running the same network at the same peak rate for the same 4,000 steps under cosines declared for six different horizons, plus warmup-stable-decay, which declares nothing and decays only over the final fifth of whatever run it is given:

Across those six settings the loss at the stopping point spans a factor of 60, and the conventional choice — declare the horizon you intend to run — is the worst of the six, sixty times behind the best, which declared three times the run it actually got. The schedule that declares nothing beats every cosine at every checkpoint, and beats the matched cosine by a factor of 103 at the end. The mechanism is not mysterious: a matched cosine spends most of its budget already decayed, so it trades late-stage progress for an early-stopping curve nobody reads, while warmup-stable-decay holds the peak rate through the bulk of the run and collapses it only at the end. This is the finding that made warmup-stable-decay standard in the MiniCPM and DeepSeek recipes, and its practical value is not primarily the loss — it is that a run whose schedule does not depend on a declared endpoint can be extended, checkpointed and branched without invalidating everything before the branch point.

Read the learning rate through a *stability* lens rather than a *step-size* lens. As a step size it is a knob with a good value somewhere in the middle, and that picture predicts none of the behavior above. As a distance from the divergence threshold it predicts all of it: the optimum sits at condition-number/(condition-number+1) of the threshold because the threshold is what binds; the sharpness rises to two divided by the rate because the threshold is what the dynamics equilibrate against; the batch-size ceiling appears at the critical batch because past that point the threshold rather than the noise is what limits the step; and the maximal-update parameterization works because it holds the threshold fixed as width grows. Every reliable rule about learning rates is a rule about where that boundary sits and what moves it, and every quantity that moves it — the optimizer's units, the parameterization, the batch, the loss scale, the schedule's declared endpoint — is a decision made outside the model. Tune the rate last, after those are settled, and it becomes a one-dimensional search for a cliff edge. Tune it first and it absorbs the error in all of them.