lora

The LoRA paper's headline is that it trains 10,000 times fewer parameters, and the number is honest — GPT-3 175B with rank-4 adapters on the query and value projections has 18.9M trainable parameters against 175.2B, a ratio of 9,281. What almost nobody checks is what that ratio does to the three quantities a training run is actually billed for. Peak memory falls 7.9 times, not 9,281. Training FLOPs fall 1.5 times. Merged inference latency falls not at all. The gap between the headline and the bill is the whole subject, and it is arithmetic rather than opinion.

The parameter ratio is real, and it is a ratio of the wrong quantity. Under standard mixed-precision AdamW, a trainable parameter costs 16 bytes (bf16 weight, bf16 gradient, fp32 master copy plus two moments); a frozen parameter costs two. So as the trainable fraction shrinks toward zero, the ratio of full-fine-tune memory to LoRA memory climbs toward a ceiling of eight — and no further, since fourteen of the sixteen bytes belong to the optimizer and the other two belong to weights that must stay resident regardless. Driving the trainable fraction from one part in eight hundred to one in ten thousand moves the memory ratio from only 7.92 to 7.99. Llama-2 7B is 100.4 GB to fine-tune fully and 12.7 GB under LoRA at rank 16 on q and v — 12.55 GB of that 12.7 GB is just the frozen model; the adapters and their Adam moments together are 0.11 GB.

Read the last two columns against the fifth. The parameter ratio moves by a factor of 11.6 down that table and the memory ratio moves by 1.3 percent. They are not measuring the same thing and they never were.

LoRA removes half of the backward pass, not the backward pass. A forward pass through a linear layer costs one unit of compute, C; the backward pass costs 2C, split into a gradient-to-input job and a gradient-to-weights job. Freezing a matrix deletes only the second job, because adapters in earlier layers still need their gradients, routed back through every frozen matrix above. Full fine-tuning is 3C per step; LoRA is 2C — a 33 percent saving, against a 99.88 percent trainable-parameter saving in the same configuration. With gradient checkpointing, which nearly every long-context recipe uses, the recompute adds one more C to both sides, dropping the saving to 25 percent. A LoRA run finishes in roughly two-thirds of the time, not a tenth of it, and step-time wins larger than that are almost always a bigger batch fitting in memory, not faster arithmetic.

Once the optimizer term is gone, the frozen weights are the memory, so the next factor has to come from a number format. This is what QLoRA is built on: with optimizer and gradient bytes removed, the only remaining lever is bytes per frozen parameter. Dettmers and colleagues replaced bf16 with 4-bit NormalFloat quantization, cutting that term roughly fourfold. Llama-2 7B goes 100.4 GB to 12.7 GB to 3.3 GB — 31 times a full fine-tune, and the last factor of 3.9 is a data type, not a rank. The 70B case changed practice: a full fine-tune holds 771 GB of AdamW state alone, ten A100-80GB cards before a single activation tensor, while 4-bit LoRA holds the whole model plus adapters in 32.6 GB — one card. Most claims about LoRA making 70B fine-tuning possible on a single GPU are claims about quantization wearing LoRA's name.

A rank sweep at fixed learning rate is also a learning-rate sweep. LoRA trains the product of a scale and two low-rank matrices, with the scale conventionally set to alpha divided by rank. One matrix starts at zero, so the first gradient step moves that product by roughly the learning rate times the scale squared — rank sits inside the effective step size, squared. Rank up sixteenfold means the effective step falls 256-fold. On a controlled regression, steps to cut the loss by 90 percent go 2,180 at rank 4 up to 12,300 at rank 64 — more rank, slower convergence, backwards from what expressivity predicts. Kalajdzievski's rank-stabilized scaling replaces alpha-over-rank with alpha over the square root of rank, and on the identical problem the sweep instead runs from 1,780 down to 1,180 steps: faster with more rank, the direction expressivity actually predicts. A rank sweep under the default scaling mostly measures the learning rate, and the common finding that rank barely matters is partly an artifact of that confound.

Merging is exact and free for one adapter, and it is catastrophic for a thousand. The merge is not an approximation: the adapted matrix is a plain matrix of the same shape as the original, so a merged adapter adds zero inference parameters and zero latency. The trouble is merging destroys sharing — ten task-specific adapters merged into ten copies of a 7B model is 125.5 GB of weights, where one shared base plus ten unmerged adapters is 12.7 GB; at a thousand adapters that gap is 445x. A multi-tenant server keeps adapters unmerged and pays a small batched low-rank matmul per token — what S-LoRA, Punica and vLLM's multi-LoRA paths are for — while a single-tenant deployment merges and pays nothing. LoRA's inference cost is a serving-topology decision, not a property of the method.

LoRA's real failure mode is that it cannot move what it cannot reach. The frozen set in a standard configuration includes the embedding matrix and output head — 262M parameters on a 7B model, four percent of it and all of the token geometry. A task needing new tokens, a new script, or a poorly-tokenized language asks the model to move exactly the tensors LoRA left alone, and no rank fixes that. The same applies to continued pretraining on a new domain, where the update is genuinely high-rank and spread across every layer: Biderman and colleagues found LoRA lagging full fine-tuning on code and math continued pretraining, while matching it on instruction tuning — the honest boundary of the method, with the upside that LoRA forgets the base distribution less, because it moved less.

Read LoRA through a memory-hierarchy lens rather than a parameter-count lens. The parameter count is a true statement about an object that was never expensive; the real bill is optimizer bytes, backward-pass FLOPs, and how many copies of the base model a serving fleet holds — each with its own ceiling. Memory stops at eight times because the frozen weights still have to be resident; compute stops at 1.5 times because the gradient still has to reach the earliest adapter; latency does not move once the matrices are merged. The one genuinely large factor anywhere in the method, QLoRA's 31 times, comes from writing the frozen weights in four bits — a number-format decision LoRA merely made affordable. Rank is the last thing to tune and the first thing everybody tunes; the real question is whether the change being asked for is small enough to live in a low-rank correction to the matrices left unfrozen.