hyperparameter

A hyperparameter is any training setting a human chooses before training starts, as opposed to a parameter — a weight the training process itself learns by gradient descent. GPT-3, described in Brown et al.'s May 2020 paper, has 175 billion learned parameters, but the entire run was governed by a short list of hyperparameters chosen in advance: a learning rate of 6×10⁻⁵, a batch size of roughly 3.2 million tokens, and a 300-billion-token training budget. Change one of those numbers and you get a measurably different model from the identical architecture and data — which is the whole reason hyperparameters get their own name instead of being lumped in with everything else the model "learns."

Human chooses a value
learning rate, batch size, layer count — set before training
Model trains
parameters (weights) are learned via gradient descent
Scored on validation set
never the test set — see validation-set
Underfits or diverges
adjust the hyperparameter, retrain
Validation score improves
keep this setting, tune the next hyperparameter
Final configuration locked
used for the production training run

The learning rate is usually the first hyperparameter tuned, because it dominates everything else. Too high, and training diverges — the loss oscillates or explodes instead of decreasing; too low, and training converges so slowly that a model never reaches a competitive loss within its compute budget. DeepMind's 2022 Chinchilla paper (Hoffmann et al.) showed that most large language models up to that point had actually gotten a different hyperparameter wrong at a much bigger scale: given a fixed compute budget, labs had been training models too large on too few tokens, and a 70-billion-parameter model trained on 1.4 trillion tokens with corrected hyperparameters outperformed the 280-billion-parameter Gopher model it was built to compare against.

What Gets Learned vs. What Gets Chosen GPT-3 (Brown et al., 2020) — 175 billion of one, a handful of the other Parameters (learned) set automatically by gradient descent 175,000,000,000 weight and bias values Updated every training step, by the backward pass, millions of times over the training run. No human sets these values directly — only the process that finds them. Hyperparameters (chosen) fixed before training ever begins Learning rate6 × 10⁻⁵ Batch size~3.2M tokens Training tokens300B Layers96 Attention heads96 Context window2,048 tokens Roughly a dozen numbers that determine how those 175 billion weights get set. Get the dozen wrong, and no amount of correct gradient descent fixes it — see Chinchilla, 2022.

Searching for good hyperparameters is a separate, well-studied problem from defining what they are. Grid search, random search, Bayesian optimization, and population-based training are all strategies for exploring the hyperparameter space efficiently rather than training every possible combination — each with its own trade-offs between compute cost and search quality, and each covered in its own entry on this site rather than repeated here. What matters at the conceptual level is simpler: a hyperparameter search is itself scored against a validation set, never the test set, for exactly the same leakage reason described in cross-validation — tuning against the test set turns it into a second validation set and invalidates the final reported number.

Architecture choices are hyperparameters too, not just numeric training settings. The number of transformer layers, the number of attention heads, the hidden dimension size, and the choice of activation function are all fixed before training and never updated by gradient descent, which puts them in the same category as the learning rate even though they shape the model's structure rather than its optimization trajectory — this is why "hyperparameter tuning" in practice covers both knobs on the training loop and knobs on the architecture itself, and why changing either one requires training a new model from scratch rather than simply continuing the old run.

The distinction that matters is not how a setting is phrased but who is responsible for it: a parameter is whatever gradient descent is left to discover on its own, and a hyperparameter is whatever a human — or a search algorithm standing in for one — decided in advance, which is exactly why getting a dozen numbers wrong can waste a training run that correctly optimizes 175 billion weights.

Go deeper with CFSGPT

Get AI-powered deep-dives, save terms, and run advanced simulations — free account.

Create Free Account