instruction tuning

Instruction tuning is supervised fine-tuning on a dataset of (instruction, response) pairs, and its purpose is narrow and concrete: turn a raw next-token-predicting base model into one that actually follows directions instead of just continuing text in whatever direction the training corpus made statistically likely. Wei and colleagues' 2021 Google paper "Finetuned Language Models Are Zero-Shot Learners" introduced the technique under the name FLAN, fine-tuning a model on more than 60 NLP tasks rephrased as natural-language instructions and showing the resulting model performed dramatically better, zero-shot, on tasks it had never seen phrased that way during fine-tuning — the first clear demonstration that teaching a model the general shape of "instruction in, helpful response out" transfers far beyond the specific instructions it was trained on.

A raw base model's failure mode is concrete and easy to demonstrate: asked to write a poem, it may not write one at all. Trained purely to predict the statistically likely continuation of text, a base model given the prompt "write a poem about the sea" has learned that prompts like this are frequently followed by more writing-advice text, a list of other topics, or a continuation of an essay about poetry — because that's what actually follows similar text across the training corpus — rather than reliably producing an actual poem, since nothing in pure next-token prediction specifically rewards treating an instruction as something to obey. Instruction tuning exists to close exactly that gap, by showing the model enough real examples of instructions paired with the response a person actually wanted that it learns the general pattern: this kind of text is a request, and the job is to satisfy it.

Generating the instruction-response pairs by hand doesn't scale, which is exactly the problem Self-Instruct and Alpaca solved by having one model generate training data for another. Wang and colleagues' 2022 Self-Instruct method used a capable language model to generate new instructions and example responses on its own, bootstrapping from a small human-written seed set; Stanford's Alpaca project in 2023 applied exactly this approach, using OpenAI's GPT-3 to generate 52,000 instruction-response examples and fine-tuning a 7-billion-parameter LLaMA base model on them for a total cost reported at roughly 600 dollars — producing a model with surprisingly strong instruction-following ability for a fraction of the cost of hand-curating a comparable dataset, and demonstrating that instruction tuning's data bottleneck could be substantially automated rather than requiring armies of human writers.

Instruction tuning and RLHF solve genuinely different problems and typically happen in sequence, not as alternatives to each other. Instruction tuning, a supervised step, teaches the model the basic shape of following directions — what a reasonable response to an instruction even looks like; RLHF and its alternatives, covered under alignment, then further shape the model's behavior using human (or AI) preference judgments between multiple candidate responses, refining tone, safety, and helpfulness beyond what supervised examples alone can specify. OpenAI's InstructGPT, described in Ouyang and colleagues' 2022 paper, ran exactly this sequence — supervised instruction tuning first, then RLHF on top of it — and the paper's naming makes the division explicit: instruction tuning gets you a model that understands it's supposed to follow instructions, and RLHF is what tunes how well and how safely it does so.

Instruction tuning's real contribution was recognizing that "follow directions" is itself a learnable skill, separable from raw language modeling ability, and teachable with a comparatively small, even synthetically generated dataset — which is exactly why it became the standard first step in turning a base model into something that behaves like an assistant, before any of the preference-based refinement covered under alignment ever begins.