mechanistic

Mechanistic interpretability reverse-engineers a neural network the way a security researcher reverse-engineers compiled software: not approximating what it does with a separate explanatory model, but reading the actual weights to recover the specific algorithm they implement. This is a stricter and harder goal than the post-hoc explanation methods — SHAP, LIME, attention visualization — that interpretability more broadly includes, because mechanistic interpretability isn't satisfied with a plausible story about what a model might be doing; it wants causal proof of which components compute which part of a behavior, verified by intervening on those components directly and checking that the behavior changes exactly as predicted.

Induction heads were the first clean example of a specific, identifiable algorithm found inside a transformer's weights. Olsson and colleagues at Anthropic (2022) found attention head pairs that implement a simple but powerful in-context pattern: search the context for a prior occurrence of the current token, find what token followed it last time, and predict that same token again — literally completing the pattern [A][B]...[A] by predicting [B]. What made the discovery significant wasn't just finding this specific circuit, but when it forms: induction heads emerge during a sudden, sharp phase transition early in training, and that transition coincides almost exactly with the model's in-context learning ability suddenly improving — strong evidence that a single, identifiable mechanical component is directly responsible for a capability that otherwise looks like a vague, emergent property of scale.

The indirect object identification circuit in GPT-2 is the field's most complete worked example of this methodology end to end. Wang and colleagues at Redwood Research (2022) analyzed how GPT-2 completes a sentence like "John gave Mary the book; Mary gave ___" with the correct name, and traced the behavior to 26 specific attention heads, each with an identified role — heads that inhibit the subject from being repeated, heads that "move" the correct name into the output position, backup heads that take over if the primary ones are ablated. This wasn't a correlational finding; each head's role was confirmed by intervening on it directly and observing a specific, predicted change in the model's output, which is what separates a mechanistic circuit claim from a plausible-sounding story about what a component might be doing.

Grokking showed that a model can be internally running a clean, interpretable algorithm long before its training loss shows any sign of it. Models trained on modular arithmetic tasks famously exhibit "grokking" — they first appear to memorize the training set, with training loss near zero and test loss still high, and then, often much later in training, generalization suddenly appears and test loss drops sharply. Mechanistic analysis of these models (Nanda and colleagues, 2023) found that during the long period that looked like pure memorization, the model was actually converging on a specific Fourier-transform-based algorithm for modular arithmetic internally — generalization wasn't a sudden new capability appearing from nowhere, it was the final cleanup phase of an algorithm the model had already mostly learned, visible only once researchers looked at the weights directly rather than at the loss curve.

Superposition is the reason a single neuron rarely means just one thing, and sparse autoencoders exist specifically to undo it. A network has far more concepts to represent than it has neurons or dimensions to represent them in, so models learn to pack multiple, only loosely related features into overlapping, nearly-orthogonal directions within the same activation space — a single neuron can fire for several unrelated concepts at once, a property called polysemanticity, which makes directly reading meaning off of individual neurons largely useless. Sparse autoencoders (central to Anthropic's "Towards Monosemanticity" work) are trained to re-expand that compressed, overlapping representation into a much larger, sparser set of directions, each corresponding to a single, more human-interpretable feature — effectively undoing the model's own compression so individual features become legible again, at the cost of training an entirely separate model just to decode the first one.

Read mechanistic interpretability as treating a trained network's weights as a program to be decompiled rather than a black box to be probed from outside: every landmark result in the field — induction heads, the IOI circuit, grokking's hidden Fourier algorithm, superposition's feature-packing — came from insisting on a causal, falsifiable account of what specific components compute, which is a far higher bar than a plausible-looking explanation, and is exactly the bar the field holds itself to precisely because the eventual goal is verifying what a model is actually doing, not just what it appears to be doing.