encoder decoder

An encoder-decoder model splits sequence-to-sequence work into two distinct halves: an encoder that reads the entire input with full bidirectional context, and a decoder that generates the output one token at a time, looking back at the encoder's representations through cross-attention while it writes. Ilya Sutskever, Oriol Vinyals, and Quoc Le introduced the architecture for neural machine translation in their 2014 paper "Sequence to Sequence Learning with Neural Networks," compressing an entire input sentence into one fixed-length vector that the decoder then had to unpack — a bottleneck that degraded badly on long sentences, since one vector has to carry everything regardless of input length. Dzmitry Bahdanau and colleagues fixed this the same year by adding attention, letting the decoder look back at every one of the encoder's hidden states instead of just the final compressed one — the mechanism that survived, scaled up, into the Transformer's encoder-decoder architecture described in Vaswani et al.'s 2017 "Attention Is All You Need."

Encoder reads the full input
bidirectional — every token sees every other token
Encoder outputs one vector per input token
a full set of representations, not one compressed summary
Decoder generates one output token
attends to its own prior outputs AND every encoder vector
End token generated
full output sequence complete

Cross-attention is the specific mechanism that lets the decoder "read" the input while it writes, and it's worth tracing concretely for a single word. When a translation model is generating the French word for "cat" partway through a sentence, cross-attention computes a weighted combination of every encoder output vector, and that combination ends up weighted most heavily toward whichever English input token — almost certainly "cat" itself — is most relevant to the word being generated right now; this per-step, input-wide lookup is exactly what Bahdanau's 2014 attention mechanism added, and exactly what a single fixed-length vector could never provide, since it has to serve every decoding step identically regardless of which part of the input matters at that moment.

Two Halves vs. One Stream encoder-decoder (T5, BART) vs. decoder-only (GPT) — same transformer blocks, different wiring Encoder-Decoder (T5, BART) Encoder bidirectional Decoder causal + cross-attn cross-attn Input and output are structurally separate — clean fit for translation, summarization, Q&A T5 (Raffel et al., 2020): 60M to 11B parameters Long input, short output — encode once, decode cheaply Decoder-only (GPT) Single causal stack input and output share one stream No architectural separation — prompt and generation are just earlier and later tokens in the same sequence GPT-3, Llama, Claude: this is what scaled Simpler to scale — one stack, one training objective, massively parallel Decoder-only won the scaling race; encoder-decoder still wins the task fit for structurally distinct input/output.

Encoder-decoder's advantage hasn't disappeared, it's narrowed to specific cases where the task structure genuinely has two separate halves. Google's T5 (Raffel et al., 2020) demonstrated that reframing every NLP task — classification, translation, summarization, question answering — into a uniform "text in, text out" format let one architecture and one pretraining procedure, called span corruption, handle all of them; BART (Lewis et al., 2019) trained the same architecture as a denoising autoencoder instead. Both remain genuinely efficient specifically when the output is much shorter than the input, because the encoder processes the long input exactly once while the decoder only has to generate the short remainder, rather than re-attending over the entire growing sequence the way a decoder-only model does.

Decoder-only models won the race to massive scale not because the encoder-decoder's task fit was wrong, but because having one stack instead of two is simpler to train, simpler to scale, and sufferable for far more tasks than architecture purists initially expected. GPT, Llama, and Claude all treat a prompt and its continuation as one continuous sequence rather than two structurally distinct halves, which sacrifices the clean input/output separation encoder-decoder provides but buys a uniform training objective and architecture that scales to hundreds of billions of parameters without the added complexity of coordinating two separate stacks and a cross-attention bridge between them.

The real dividing line between the two architectures isn't quality, it's whether a task actually has two separable halves worth modeling differently — translation genuinely has distinct source and target languages, summarization genuinely compresses a long input into a short output, and for those cases encoder-decoder's explicit split keeps paying off; most open-ended generation and conversation does not have that clean split, which is exactly why decoder-only architectures, not encoder-decoder, became the default for general-purpose language models.

Go deeper with CFSGPT

Get AI-powered deep-dives, save terms, and run advanced simulations — free account.

Create Free Account