rnn

A recurrent neural network (RNN) is the architecture built for sequences — text, audio, sensor streams, anything where order matters — and it handles sequence the simplest way possible: walk through the input one element at a time, and after each step update a single hidden-state vector meant to summarize everything seen so far. That hidden state is the RNN's entire memory, which is both the architecture's core idea and its central weakness, since cramming an arbitrarily long history into one fixed-size vector doesn't scale cleanly. At a high level, one pass through a sequence looks like this:

The vanishing-gradient problem is why plain RNNs can't remember anything far back in a long sequence, and it's a direct consequence of the chain rule. Backpropagation through a 100-step sequence multiplies roughly 100 gradient terms together; if those terms are each slightly less than 1, their product shrinks toward zero exponentially, meaning the gradient signal from a mistake at step 100 barely reaches the weights responsible for what happened at step 1. LSTM (Long Short-Term Memory) and GRU (Gated Recurrent Unit) cells were specifically engineered to fix this with learned gates that let gradients bypass most of that multiplicative decay — a dedicated "cell state" that information can flow through largely unchanged unless the gates decide to forget it.

RNNs still exist and still matter in production, specifically where their sequential, constant-memory nature is an advantage rather than a bottleneck. A transformer's attention mechanism needs memory proportional to sequence length squared, which becomes expensive for extremely long or truly streaming inputs; an RNN's hidden state stays a fixed size no matter how long the sequence runs, which is exactly why RNN-style or RNN-hybrid architectures remain common in real-time speech recognition, keyboard autocomplete, and other latency-sensitive streaming applications where a bounded, constant-size memory footprint matters more than modeling very long-range dependencies.

Training an RNN is structurally harder to speed up on a GPU than training almost any other common architecture, and this is a hardware story more than a software one. Backpropagation through time requires unrolling the network across every timestep and processing them in strict order, so a GPU's parallel cores can batch across different training examples but not across steps within one sequence — this is the specific, measurable reason RNN training throughput scales far worse with more GPUs than transformer training does, and it's the real economic pressure (not just accuracy) that pushed the field toward attention-based architectures once GPU clusters became the default way to train at scale.

Read an RNN as the architecture that modeled time the most literally possible way — one state, updated one step at a time — and was replaced for most large-scale work the moment GPU hardware made a fully parallel alternative (the transformer) both more accurate and far cheaper to train at the same compute budget; where a bounded memory footprint and true streaming input still matter more than raw scale, the RNN's core idea is still the right tool.