gatedcnn

Gated CNN is a convolutional architecture that uses gated linear units (GLU) instead of standard activation functions — enabling content-dependent feature selection through learned multiplicative gates, achieving competitive results with RNNs on sequence modeling tasks.

How Does Gated CNN Work?

• Architecture: Standard 1D convolutions (for sequence data), but each layer uses GLU activation.
• Residual Connections: Combined with residual/skip connections for gradient flow.
• Parallel: Unlike RNNs, all positions are computed in parallel -> much faster training.
• Paper: Dauphin et al., "Language Modeling with Gated Convolutional Networks" (2017).

Why It Matters

• Pre-Transformer: Demonstrated that CNNs with gating could match LSTM performance on language modeling.
• Speed: Fully parallelizable — 10-20x faster training than equivalent LSTMs.
• Influence: The gating mechanism directly influenced the FFN design in modern transformers (SwiGLU).

Gated CNN is the convolutional language model — proving that convolutions with gates could challenge the RNN dominance in sequence modeling.