strided attention

Strided Attention is a sparse attention pattern where each token attends to every s-th token in the sequence — creating a dilated attention pattern that efficiently captures long-range dependencies without computing full O(N^2) attention.

How Does Strided Attention Work?

• Pattern: Token i attends to tokens (i - s, i - 2s, ...) (every s-th previous token).
• Stride s: Typically s = sqrt(N) so each token attends to sqrt(N) positions.
• Combined: Often paired with local attention — local captures nearby context, strided captures distant context.
• Paper: Child et al. (2019, Sparse Transformer).

Why It Matters

• Long-Range: Captures dependencies across the full sequence length with only O(sqrt(N)) attention per token.
• Complementary: Combined with local attention, provides both fine-grained local and coarse global context.
• Image Generation: Originally designed for autoregressive image generation (attending to spatially distant pixels).

Strided Attention is dilated convolution for attention — skipping tokens at regular intervals to efficiently reach across the entire sequence.