synthesizer

Synthesizer is a transformer variant that generates attention weights without computing query-key dot products — "synthesizing" attention maps directly from the input or from learned parameters, questioning whether explicit pairwise comparisons are necessary.

How Does Synthesizer Work?

• Dense Synthesizer: A = softmax(f(X)) where f is a feedforward network. Attention from content, no Q-K dot product.
• Random Synthesizer: A = softmax(R) where R is a learnable random matrix. No input dependence at all.
• Mixture: Combine dense, random, and standard dot-product attention.
• Paper: Tay et al. (2021).

Why It Matters

• Provocative: Random attention (no Q-K interaction!) performs surprisingly well on many benchmarks.
• Insight: Suggests that the specific pairwise token comparison in standard attention may not always be necessary.
• Efficiency: Dense/random synthesizers can be faster than full dot-product attention.

Synthesizer is the experiment that questioned attention — showing that attention weights can be generated without even comparing tokens to each other.