dense synthesizer

Dense Synthesizer is a variant of the Synthesizer model where attention weights are generated by a feedforward network applied to each token independently — replacing the pairwise query-key dot product with a per-token MLP that directly predicts attention over all positions.

How Does Dense Synthesizer Work?

• Per-Token: For each token $x_i$, compute $a_i = W_2 \cdot \text{ReLU}(W_1 \cdot x_i)$ producing a vector of length $N$.
• Attention: $A = \text{softmax}([a_1; a_2; ...; a_N])$ (each row from one token's MLP output).
• No Key Interaction: Token $i$'s attention weights are computed without looking at any other token.
• Value Aggregation: Standard weighted sum of values using the synthesized attention.

Why It Matters

• Content-Dependent but Not Pairwise: Attention depends on the query token's content but not on explicit key comparison.
• Competitive: Matches or approaches standard attention on sequence-to-sequence and classification tasks.
• Hybrid: Can be combined with standard dot-product attention for best results.

Dense Synthesizer is attention from a single perspective — each token decides its attention pattern based solely on its own content, without consulting keys.