relative position bias

Relative Position Bias is a position encoding method that adds a learnable bias to attention logits based on the relative distance between query and key positions — directly encoding "how far apart" two tokens are, rather than their absolute positions.

How Does Relative Position Bias Work?

• Bias Table: A learnable matrix $B \in \mathbb{R}^{(2M-1) \times (2M-1)}$ indexed by relative position $(i-j)$.
• Addition: $\text{Attention}(Q, K) = \text{softmax}(QK^T / \sqrt{d} + B)$.
• Per-Head: Different attention heads can have different relative position biases.
• Used In: Swin Transformer, T5, DeBERTa.

Why It Matters

• Translation Invariance: The same relative distance gets the same bias regardless of absolute position.
• Extrapolation: Can generalize to longer sequences than seen during training (with appropriate handling).
• SOTA: T5's relative position bias is among the most effective position encodings for NLP.

Relative Position Bias is distance-based attention adjustment — telling the model how to weight attention based on how far apart tokens are, not where they are.