sinusoidal position encoding

Sinusoidal Position Encoding is the original position encoding from the Transformer paper — using fixed sine and cosine functions at different frequencies to encode absolute position, based on the idea that relative positions can be represented as linear transformations.

How Does It Work?

• Formula: PE_{(pos, 2i)} = sin(pos / 10000^{2i/d}), PE_{(pos, 2i+1)} = cos(pos / 10000^{2i/d})
• Frequencies: Each dimension uses a different frequency, from high (position 0) to low (position d-1).
• Relative Position: PE_{pos+k} can be represented as a linear function of PE_{pos} for any fixed k.
• Paper: Vaswani et al. (2017).

Why It Matters

• No Parameters: Completely deterministic — no learnable parameters for position encoding.
• Extrapolation: Can theoretically encode positions beyond the training length.
• Foundation: Inspired RoPE, ALiBi, and other modern position encodings.

Sinusoidal Position Encoding is the mathematical clock of the original Transformer — encoding position through harmonic frequencies at no parameter cost.