landmark attention
Landmark Attention is the efficient transformer attention mechanism that reduces computational complexity by routing all token attention through a sparse set of landmark (anchor) tokens that serve as information hubs — achieving sub-quadratic attention cost while preserving global information flow — the architecture that demonstrates how strategically placed landmark tokens can serve as a compressed global context, enabling long-sequence processing without the full O(n²) cost of standard self-attention.
What Is Landmark Attention?
- Definition: A modified attention mechanism where regular tokens attend only to nearby local tokens and to a set of specially designated landmark tokens, while landmark tokens attend to all other landmarks — creating a two-level attention hierarchy with O(n × k) complexity where k << n is the number of landmarks.
- Landmark Selection: Landmarks are chosen at fixed intervals (every m-th token), at content boundaries (sentence/paragraph breaks), or through learned prominence scoring — they serve as representative summaries of their local region.
- Two-Level Attention: (1) Local tokens attend to their neighborhood + all landmarks (sparse), (2) Landmarks attend to all other landmarks (dense but small) — global information propagates through the landmark network while local processing remains efficient.
- Information Bridge: Landmarks act as bridges between distant sequence regions — a token at position 1 can influence a token at position 10,000 through their respective nearest landmarks, which are connected via landmark-to-landmark attention.
Why Landmark Attention Matters
- Sub-Quadratic Complexity: Standard attention is O(n²); Landmark attention is O(n × k + k²) where k << n — for k = √n, this becomes O(n^1.5), dramatically more efficient for long sequences.
- Global Information Preservation: Unlike local-only attention (which loses distant context), landmark-to-landmark attention maintains a global information pathway — important for tasks requiring full-document understanding.
- Minimal Quality Loss: Well-placed landmarks preserve 95%+ of full attention's information — the compression through landmarks retains the most important global signals.
- Compatible With Flash Attention: The local attention windows and landmark attention patterns can be implemented efficiently with existing optimized kernels.
- Configurable Trade-Off: Adjusting landmark density (k) provides a smooth trade-off between efficiency and information retention — more landmarks = more global information at higher cost.
Landmark Attention Architecture
Landmark Placement Strategies:
- Fixed Stride: Every m-th token is a landmark — simplest, works well for uniform-density text.
- Learned Selection: A scoring network assigns prominence scores; top-k scoring tokens become landmarks — content-aware, better for heterogeneous inputs.
- Boundary-Based: Landmarks placed at sentence boundaries, paragraph breaks, or topic transitions — aligns with natural information structure.
Attention Pattern:
- Regular token t attends to: local window [t−w, t+w] UNION all landmarks.
- Landmark l attends to: its local region UNION all other landmarks.
- This creates a sparse attention pattern with guaranteed global connectivity.
Complexity Comparison
| Method | Attention Complexity | Global Context | Memory |
|---|---|---|---|
| Full Attention | O(n²) | Complete | O(n²) |
| Local Window | O(n × w) | None | O(n × w) |
| Landmark Attention | O(n × k + k²) | Via landmarks | O(n × k) |
| Longformer | O(n × (w + g)) | Via global tokens | O(n × (w + g)) |
Landmark Attention is the information-routing architecture that proves global context can be maintained through strategic compression — using a sparse network of landmark tokens as information hubs that connect distant sequence regions at sub-quadratic cost, achieving the practical efficiency of local attention with the semantic capability of global attention.