attention mechanism hierarchical

Hierarchical Attention is an attention mechanism that operates at multiple levels of granularity — first computing attention within local groups (words, patches, tokens), then computing attention over group-level representations, creating a multi-scale attention hierarchy.

Common Hierarchical Patterns

• HAT: Word-level attention → sentence-level attention → document-level attention.
• Swin Transformer: Window-level attention → shifted window (inter-window communication).
• HiP Attention: Hierarchical token pooling with attention at each level.
• Nested Transformers: Attention within regions, then attention across regions.

Why It Matters

• Long Sequences: Handles very long sequences (documents, high-res images) by processing locally first, then globally.
• Efficiency: $O(N \cdot k)$ where $k$ is the local group size, vs. $O(N^2)$ for global attention.
• Multi-Scale: Naturally captures both fine-grained local patterns and coarse global patterns.

Hierarchical Attention is zoom-in-zoom-out attention — processing information at multiple scales from local details to global summaries.