induced set attention block

ISAB (Induced Set Attention Block) is a memory-efficient attention block that uses a small set of learnable inducing points to compress the $O(N^2)$ self-attention — tokens first attend to the inducing points (forming a bottleneck), then the inducing points attend back to the tokens.

How Does ISAB Work?

• Inducing Points: $I \in \mathbb{R}^{m \times d}$ — a set of $m$ learnable vectors ($m ll N$).
• Step 1: $H = \text{MAB}(I, X)$ — inducing points attend to input tokens. $H \in \mathbb{R}^{m \times d}$.
• Step 2: $\text{ISAB}(X) = \text{MAB}(X, H)$ — input tokens attend to the compressed inducing points.
• Complexity: $O(N \cdot m)$ instead of $O(N^2)$.

Why It Matters

• Bottleneck Attention: The $m$ inducing points act as a compressed representation of the entire set.
• Scalable: With $m = 32-128$, can process sets of thousands of elements efficiently.
• Perceivers: The same principle was later adopted by Perceiver and Perceiver IO for general-purpose architectures.

ISAB is attention through a bottleneck — using a small set of learned summary points to avoid the quadratic cost of full self-attention.