adaptive token selection

Adaptive Token Selection is an efficiency technique where the model dynamically selects a subset of tokens to process based on the input difficulty — easier inputs use fewer tokens (and exit earlier), while harder inputs use more computation, creating an input-adaptive compute budget.

How Does Adaptive Token Selection Work?

• Halting Score: Each token receives a halting probability at each layer (like Adaptive Computation Time).
• Exit Criterion: Tokens that have "converged" (halting score > threshold) stop being processed.
• Budget Control: A regularization loss encourages the model to halt early when possible.
• Example: A-ViT (Adaptive Vision Transformer) uses per-token adaptive halting.

Why It Matters

• Input-Dependent Compute: Simple images (clear sky) use fewer tokens/layers than complex scenes.
• No Fixed Budget: Unlike static pruning, the compute budget adapts to each input dynamically.
• Anytime Prediction: Can produce predictions at any computational budget by halting earlier.

Adaptive Token Selection is pay-per-difficulty inference — allocating more computation to hard inputs and saving resources on easy ones.