token dropping in moe

Token dropping in MoE is the overflow handling behavior where tokens exceeding expert capacity are skipped or routed through fallback paths - it protects system stability when router assignments temporarily exceed per-expert processing limits.

What Is Token dropping in MoE?

• Definition: Capacity-control mechanism that limits tokens processed by each expert per step.
• Trigger Condition: Occurs when router assignments exceed configured expert capacity factor.
• Fallback Modes: Dropped tokens may pass through residual paths, backup experts, or deferred handling logic.
• Systems Context: Relevant in top-k routing schemes where load spikes can be highly uneven.

Why Token dropping in MoE Matters

• Stability Protection: Prevents runtime failures from expert buffer overflow.
• Throughput Control: Keeps step latency predictable under routing imbalance.
• Quality Risk: Excessive drops can hurt model accuracy and gradient quality.
• Capacity Planning: Drop rate is a key signal for tuning expert count and routing policy.
• Operational Monitoring: Persistent dropping indicates load-balancing or architecture issues.

How It Is Used in Practice

• Metric Tracking: Monitor drop fraction by layer, expert, and training phase.
• Router Tuning: Adjust capacity factors, auxiliary losses, and routing temperature to reduce overflow.
• Fallback Design: Implement robust residual or backup routing to limit quality degradation.

Token dropping in MoE is an important safeguard but also a diagnostic signal - controlled low drop rates indicate healthy routing and efficient expert utilization.