slimmable networks

Slimmable Networks are neural networks trained to execute at multiple preset width configurations — a single model that can run at 0.25×, 0.5×, 0.75×, or 1.0× width, allowing runtime selection of the accuracy-efficiency trade-off without retraining.

Slimmable Training

• Switchable Batch Norm: Each width uses its own batch normalization statistics (separate running means/variances).
• Training: For each mini-batch, randomly select a width and train at that width — all widths share the same weights.
• Inference: Select the width at runtime based on the available computation budget.
• Width Configs: Typically 4 preset widths, but can be extended to more.

Why It Matters

• One Model, Many Budgets: Deploy a single model that adapts to varying computational resources at runtime.
• No Retraining: Switch between accuracy levels without retraining or storing multiple models.
• Device Heterogeneity: Different devices run the same model at different widths matching their hardware capability.

Slimmable Networks are the adjustable-width neural network — one model trained to operate at multiple efficiency levels, selected at runtime.