switchable normalization

Switchable Normalization is a meta-normalization technique that learns to combine BatchNorm, InstanceNorm, and LayerNorm — using learnable weights to adaptively select the optimal normalization method for each layer and each channel during training.

How Does Switchable Normalization Work?

• Three Statistics: Compute BN, IN, and LN statistics simultaneously.
• Learnable Weights: hat(mu) = lambda_BNmu_BN + lambda_INmu_IN + lambda_LNmu_LN (and same for variance).
• Softmax: Weights are softmax-normalized -> always sum to 1.
• Learning: The network learns which normalization is best for each layer.
• Paper: Luo et al. (2019).

Why It Matters

• Automatic Selection: No need to manually choose between BN, IN, LN — the network decides.
• Task-Adaptive: Different tasks (classification, style transfer, detection) benefit from different normalizations.
• Insight: Analysis of learned weights reveals which normalization is preferred at different depths and for different tasks.

Switchable Normalization is letting the network choose its own normalization — a meta-learning approach that adapts normalization strategy per layer.