nfnet

NFNet (Normalizer-Free Networks) is a high-performance CNN architecture that achieves state-of-the-art accuracy without using batch normalization — using Adaptive Gradient Clipping (AGC) and carefully designed signal propagation to replace BatchNorm entirely.

What Is NFNet?

• No BatchNorm: Eliminates all BN layers. Uses Scaled Weight Standardization + AGC instead.
• AGC: Clips gradients based on the ratio of gradient norm to parameter norm (unit-wise).
• Signal Propagation: Carefully designed variance-preserving residual connections using a scaling factor.
• Paper: Brock et al. (2021).

Why It Matters

• SOTA Without BN: NFNet-F1 achieves 86.5% ImageNet top-1 (SOTA at time of release) without any normalization.
• Large Batch Friendly: No BN -> no batch size dependency -> cleaner distributed training.
• Simplicity: Removes the BN dependency that complicates training, transfer learning, and inference.

NFNet is the proof that BatchNorm is optional — achieving record accuracy by replacing normalization with principled gradient clipping and signal propagation.