feature learning regime

Feature Learning Regime is the operating mode where neural networks actively learn useful internal representations during training — as opposed to the lazy regime where features remain random. This is the regime where deep learning achieves its remarkable empirical success.

What Is Feature Learning?

• Condition: Networks with practical width, learning rate, and initialization (not the infinite-width NTK limit).
• Feature Evolution: Hidden representations change significantly during training, adapting to the data.
• Beyond NTK: NTK theory describes lazy training. Feature learning is the more complex, nonlinear regime.
• Muᵖ Parameterization: The maximal update parameterization (muP) provably enables feature learning at any width.

Why It Matters

• Performance: Feature learning is what makes deep learning work. Lazy training networks underperform.
• Representation: The ability to learn hierarchical features (edges -> textures -> objects) is deep learning's key advantage.
• Theory Gap: Feature learning is theoretically harder to analyze, creating a gap between NTK theory and practice.

Feature Learning is the real revolution of deep learning — the regime where networks actually learn the right internal representations, not just linearly combine random features.