normalizing flow

Normalizing Flow is a generative model that learns an invertible mapping between a simple base distribution (Gaussian) and a complex data distribution — enabling exact likelihood computation and efficient sampling, unlike VAEs (approximate inference) or GANs (no likelihood).

Core Idea

• Learn invertible transformation $f_\theta: z \rightarrow x$ where $z \sim N(0,I)$.
• Change of variables: $\log p_X(x) = \log p_Z(z) + \log |\det J_{f^{-1}}(x)|$
• Train by maximizing log-likelihood directly — no approximation.
• Sample: $z \sim N(0,I)$, compute $x = f_\theta(z)$.

Key Architectural Requirement

• $f$ must be: (1) Invertible, (2) Differentiable, (3) Jacobian determinant efficiently computable.
• Most neural networks fail (2) and (3) — flows use special architectures.

Major Flow Architectures

Coupling Layers (RealNVP):
- Split $x$ into $x_1, x_2$. $y_1 = x_1$; $y_2 = x_2 \odot \exp(s(x_1)) + t(x_1)$.
- Jacobian is triangular → det = product of diagonal.
- $s, t$: Arbitrary neural networks — no invertibility constraint.
- Inverse: $x_2 = (y_2 - t(y_1)) \odot \exp(-s(y_1))$ — trivially invertible.

Autoregressive Flows (MAF, IAF):
- Each dimension conditioned on all previous.
- MAF: Fast training, slow sampling. IAF: Fast sampling, slow training.

Continuous Flows (Neural ODE-based):
- Continuous Normalizing Flow (CNF): $dx/dt = f_\theta(x,t)$.
- Exact log-det via Hutchinson trace estimator.
- Flow Matching (2022): Simpler training for CNFs — straight-line trajectories.

Applications

• Density estimation: Anomaly detection (any outlier has low likelihood).
• Image generation: Glow (OpenAI, 2018) — high-quality image generation with flows.
• Variational inference: Richer posteriors than diagonal Gaussian.
• Protein structure: Boltzmann generators for molecular conformations.

Normalizing flows are the theoretically elegant solution for exact generative modeling — their tractable likelihood makes them uniquely suited for scientific applications requiring probability estimation, though diffusion models have superseded them for image generation quality.