neural radiance field
Neural Radiance Fields (NeRF) are neural networks that represent 3D scenes as continuous volumetric functions mapping spatial coordinates and viewing direction to color and density — enabling photorealistic novel-view synthesis from a sparse set of 2D photographs by training a network to predict what any point in 3D space looks like from any angle.
How NeRF Works
1. Input: 5D coordinates — 3D position (x, y, z) + 2D viewing direction (θ, φ).
2. Network: MLP (8 layers, 256 units) outputs color (r, g, b) and volume density σ.
3. Volume Rendering: Cast rays from camera through each pixel, sample points along each ray.
4. Color Integration: $C(r) = \sum_{i=1}^{N} T_i (1 - \exp(-\sigma_i \delta_i)) c_i$ where $T_i = \exp(-\sum_{j<i} \sigma_j \delta_j)$.
- T_i: Accumulated transmittance (how much light reaches this point).
- σ_i · δ_i: Opacity contribution of sample i.
5. Training: Minimize photometric loss between rendered pixels and ground truth images.
Key Innovations
Positional Encoding
- Raw (x,y,z) inputs can't represent high-frequency details.
- Map coordinates through sinusoidal functions: $\gamma(p) = [\sin(2^0\pi p), \cos(2^0\pi p), ..., \sin(2^{L-1}\pi p), \cos(2^{L-1}\pi p)]$.
- L=10 for position, L=4 for direction — enables sharp edges and fine textures.
Hierarchical Sampling
- Coarse network: Sample uniformly along ray → identifies regions with high density.
- Fine network: Concentrate samples in important regions → better quality, same cost.
NeRF Evolution
| Model | Speed | Quality | Key Innovation |
|---|---|---|---|
| NeRF (2020) | Hours to train, seconds to render | Excellent | Original formulation |
| Instant-NGP (2022) | Minutes to train | Excellent | Hash-based feature grids |
| 3D Gaussian Splatting (2023) | Minutes to train, real-time render | Excellent | Explicit Gaussian primitives |
| Zip-NeRF | Hours | SOTA quality | Anti-aliased + grid-based |
| Nerfacto (Nerfstudio) | ~30 min | Very good | Best practices combined |
3D Gaussian Splatting (Successor)
- Represents scene as millions of 3D Gaussian ellipsoids instead of an implicit field.
- Each Gaussian has: position, covariance (shape), opacity, spherical harmonics (color).
- Rendering: Project (splat) Gaussians onto image plane — differentiable rasterization.
- 100-1000x faster rendering than NeRF — enables real-time novel view synthesis.
Applications
- Virtual/augmented reality content creation.
- Autonomous driving simulation (reconstruct scenes from dashcam footage).
- Cultural heritage digitization.
- E-commerce product visualization.
NeRF and its successors are revolutionizing 3D content creation — the ability to reconstruct photorealistic 3D scenes from ordinary photographs has applications spanning entertainment, robotics, simulation, and spatial computing.