sophia

Sophia (Second-order Clipped Stochastic Optimization) is a lightweight second-order optimizer designed for large language model pre-training — using a diagonal Hessian estimate to provide per-parameter curvature information, achieving 2x faster training than Adam with minimal overhead.

How Does Sophia Work?

• Hessian Estimate: Uses a diagonal Gauss-Newton approximation or Hutchinson estimator to estimate per-parameter curvature h_t.
• Update: theta_t+1 = theta_t - eta · clip(m_t / max(h_t, gamma), 1) (gradient divided by curvature, clipped).
• Clipping: The clipping prevents excessively large steps in flat directions (where h_t ≈ 0).
• Cost: ~15% overhead over Adam (Hessian estimated every 10-20 steps).

Why It Matters

• 2x Speedup: Reaches the same loss as Adam in half the training steps for LLMs (GPT-2, LLaMA).
• Curvature-Aware: Steps more aggressively in flat directions and conservatively in sharp ones — optimal for the heterogeneous loss landscape of LLMs.
• Scalable: Designed for billion-parameter models.

Sophia is the curvature-aware optimizer for LLMs — using lightweight second-order information to navigate the loss landscape twice as fast as Adam.