curiosity-driven learning

Curiosity-Driven Learning is a specific form of intrinsic motivation where the agent is rewarded for encountering situations that are difficult to predict — the agent's curiosity reward is the prediction error of a forward dynamics model, driving it toward novel, surprising states.

ICM (Intrinsic Curiosity Module)

• Forward Model: Predicts next state features: hat(phi)(s_t+1) = f(phi(s_t), a_t).
• Curiosity Reward: r_i = |hat(phi)(s_t+1) - phi(s_t+1)|^2 — prediction error = surprise.
• Feature Space: Predict in a learned feature space, not raw pixels — avoids the "noisy TV" problem.
• Inverse Model: Predict action from consecutive states — ensures the feature space captures actionable information.

Why It Matters

• No Reward Needed: The agent explores effectively driven purely by curiosity — no external reward required.
• Game Playing: Curiosity-driven agents learn to play Atari games with zero external reward — remarkable emergent behavior.
• Transfer: Curiosity-learned representations transfer to downstream tasks.

Curiosity-Driven Learning is exploring the unpredictable — rewarding the agent for encountering states it cannot yet predict.