maximum entropy rl

Maximum Entropy RL is a reinforcement learning framework that augments the standard reward with an entropy bonus — the agent maximizes the expected reward PLUS the entropy of its policy, encouraging exploration and leading to more robust, multi-modal policies.

MaxEnt RL Objective

• Objective: $pi^* = argmax_pi \sum_t \mathbb{E}[r_t + \alpha H(pi(cdot|s_t))]$ — reward + entropy.
• Temperature ($\alpha$): Controls the trade-off between reward maximization and entropy maximization.
• Optimal Policy: $pi^*(a|s) propto \exp(Q^*(s,a) / \alpha)$ — the Boltzmann (softmax) policy.
• Soft Bellman: $V(s) = \alpha \log \sum_a \exp(Q(s,a)/\alpha)$ — the soft value function.

Why It Matters

• Exploration: High entropy prevents the policy from collapsing to a single action — maintains exploration.
• Robustness: MaxEnt policies are more robust to perturbations — they maintain multiple viable strategies.
• Foundation: The theoretical foundation for SAC (Soft Actor-Critic), one of the most successful continuous control algorithms.

MaxEnt RL is rewarding uncertainty — encouraging the agent to maintain diverse, exploratory behavior while maximizing reward.