conservative q-learning

CQL (Conservative Q-Learning) is the leading offline RL algorithm that learns a conservative (lower-bound) Q-function — penalizing Q-values for out-of-distribution actions to prevent the overestimation that causes standard Q-learning to fail in the offline setting.

CQL Algorithm

• Conservative Penalty: Minimize Q-values under the current policy: min_Q alpha E_a ~ pi[Q(s,a)].
• Data Support: Maximize Q-values for actions in the dataset: max_Q E_a ~ D[Q(s,a)].
• Combined: L_CQL = alpha (E_pi[Q] - E_D[Q]) + L_TD — pushes down OOD, pushes up in-distribution.
• Lower Bound: The learned Q-function is a provable lower bound of the true Q-function — conservative but safe.

Why It Matters

• No Overestimation: CQL prevents the catastrophic overestimation of out-of-distribution actions that kills standard offline Q-learning.
• Provable: CQL provides provable lower bounds on the policy's true value — safe deployment guarantees.
• State-of-Art: CQL is one of the strongest offline RL baselines — works across many domains.

CQL is pessimistic Q-learning — learning a deliberately conservative value function for safe, reliable offline RL.