dueling dqn

Dueling DQN is a DQN architecture that separates the Q-function into a state value function and an advantage function — Q(s,a) = V(s) + A(s,a) - mean(A(s,cdot)), allowing the network to independently learn the value of being in a state and the relative advantage of each action.

Dueling Architecture

• Shared Backbone: Convolutional layers shared for feature extraction.
• Value Stream: Fully connected layers outputting V(s) — scalar value of the state.
• Advantage Stream: Fully connected layers outputting A(s,a) — advantage of each action relative to average.
• Combination: Q(s,a) = V(s) + A(s,a) - (1)/(|A|)∑_a' A(s,a') — centering for identifiability.

Why It Matters

• State vs. Action: Many states have similar value regardless of action — dueling architecture captures this.
• Sample Efficiency: The value stream updates for every action — more efficient learning of state values.
• Complements: Combines well with Double DQN and Prioritized Experience Replay.

Dueling DQN is separating what matters from what to do — independently learning state value and action advantages for more efficient Q-learning.