Dueling Network Architecture for Deep Reinforcement Learning

Ziyu Wang et al.
Published: 5 April, 2016
https://doi.org/10.48550/arXiv.1511.06581

Introduction

This paper talks about a new neural network architecture for model-free reinforcement learning. Essentially this paper introduces a new technique to calculate the Q-value predicted by a reinforcement learning model by separating the estimation of state values and (state-dependent) action advantages. In the traditional method proposed by Minh et al., the DQN uses a single stream architecture to compute the Q-value. Many papers before this have attempted to improvise the underlying RL algorithms itself but this paper is taking a different approach and improving the fully connected layers that come later while calculating Q-values. This is why this approach claims to be model-free, meaning any reinforcement learning model can be plugged into this architecture.

Main Idea

The goal of this experiment is that an RL agent with the same structure and hyper-parameters must be able to play 57 different games by observing pixels and game scores only. The SOTA model before this was the DDQN proposed by Minh et al. (2016) which was outperformed by the Prioritized replay model proposed by Schaul et al. (2016). So the comparison of both these models plugged into the dueling architecture is done in this paper.

As of the background for this paper, earlier methods of how the advantage function is discussed.

$$ Aπ (s, a) = Qπ (s, a) − V π (s) \ ⇒ Qπ (s, a) = V π (s) + Aπ (s, a) $$

Here,

The value function V measures how good it is to be in a particular state s
Q measures the value of choosing a particular action when in this state
The Advantage function (A) subtracts the value of the state from the Q function to obtain the relative importance of each action

The key idea behind the dueling architecture is that, for many states it is unnecessary to estimate the value of each action choice. In some states it is of extreme importance to know what action to take, but in many other states the choice of action has no impact on what happens.

Since V and A are calculated separately, a thoughtful design is necessary to combine the output of two streams. Upon experimenting with different methods that act as regularization methods to compute Q value, the Q value is finally computed using the equation

$$ Q(s, a; θ, α, β) = V (s; θ, β) + (A(s,a;θ,α)− 1/|A|\sum_{i=a'}A(s,a';θ,α)) $$

Both advantage and the value stream propagate gradients to the last convolutional layer in the backward propagation, the combined gradient is rescaled by 1/√2. Along with this, the gradients are clipped to have norm less than or equal to 10.

Result

Different comparisons between the Duel clipped architecture and Single Clip and bare Single architectures are made with both uniform (Minh et al., 2015) and prioritized replay (Schaul et al., 2016) methods.

These architectures were tested on Arcade Learning Environment (ALE) that consists of 57 games, under two different conditions, first one being 30 no-ops (random starting positions) and Human Starts (agents are evaluated only on rewards gained after the human starts). In both the cases (uniform and prioritized replay), the duel architecture outperformed the Single Clip and Single architectures.

Duel Clip vs Single Clip: 43 / 57 (75.4%)

Duel Clip vs Single: 46 / 57 (80.7)

Conclusion

In conclusion, the prioritized duel agent performs significantly better than both the prioritized baseline agents and the dueling agent alone and hence was the SOTA.

Hold on