Deep Q-Network Algorithm

The deep Q-network algorithm is a reinforcement learning method that combines Q-learning, which learns the expected long-term value of taking a given action in a given state, with a deep convolutional neural network that estimates those values directly from raw input such as image pixels, extending Q-learning to problems with far too many possible states to store in an ordinary lookup table. Training a neural network this way is normally unstable, since reinforcement learning can diverge when a nonlinear function approximator is used to represent the values being learned, so the algorithm adds two stabilizing techniques, experience replay, which stores past transitions and trains on random samples drawn from that stored history rather than only the most recent transition, breaking the strong correlation between consecutive experiences, and a separate target network, whose parameters are copied over from the main network only periodically rather than every step, which keeps the values the network is being trained toward from shifting on every single update. Researchers at Google DeepMind developed the deep Q-network, patenting the approach in 2014, and demonstrated it playing a range of Atari 2600 video games directly from screen pixels at a level competitive with expert human players.