Actor-Critic Algorithm

The actor-critic algorithm is a family of reinforcement learning methods that combines a policy-based component, called the actor, which selects actions according to a learned policy, with a value-based component, called the critic, which learns to estimate how good those actions are and uses that estimate to guide the actor's updates. Training the actor directly from the rewards it receives, the way a pure policy gradient method such as REINFORCE does, produces gradient estimates with very high variance; the critic reduces this variance by providing a baseline, an estimate of a state's or an action's expected value, against which the actual outcome can be compared, so the actor is pushed toward actions that did better than the critic expected rather than reacting to the raw size of the reward itself, making learning more stable without introducing bias into the update. Vijay Konda and John Tsitsiklis formalized the actor-critic framework in a 1999 paper, and different variants since have applied it to both continuous and discrete action spaces and used it in either an on-policy or an off-policy form.