Proximal Policy Optimization Algorithm

Proximal policy optimization, known as PPO, is a policy gradient algorithm in reinforcement learning for training an agent to choose actions that maximize its expected reward, developed to make policy gradient training more stable when the policy is represented by a large neural network. Ordinary policy gradient methods can take an update step that is too large and pushes the policy toward a much worse region with little chance of recovery, or a step so cautious that learning becomes very slow; PPO addresses this by computing the ratio between the probability the updated policy assigns to an action and the probability the previous policy assigned to it, then clipping that ratio to a narrow range, typically limiting it to roughly plus or minus twenty percent, before using it in the training objective, which keeps any single update from moving the policy too far from where it started while still allowing larger, more efficient steps than an overly conservative method would. Researchers at OpenAI, led by John Schulman, published proximal policy optimization in 2017, building on earlier work on trust region methods, and it has since become one of the most widely used reinforcement learning algorithms because of its combination of stability and comparative simplicity to implement.