Stochastic Gradient Descent Algorithm

Stochastic gradient descent, often abbreviated SGD, is an iterative method for optimizing an objective function with suitable smoothness properties, and can be regarded as a stochastic approximation of ordinary gradient descent that replaces the true gradient, calculated from the entire data set, with an estimate calculated from a randomly selected subset of the data. This substitution reduces the heavy computational burden of high-dimensional optimization problems, trading a lower convergence rate for much faster iterations. Its basic idea traces back to the Robbins-Monro algorithm of the 1950s, and it has become an important optimization method in machine learning.