Why machine learning models learn faster by looking at less data

Why machine learning models learn faster by looking at less data
Image: Intellect (generated)

In short: Stochastic gradient descent is the engine behind modern machine learning. Instead of calculating the perfect path to an answer using an entire dataset, it takes a shortcut by estimating from a single random sample. This trade-off sacrifices precision for speed, allowing models to train on massive, complex datasets efficiently.

At its core, stochastic gradient descent (SGD) is an optimization method designed to minimize an objective function. In classical batch gradient descent, the algorithm calculates the gradient of the entire dataset before making a single update. When dealing with enormous datasets, this process becomes computationally prohibitive. SGD solves this by approximating the true gradient using only a randomly selected subset of data, or even a single observation, at each iteration. This significantly reduces the computational burden, enabling faster iterations even if the convergence rate is lower.

The intellectual roots of this approach trace back to the 1950s. Herbert Robbins and Sutton Monro introduced early stochastic approximation methods in 1951, which laid the groundwork for the field. Shortly after, in 1952, Jack Kiefer and Jacob Wolfowitz developed an optimization algorithm using central differences that closely resembled modern SGD. By the late 1950s, Frank Rosenblatt successfully applied these concepts to optimize his perceptron model, marking the first time the technique was used for neural networks.

Modern implementations often use a 'mini-batch' approach, which sits between the extremes of batch gradient descent and single-sample SGD. By averaging the gradient over a small group of samples, this method allows for the use of vectorization libraries, resulting in smoother convergence. The behavior of SGD also changes based on the model's complexity. In overparameterized scenarios, it converges to an interpolation solution with the minimum distance from the starting point, even with a constant learning rate. In underparameterized cases, however, a constant learning rate prevents convergence. Under relatively mild assumptions, the algorithm is guaranteed to converge to a global minimum for convex functions, or a local minimum otherwise, a result supported by the Robbins-Siegmund theorem.

Opens the Intellect+ feed with this card first. Swipe up for the next one.

Sources

To cite this card, link to https://intellectplus.co.uk/app/science-stochastic-gradient-descent.

How we source, check and correct cards: Editorial standards.

General information, not professional advice. Spotted an error? Flag the card or see corrections and complaints.

Intellect Plus is a visual educational discovery platform for exploring fascinating facts and ideas. Brain Food, Not Brain Rot. What is Intellect Plus?

Related

Finding something worth knowing…