A Simple Mental Model for Gradient Descent

Gradient descent becomes easier to reason about with a simple linear regression example.

Suppose we have several points on an $x$-$y$ graph and want to find the straight line that best fits them.

For a candidate line, we can measure how well it fits using the sum of squared residuals:

The smaller the loss, the better the fit.

From a line to a loss curve

To keep the example simple, assume the slope of the line is fixed and we only want to find the best y-intercept, $b$.

We can try different values of $b$, calculate the loss for each one, and plot:

• x-axis: intercept $b$
• y-axis: loss

For linear regression, this gives us a bowl-shaped loss curve. Our goal is to find the point at the bottom of that curve.

The derivative tells us which way to move

At any point on the loss curve, the derivative tells us its slope.

• A positive derivative means the loss increases as we move to the right.
• A negative derivative means the loss decreases as we move to the right.
• At the minimum, the curve is flat:

Gradient descent updates the intercept in the opposite direction of the gradient:

Here, $\eta$ is the learning rate, which controls how large each step is.

When the loss curve is steep, the gradient is larger, so the update tends to be larger. As we get closer to the minimum, the gradient becomes smaller, so the steps become smaller too.

The mental model

Gradient descent can be thought of as:

Look at the slope where you currently are, take a controlled step downhill, and repeat.

The loss function defines the landscape, the gradient tells us which direction is uphill, and gradient descent moves in the opposite direction toward lower loss.