Loading…
Glossary term
An optimization method that repeatedly adjusts parameters in the direction that reduces the loss.
Picture standing on a hilly landscape in fog, where height is the loss. Gradient descent feels which way is downhill and takes a small step that way, again and again, until it reaches a low point.
The size of each step is the learning rate. Variants such as stochastic gradient descent and Adam estimate the slope from small batches of data.
Minimize (w - 3)² by gradient descent
w, lr = 0.0, 0.1
for step in range(50):
grad = 2 * (w - 3) # slope of (w - 3)^2
w -= lr * grad # step downhill
print(round(w, 3)) # about 3.0Gradient descent is the engine behind almost all neural network training.