Glossary term

Learning rate

A setting that controls how large each parameter update is during training.

How it works

The learning rate scales every weight update. Too high and training overshoots and may diverge; too low and it crawls or gets stuck.

Schedules often start with a short warm-up, then lower the rate over time.

Learning rate too high vs too low

Too high

Loss bounces or explodes
Overshoots the minimum

Too low

Loss falls very slowly
May stall before it is good

In code

Set a learning rate in PyTorch

import torch

optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

Why it matters

It is widely considered the single most important hyperparameter to get right.

Related terms

Gradient descent Hyperparameter Training