Glossary term

Loss function

A measure of how wrong a model's predictions are, which training tries to minimize.

How it works

A loss function turns 'how wrong is the model?' into a single number. Training tries to make that number as small as possible.

Different tasks use different losses: mean squared error for predicting numbers, cross-entropy for classification and for next-token prediction.

Loss falls as the model learns

In code

Mean squared error by hand

import numpy as np

y_true = np.array([3.0, 5.0, 2.0])
y_pred = np.array([2.5, 5.5, 1.0])
loss = np.mean((y_true - y_pred) ** 2)
print(loss)   # 0.5

Why it matters

The loss defines what the model is actually being trained to do, so choosing it badly can optimize the wrong thing.

Related terms

Gradient descent Mean squared error Training