Loading…
Glossary term
A measure of how well a language model predicts text; lower values mean better prediction.
Perplexity measures how surprised a language model is by real text. It is the exponential of the average negative log-probability the model gave to each actual token. Lower is better: the model was less surprised.
A perplexity of 10 roughly means the model is as uncertain as choosing among 10 equally likely words.
Perplexity from token probabilities
import math
probs = [0.2, 0.5, 0.1, 0.4] # model's probability of each real token
nll = -sum(math.log(p) for p in probs) / len(probs)
print(math.exp(nll)) # lower is betterIt is a standard way to compare language models during training, though it does not capture usefulness on tasks.