Loading…
Glossary term
Learning in which the training signal comes from the data itself, such as predicting the next word in a sentence.
In self-supervised learning the labels come from the data itself. A language model hides the next word and learns to predict it; an image model hides a patch and learns to fill it in.
Since no human labelling is needed, it can use enormous amounts of raw text, images and audio.
The model sees the earlier tokens and must predict the hidden next one. The text itself supplies the answer.
This is how large language models are pretrained, and it is the reason they could scale so far.