Loading…
Glossary term
Training one model across many chips or machines at the same time.
Frontier models are too big and slow to train on one chip, so training is spread across thousands. The work is divided by splitting the data, the model or both, and chips constantly synchronize their progress.
Keeping all of them busy and recovering from hardware failures is a major engineering challenge.
It is what makes training today's largest models possible, and one reason for the huge cost.