Loading…
Glossary term
A finding that, for a fixed compute budget, model size and training data should grow together.
A 2022 study from DeepMind found many large models were undertrained: for a fixed compute budget, model size and the number of training tokens should grow together, roughly in proportion.
A smaller model trained on more data can beat a bigger one trained on less.
It changed how labs allocate compute, favoring more data per parameter. Many modern models are trained on far more tokens than the original rule suggests so they are cheaper to run.