Loading…
Glossary term
Training with a combination of lower- and higher-precision numbers to save memory and time.
Mixed-precision training does most calculations in a compact 16-bit format for speed and memory, while keeping a few sensitive values, like the master copy of the weights, in 32-bit. This roughly doubles speed and saves memory with little quality loss. It relies on hardware such as tensor cores.
Mixed precision in PyTorch
import torch
with torch.autocast(device_type="cuda", dtype=torch.bfloat16):
loss = model(x).sum()It is standard practice for training modern neural networks efficiently.