Loading…
Glossary term
Splitting the math inside individual layers across devices.
Tensor parallelism splits the computation inside a single layer, for example a giant matrix multiplication, across several devices. Each computes a part and the pieces are combined.
It needs very fast links between devices because they exchange data in every layer.
It helps run and train layers too large for one chip, and is often used inside a server with NVLink.