Loading…
Glossary term
Splitting a model into stages across devices so different batches flow through them like an assembly line.
The model is split into consecutive stages, each on a different device. Batches are cut into micro-batches that flow through the stages like an assembly line, so all devices work at once.
Some idle time, called the pipeline bubble, remains at the start and end.
It lets very deep models be trained across many devices with modest communication.