Loading…
Glossary term
A property of models or data in which most values are zero, which can be exploited to save computation.
Sparsity means most values are zero. A sparse model, or sparse activations, can in principle skip zero computations and save memory and time. Mixture-of-experts models use a form of sparsity by activating only some experts.
Realizing the savings needs hardware and software that exploit the zeros.
| Col 1 | Col 2 | Col 3 | Col 4 | |
|---|---|---|---|---|
| Row 1 | 0 | 0 | 7 | 0 |
| Row 2 | 3 | 0 | 0 | 0 |
| Row 3 | 0 | 0 | 0 | 5 |
Measure sparsity
import numpy as np
w = np.array([[0, 0, 7, 0], [3, 0, 0, 0], [0, 0, 0, 5]])
print((w == 0).mean()) # 0.75It offers a route to more capable models at lower compute cost.