Loading…
Glossary term
An architecture that routes each input to a subset of specialized sub-networks to save compute.
A mixture-of-experts model replaces one big feed-forward layer with many smaller 'experts'. A small router looks at each token and sends it to just a few experts, for example the top two out of eight.
That is why such models quote two sizes: total parameters (all experts) and active parameters (those used for a given token).
A router picks the top 2 of 8 experts per token
import torch
router_logits = torch.randn(4, 8) # 4 tokens, 8 experts
probs = router_logits.softmax(dim=-1)
weights, experts = probs.topk(2, dim=-1) # each token chooses 2 experts
print(experts)It lets a model hold a lot of knowledge while using only a fraction of its compute per token, which cuts the cost of running very large models.