Loading…
Glossary term
A neural network in which information moves in one direction, from input to output.
In general, a feed-forward network sends information in one direction from input to output, with no loops. Inside a transformer block, the feed-forward part is a small two-layer network applied to each token separately.
It holds a large share of a transformer's parameters and is thought to store much of its knowledge.
In a transformer, attention mixes information between tokens and the feed-forward layer processes each token, so the two work as a pair.