Loading…
Glossary term
Attention applied within a single sequence, so each element can relate to every other element.
In self-attention the queries, keys and values all come from the same sequence, so every token can gather information from every other token in the same text.
Multi-head self-attention runs several attention operations in parallel, letting the model track different kinds of relationships at once.
| the | cat | sat | |
|---|---|---|---|
| the | 0.2 | 0.5 | 0.3 |
| cat | 0.1 | 0.6 | 0.3 |
| sat | 0.1 | 0.7 | 0.2 |
It is how a transformer builds context-aware representations of each word.