Loading…
Glossary term
A method that lets a model weigh which parts of its input are most relevant when producing each output.
Attention lets a model decide which parts of its input to focus on when producing each output. For each position it computes scores against every other position, turns the scores into weights, and takes a weighted mix of the information.
In a sentence like 'the trophy didn't fit in the case because it was too big', attention helps connect 'it' to the right noun.
Attention in NumPy
import numpy as np
def softmax(x):
e = np.exp(x - x.max(axis=-1, keepdims=True))
return e / e.sum(axis=-1, keepdims=True)
def attention(Q, K, V):
scores = Q @ K.T / np.sqrt(K.shape[-1]) # how relevant is each key?
weights = softmax(scores) # turn scores into weights
return weights @ V # weighted mix of valuesAttention is the key idea in transformers and the reason language models can use context across long passages.