Loading…
Glossary term
Information added to a transformer's input so it knows the order of tokens.
Attention by itself treats a sentence as an unordered bag of tokens. Positional encoding adds information about each token's position, so 'dog bites man' differs from 'man bites dog'.
The original transformer used sine and cosine waves of different frequencies; many newer models use rotary or learned position schemes.
Position 0, 1, 2, 3 is encoded and added to each token's embedding.
Sinusoidal positional encoding
import numpy as np
def positional_encoding(seq_len, d):
pos = np.arange(seq_len)[:, None]
i = np.arange(d)[None, :]
angle = pos / np.power(10000, (2 * (i // 2)) / d)
pe = np.zeros((seq_len, d))
pe[:, 0::2] = np.sin(angle[:, 0::2])
pe[:, 1::2] = np.cos(angle[:, 1::2])
return peWithout position information a transformer could not understand word order.