Loading…
Glossary term
A common tokenization method that builds a vocabulary by repeatedly merging frequent pairs of characters.
BPE builds a vocabulary by starting with single characters or bytes and repeatedly merging the most frequent adjacent pair into a new token. After many merges, common words and word parts become single tokens.
Working at the byte level means any text can be encoded, including emoji and unusual scripts.
One BPE merge step
from collections import Counter
words = [list("lower"), list("lowest"), list("low")]
pairs = Counter((a, b) for w in words for a, b in zip(w, w[1:]))
best = pairs.most_common(1)[0][0] # most frequent pair, e.g. ('l', 'o')BPE is the basis of the tokenizers used by many leading language models.