Loading…
Glossary term
The component that splits text into tokens and converts them to numbers a model can process.
A tokenizer breaks text into tokens and maps each to an integer ID that the model can process. It also does the reverse to turn model output back into text.
Because tokenizers are built from patterns in training text, common words are one token while rare words and other languages may take several.
Real tokenizers differ, but all split text into reusable chunks.
Tokenize with tiktoken
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
ids = enc.encode("Tokenization splits text.")
print(ids, len(ids))
print(enc.decode(ids))Tokens determine cost and context limits, and tokenization quirks explain some odd model mistakes, such as with counting letters.