Loading…
Glossary term
Choosing the next token at random according to the model's probabilities.
Sampling picks the next token at random according to the model's probabilities, so a 60% token is chosen about 60% of the time. That is why the same prompt can give different answers each run.
Temperature, top-k and top-p shape the distribution being sampled from.
Random choice weighted by probability
import numpy as np
tokens = ["cat", "dog", "bird"]
probs = [0.6, 0.3, 0.1]
print(np.random.choice(tokens, p=probs))Sampling is what makes chat models varied and creative rather than identical every time.