Loading…
Glossary term
A decoding method that picks the next token from the smallest set whose probabilities add up to a chosen value.
Also called nucleus sampling. Sort tokens by probability and keep the smallest group whose probabilities add up to p, for example 0.9. The next token is chosen only from that group.
Unlike a fixed number of candidates, the group grows when the model is unsure and shrinks when it is confident.
Nucleus (top-p) filtering
import numpy as np
def top_p(probs, p=0.9):
order = np.argsort(probs)[::-1]
cum = np.cumsum(probs[order])
keep = order[: np.searchsorted(cum, p) + 1]
out = np.zeros_like(probs)
out[keep] = probs[keep]
return out / out.sum()It trims the long tail of unlikely tokens, keeping text varied but coherent.