Loading…
Glossary term
Stored intermediate attention results that let a model avoid recomputing earlier tokens while generating.
When generating, each new token needs the attention keys and values of every earlier token. Rather than recompute them all each step, the model stores them in a key-value cache and only computes the new token's entries.
The cache grows with the length of the conversation and can take a lot of GPU memory.
The idea in a few lines
cache_k, cache_v = [], []
def step(new_k, new_v, attend):
cache_k.append(new_k) # store this token's key and value once
cache_v.append(new_v)
return attend(cache_k, cache_v) # reuse all earlier onesKV caching is what makes text generation fast, and its memory use is a main limit on long contexts and many simultaneous users.