Loading…
Glossary term
The memory attached to a GPU, which limits the size of model it can hold.
VRAM is the memory attached to a GPU. A model's weights must fit in it, plus working memory for activations and the KV cache, so VRAM sets the largest model a GPU can run.
A rough rule: memory for weights equals parameters times bytes per parameter.
Estimate memory for the weights
params = 7e9 # a 7B-parameter model
bytes_per_param = 2 # 16-bit
print(params * bytes_per_param / 1e9, "GB") # 14.0 GBMemory is often the limiting factor for running models locally, which is why quantization is so popular.