Loading…
Glossary term
Low-rank adaptation: a parameter-efficient method that adds small trainable matrices to a frozen model.
Low-rank adaptation freezes the original weight matrix W and learns two small matrices, A and B, whose product is added to it. Only A and B are trained, so far fewer numbers change.
After training, the adapter can be kept separate or merged into the base model.
LoRA with the PEFT library
from peft import LoraConfig, get_peft_model
config = LoraConfig(r=8, lora_alpha=16, target_modules=["q_proj", "v_proj"], lora_dropout=0.05)
model = get_peft_model(base_model, config)
model.print_trainable_parameters() # a tiny fraction of the totalLoRA is why individuals and small teams can fine-tune very large models.