LoRA
/ˈlɔːrə/
Low-Rank Adaptation; fine-tuning only a small subset of parameters
LoRA in a sentence
“LoRA makes fine-tuning large models computationally affordable.”
Origin of LoRA
Acronym: Low-Rank Adaptation (Hu et al., 2021)
Related Words
distillation
training a smaller 'student' model to mimic a larger 'teacher' model
Mixture of Experts
using multiple specialized sub-models (experts) and routing tokens to them
speculative decoding
using a small model to draft tokens for verification by a large model
KV cache
storing attention calculations to speed up generation
context caching
saving the processed state of a prompt prefix to avoid recomputing it
quantization
reducing the precision of model weights (e.g., to 4-bit) to save memory