context caching
/ˈkɒntekst ˌkæʃɪŋ/
saving the processed state of a prompt prefix to avoid recomputing it
context caching in a sentence
“Context caching is ideal for chatting with long documents.”
Origin of context caching
Latin contextus + caching
Related Words
quantization
reducing the precision of model weights (e.g., to 4-bit) to save memory
LoRA
Low-Rank Adaptation; fine-tuning only a small subset of parameters
distillation
training a smaller 'student' model to mimic a larger 'teacher' model
Mixture of Experts
using multiple specialized sub-models (experts) and routing tokens to them
speculative decoding
using a small model to draft tokens for verification by a large model
KV cache
storing attention calculations to speed up generation