KV cache
/ˌkeɪ ˈviː ˌkæʃ/
cached key-value pairs from previous tokens to speed up autoregressive generation
KV cache in a sentence
“The KV cache avoids recomputing attention for earlier tokens at each step.”
Origin of KV cache
Key-Value cache, from database terminology
Related Words
inference
the process of using a trained model to generate predictions or outputs
temperature
a sampling parameter that flattens or sharpens the next-token probability distribution
sampling
randomly selecting the next token from the probability distribution rather than always choosing the most likely
beam search
a search algorithm that explores multiple candidate sequences simultaneously
greedy decoding
always selecting the highest probability token at each step
top-k sampling
sampling only from the k most likely next tokens