nucleus sampling
/ˈnjuːkliəs ˌsæmplɪŋ/
sampling from tokens comprising the top cumulative probability mass (top-p)
nucleus sampling in a sentence
“Nucleus sampling with p=0.95 adapts vocabulary size to context certainty.”
Origin of nucleus sampling
Latin nucleus kernel from nux nut
Related Words
logits
raw, unnormalized scores output by the model before conversion to probabilities
softmax
a function that converts logits into a probability distribution summing to one
KV cache
cached key-value pairs from previous tokens to speed up autoregressive generation
inference
the process of using a trained model to generate predictions or outputs
temperature
a sampling parameter that flattens or sharpens the next-token probability distribution
sampling
randomly selecting the next token from the probability distribution rather than always choosing the most likely