nucleus sampling
/ˈnjuːkliəs ˌsæmplɪŋ/
sampling from tokens comprising the top cumulative probability mass (top-p)
nucleus sampling in a sentence
“Nucleus sampling with p=0.95 adapts vocabulary size to context certainty.”
Origin of nucleus sampling
Latin nucleus kernel from nux nut
Related Words
logits
raw, unnormalized scores output by the model before conversion to probabilities
softmax
a function that converts logits into a probability distribution summing to one
KV cache
cached key-value pairs from previous tokens to speed up autoregressive generation
inference
the process of using a trained model to generate predictions or outputs
temperature
a parameter controlling randomness in generation—higher means more creative, lower means more deterministic
sampling
randomly selecting the next token from the probability distribution rather than always choosing the most likely