autoregressive generation
/ˌɔːtoʊrɪˈɡresɪv ˌdʒenəˈreɪʃən/
producing output one token at a time, where each token depends on all previous tokens
autoregressive generation in a sentence
“Autoregressive generation means the model can't revise earlier words once they're written.”
Origin of autoregressive generation
Greek auto- self + Latin regressus return + generare to produce
Related Words
context window
the finite token budget a model can process for a request, including instructions, inputs, prior turns, tool data, and generated output
embedding
a dense vector representation of text in high-dimensional space where similar concepts are geometrically close
latent space
the high-dimensional space where neural networks represent concepts as directions and positions
feed-forward layer
neural network layers that process each position independently after attention
layer normalization
a technique to stabilize training by normalizing activations across features
token
a unit produced by a tokenizer for model processing; it may represent a word, subword, character, punctuation mark, or byte sequence