autoregressive generation
/ˌɔːtoʊrɪˈɡresɪv ˌdʒenəˈreɪʃən/
producing output one token at a time, where each token depends on all previous tokens
autoregressive generation in a sentence
“Autoregressive generation means the model can't revise earlier words once they're written.”
Origin of autoregressive generation
Greek auto- self + Latin regressus return + generare to produce
Related Words
context window
the finite amount of text a model can process at once, including input and output
embedding
a dense vector representation of text in high-dimensional space where similar concepts are geometrically close
latent space
the high-dimensional space where neural networks represent concepts as directions and positions
feed-forward layer
neural network layers that process each position independently after attention
layer normalization
a technique to stabilize training by normalizing activations across features
token
a sub-word unit that language models process, rather than whole words or characters