feed-forward layer
/ˌfiːd ˈfɔːrwərd ˌleɪər/
neural network layers that process each position independently after attention
feed-forward layer in a sentence
“Feed-forward layers transform the attention outputs into richer representations.”
Origin of feed-forward layer
Old English fēdan to nourish + Latin forward + layer
Related Words
layer normalization
a technique to stabilize training by normalizing activations across features
token
a unit produced by a tokenizer for model processing; it may represent a word, subword, character, punctuation mark, or byte sequence
tokenization
the process of breaking text into tokens for model processing
attention mechanism
a mechanism that weights relationships among token positions; causal language models normally restrict each position to earlier context
transformer
the neural network architecture underlying modern LLMs, based on self-attention
autoregressive generation
producing output one token at a time, where each token depends on all previous tokens