layer normalization
/ˈleɪər ˌnɔːrməlaɪˈzeɪʃən/
a technique to stabilize training by normalizing activations across features
layer normalization in a sentence
“Layer normalization helps transformers train more stably on long sequences.”
Origin of layer normalization
Latin norma carpenter's square, rule + -ization
Related Words
token
a unit produced by a tokenizer for model processing; it may represent a word, subword, character, punctuation mark, or byte sequence
tokenization
the process of breaking text into tokens for model processing
attention mechanism
a mechanism that weights relationships among token positions; causal language models normally restrict each position to earlier context
transformer
the neural network architecture underlying modern LLMs, based on self-attention
autoregressive generation
producing output one token at a time, where each token depends on all previous tokens
context window
the finite token budget a model can process for a request, including instructions, inputs, prior turns, tool data, and generated output