pre-training
initial training on vast text data to learn language patterns before task-specific fine-tuning
pre-training in a sentence
“Pre-training on internet text gives the model broad world knowledge.”
Origin of pre-training
Latin prae- before + Old French trainer to draw, drag
Related Words
fine-tuning
additional training on specific data to adapt a pre-trained model for particular tasks
RLHF
reinforcement learning from human feedback—training models using human preference judgments
supervised learning
training on labeled examples where correct outputs are provided
self-supervised learning
training where labels are derived from the data itself, like predicting masked words
loss function
a mathematical measure of how wrong the model's predictions are, minimized during training
gradient descent
an optimization algorithm that iteratively adjusts parameters to minimize loss