Neural Computation Vocabulary

a neural network architecture using self-attention for sequence processing

“Transformers revolutionized natural language processing.”

a dense vector representation of discrete items like words

“Word embeddings capture semantic relationships in vector space.”

a technique allowing models to focus on relevant parts of input

“Attention mechanisms let the model weigh which words matter most.”

a compressed representation where similar items are close together

“In latent space, semantically similar concepts cluster together.”

a unit of text (word, subword, or character) processed by a model

“The model processes text as a sequence of tokens.”

a learnable parameter that determines connection strength in a network

“Training adjusts weights to minimize prediction errors.”

the output of a neuron after applying a non-linear function

“ReLU activation introduces non-linearity to the network.”

the direction and rate of steepest increase of a function

“Backpropagation computes gradients to update weights.”

using a trained model to make predictions on new data

“Inference is computationally cheaper than training.”

adapting a pre-trained model for a specific task

“Fine-tuning on medical texts improved diagnostic accuracy.”

the maximum amount of text a model can process at once

“Longer context windows enable understanding of full documents.”

a function converting raw scores into a probability distribution

“Softmax ensures the output probabilities sum to one.”