evals
/ɪˈvælz/
systematic tests to measure model performance on specific tasks
evals in a sentence
“Running evals after every prompt change ensures no regressions.”
Origin of evals
Short for evaluations; French évaluer find the value
Related Words
LLM-as-a-Judge
using a strong LLM to evaluate the outputs of another model
ground truth
a trusted reference label or answer used for evaluation, which may itself contain uncertainty or annotation error
tracing
recording the flow of execution and data through a complex system
hallucination rate
the frequency with which a model generates incorrect information
benchmark
a standardized test used to compare performance
golden dataset
a hand-verified set of examples used as the standard for judging model output