benchmark saturation
/ˈbentʃmɑːrk ˌsætʃəˈreɪʃən/
when models max out a test so it can no longer tell them apart
benchmark saturation in a sentence
“Benchmark saturation forced the lab to commission a harder eval.”
Origin of benchmark saturation
benchmark: a surveyor's mark cut in stone + Latin saturare to fill
Related Words
observability
the degree to which a system's inner behavior can be inferred from what it emits
verbosity bias
a judge's tendency to score longer answers higher regardless of quality
evals
systematic tests to measure model performance on specific tasks
LLM-as-a-Judge
using a strong LLM to evaluate the outputs of another model
ground truth
the actual absolute truth or correct answer used for comparison
tracing
recording the flow of execution and data through a complex system