pass@k
/ˌpɑːs ət ˈkeɪ/
the probability that at least one of k sampled attempts is correct
pass@k in a sentence
“The coding model hit 90% pass@10 but only 55% pass@1.”
Origin of pass@k
coined for code-generation evaluation (Chen et al., 2021)
Related Words
drift
gradual change in inputs or behavior that quietly erodes performance
error analysis
systematically reading failures to find the patterns worth fixing
perplexity
a measure of how surprised a model is by text; lower means better prediction
benchmark saturation
when models max out a test so it can no longer tell them apart
observability
the degree to which a system's inner behavior can be inferred from what it emits
verbosity bias
a judge's tendency to score longer answers higher regardless of quality