LLM-as-a-Judge
/ˌeɫ eɫ ˈem æz ə ˈdʒʌdʒ/
using a strong LLM to evaluate the outputs of another model
LLM-as-a-Judge in a sentence
“LLM-as-a-Judge scales evaluation better than human review.”
Origin of LLM-as-a-Judge
Industry term (Zheng et al., 2023)
What does LLM-as-a-Judge really mean?
LLM-as-a-Judge is the practice of using one language model to grade another's output — scoring answers for accuracy, helpfulness, or style against a rubric. It exists because the old ways don't scale: human review is slow and expensive, and string-matching metrics can't tell a good paragraph from a bad one. The judge reads like a person, at the price and speed of software.
The story behind LLM-as-a-Judge
The term settled into AI engineering around 2023, popularized by the MT-Bench and Chatbot Arena work that showed strong models agreeing with human raters about as often as humans agree with each other. It filled the gap left by metrics like BLEU and ROUGE, which were built for translation overlap, not open-ended judgment.
How to use LLM-as-a-Judge
Use it for the evaluation pattern, usually hyphenated and capitalized as LLM-as-a-Judge: "we score regressions with an LLM-as-a-Judge pipeline." Mention its known biases when precision matters — judges favor longer answers (verbosity bias), the first option shown (position bias), and their own writing style (self-preference). The standard mitigations: a written rubric, randomized ordering, and spot-checks against human ratings.
Related Words
ground truth
a trusted reference label or answer used for evaluation, which may itself contain uncertainty or annotation error
tracing
recording the flow of execution and data through a complex system
hallucination rate
the frequency with which a model generates incorrect information
benchmark
a standardized test used to compare performance
golden dataset
a hand-verified set of examples used as the standard for judging model output
rubric
an explicit scoring guide that turns judgment into repeatable criteria