Glossary term

LLM-as-a-judge

Using a language model to grade or compare the outputs of other models.

How it works

Instead of paying people to grade every answer, a strong language model is given a rubric and asked to score or compare responses. It is fast and cheap and scales to thousands of cases.

Judges have biases, such as favoring longer answers or their own style, so they are checked against human ratings.

LLM-as-a-judge

Step 1Answer(s) to grade
Step 2Rubric + judge model
Step 3Score or winner
Step 4Spot-check against humans

In code

A judging prompt

You are grading two answers to the same question.
Criteria: accuracy, clarity, completeness.
Reply with "A" or "B" and one sentence of reasoning.

Question: {question}
Answer A: {answer_a}
Answer B: {answer_b}

Why it matters

It makes large-scale evaluation practical, with the caveat that the judge itself must be validated.

Related terms

Evaluation Leaderboard Sycophancy