LLM-as-a-judge is the practice of scoring an agent's output by asking a language model to grade it against a written rubric. The judge reads evidence from the run, such as the user's request, the agent's reply and the tool calls in between, and returns a score with a written reason. It scales human judgement to every run, but only once you have checked that it agrees with the humans it stands in for.
Why judges need checking
A judge is a sensor in the agent harness: it observes what the agent did and says whether it was good enough. Sensors earn trust by being right. A judge that passes a wrong refund amount, or fails a correct answer because it dislikes the tone, produces confident numbers that point the wrong way. Dashboards, release gates and fix decisions built on those numbers inherit the error.
The fix is not a better prompt written in isolation. It is a measurement: take sessions your domain experts have already judged, run the judge on the same sessions, and count where they differ. Until that count exists, a judge's score is an opinion, not a metric.
What a judge is made of
Whatever tool you use, an LLM judge comes down to the same parts. Each one is a place where the judge can drift from what your experts mean.
| Part | What it decides | What goes wrong |
|---|---|---|
| Instructions | The task, in plain language, and the judge's persona | Vague tasks produce vague grades |
| Criteria | The rubric the model grades against | Criteria that miss what experts actually care about |
| Evidence | Which parts of the run the judge sees: request, reply, full trajectory, retrieved context, a reference answer, metadata | Grading a reply without the context it depended on |
| Output type | The scale: a 0 to 1 score, a numeric range, ordered labels, or pass and fail | A scale that does not match how experts think about the case |
| Threshold | Where a score turns into a pass or a fail | A cutoff set by guess rather than by comparison |
| Target | Whether one trace or a whole conversation is graded | Judging one turn when the failure spans several |
Two habits prevent most surprises. First, look at exactly what the model receives on a real run before you trust the rubric, because a judge graded against a raw data blob behaves differently from one graded against the readable reply. Second, test the draft on a real sample and read the reasons it gives, not only the scores.
Model judges, rule judges and composites
Not every check needs a language model. A useful judge library mixes three kinds.
- Model judges read the evidence and return a score and a reason against your instructions. They handle the judgement calls: was the answer faithful to the retrieved policy, did the agent resolve the request.
- Rule judges call a deterministic service and map a value out of its response, with no model involved. Toxicity, moderation or grammar checks, or a scoring service your own team owns, fit here.
- Composite judges combine other judges into one number, either as a weighted sum or as an arithmetic expression over their scores. A headline quality score that blends faithfulness, relevancy and coherence is a composite.
Whichever kind you use, keep each judge narrow. One judge per failure you care about is easier to check against experts than one judge that tries to grade everything at once.
Measuring agreement with experts
Agreement is measured on sessions that have expert truth: labels from the reviewers whose verdicts define quality. Pair each judge verdict with the expert label for the same session, treat failure as the case you are trying to catch, and four numbers describe the judge.
| Metric | Question it answers |
|---|---|
| Precision | When the judge flags a failure, how often is it really one? |
| Recall | Of the real failures, how many does the judge catch? |
| Specificity | How well does it avoid false alarms on good sessions? |
| Cohen's kappa | Overall agreement, corrected for chance: 0 is about random, 1 is perfect |
Kappa matters because raw agreement flatters. If most sessions pass, a judge that passes everything agrees with experts most of the time and catches nothing. Chance correction exposes that.
The disagreements themselves are the most useful output. Split them in two:
- Missed failures: the judge passed a session an expert failed. These are the costly kind, because they are the failures that reach users unnoticed.
- False alarms: the judge failed a session an expert passed. These waste reviewer time and erode trust in every other number.
Read each disagreement with the judge's reason beside the expert's critique. Sometimes the judge is wrong. Sometimes the expert was, or the case is genuinely ambiguous, and the right move is to correct the label rather than the judge.
The calibration loop
Agreement turns judge editing from guesswork into a measured cycle.
- Measure. Check precision, recall and kappa for the judge against expert labels.
- Diagnose. Read the disagreements. Is the judge missing a failure type the experts catch, or nitpicking things they do not care about?
- Edit. Change the instructions or the criteria, and save the result as a new version so older scores stay tied to the version that produced them.
- Re-run. Run the new version over the labeled sessions, compare its results with the previous version run by run, and recompute agreement.
- Repeat until the trend converges, then trust the judge at scale, including as a gate on a release.
Plot agreement over time. A rising line means your edits are converging on human judgement. A flat, low line means the rubric still does not describe what your experts consider a failure, and more prompt tuning will not fix a criteria problem.
Mark where each judge stands. A judge still being tuned is experimental and should stay out of the metrics everyone reads. A judge that has converged is finalized. A judge you retire is archived, with its history kept.
Spending expert time where it counts
Expert labels are the scarce input. The goal is not to label everything; it is to label the sessions where a human verdict changes the most. Rank candidates for review by signals such as impact, how close a score sits to its pass or fail threshold, how unlike previously labeled traffic a session is, and whether judges contradict each other, and deprioritise clusters that already carry several labels.
Give the experts a clear role. Reviewers label; principal reviewers, the people whose verdicts define truth, settle disputed sessions with one authoritative label. Track how each reviewer agrees with that truth too, because some labelers are more reliable than others.
Be honest about sample size. Agreement is computed only on sessions with expert truth, so a high kappa on thirty similar sessions says little about the other ten thousand. Label a sample that looks like real traffic.
Questions
What is LLM-as-a-judge?
A way of scoring an agent's output by asking a language model to grade it against a written rubric. The judge reads evidence from the run, such as the user's request and the agent's reply, and returns a score with a written reason you can audit.
How do you know an LLM judge is accurate?
Compare its verdicts with labels from human experts on the same sessions. Precision, recall, specificity and Cohen's kappa tell you how often it agrees, which failures it misses and how often it raises false alarms.
Why use Cohen's kappa instead of plain agreement?
Plain agreement looks high whenever most sessions pass, even for a judge that passes everything. Cohen's kappa corrects for agreement that would happen by chance, so 0 means no better than random and 1 means perfect agreement.
How many expert labels does a judge need?
Enough to represent your real traffic. Agreement is only computed on sessions that have expert truth, so a high score on a small or narrow sample can still mislead. Label more, and more varied, sessions before you rely on the number.
Can a judge be a rule instead of a model?
Yes. Some checks are better done by a deterministic service, such as a moderation or grammar API, and some quality numbers are best built by combining several judges into one composite score.
Sources
- Neens documentation, Judges: judge types, evidence, output types, versions and composites.
- Neens documentation, Annotations and review: the review queue, expert labels and judge alignment metrics.
- Neens documentation, Scorer lifecycle: lifecycle stages and version comparison.