LLM-as-judge uses a language model to evaluate another output or interaction against a stated question or rubric. The verdict is a measurement produced by a configured component, and like any measurement it has properties that need establishing before the numbers mean anything.

Three tasks travel under one name and need separating: comparing which of two answers reads better, checking whether a factual claim holds, and approving a consequential action. Only the first is naturally a judging task.

Route what can be checked directly

Before configuring a judge, decide which properties admit a direct check. A schema, an identifier, an arithmetic result, or an observed tool effect can be verified deterministically. A model judgement covers different properties — clarity, relevance, whether an explanation addresses the question.

Keep those results separate rather than folding them into one score. A high average on answer quality cannot compensate for a prohibited action, and a scoring scheme that lets it do so has made the wrong thing fungible.

Order sensitivity is the first thing to measure

Judging methodology distinguishes pairwise comparison, single-answer grading, and reference-guided judging, and the pairwise mode carries a specific defect: the position an answer occupies affects the verdict. Work testing this proposes combining evaluations across both response orders, using historical models and a small question set, and does not establish that order balancing removes every judging error.

The test is straightforward and the bookkeeping is where it goes wrong. Run each comparison in both orders and map every verdict back to stable response identifiers before comparing. A judge that picks the first-displayed answer in both presentations has issued two verdicts that contradict each other, because the identities swapped. Recording that as two wins for whichever model happened to lead is how order sensitivity becomes a performance claim.

Where both presentations prefer the same response, that establishes consistency on this check. It does not establish that the preferred answer is correct.

Test superficial style, length, and prompt wording the same way, against cases whose expected relationship has been checked in advance.

The judge is a boundary too

The content being evaluated is data, and it can contain instructions. Test whether text embedded in an evaluated response alters the grading, and keep any execution permissions outside the scoring model.

Validate the score parser as well, and count invalid, missing, tied, and abstained results explicitly. Those categories disappear silently and take the denominator with them.

Repeated scores from the same judge on the same item are not independent votes. An explanatory paragraph attached to a verdict is not external verification of it.

What calibration is, and is not

Calibration here means developing and assessing a scoring procedure against a specified human reference process. It is not probabilistic calibration, and a judge score of 0.9 is not a 90 percent probability of correctness.

Two limits bound what calibration delivers.

Agreement is not validity. A study establishing agreement between a judge and a majority reference drawn from a few annotators has established agreement with those annotators. Reliability and validity are distinct, and raters who share an incorrect interpretation agree perfectly.

Consistency is not accuracy. Higher consistency after few-shot prompting need not mean higher accuracy, and it can introduce new bias. Tuning a judge to be more stable is therefore not evidence that it became more correct, and it is the change that gets reported as an improvement.

So calibration runs against independently recorded human judgements on the intended task, with disputes and limits preserved rather than reconciled away, and adjusting a rubric until it imitates raters does not establish that it measures the intended construct.

What to retain

Rubric, inputs, reference provenance, response identity, presentation order, verdict, failures, and version. Enough configuration and permitted evidence for a later team to investigate why a score changed.

Unresolved judgements go to a declared review route rather than being forced into a category. A judge that cannot abstain will not abstain, and the resulting labels are indistinguishable from confident ones.

The rule

What stays fixed is that a judge’s verdict is evidence about the judge until its task validity has been checked. What changes is the output distribution being graded, and a judge validated on one distribution has to be revalidated when the system it grades changes.

Not to be confused with

Human evaluation. A judge approximates a reference process at lower cost. Where the reference process itself is undefined, there is nothing to approximate.

A gate. Properties that can be verified directly should be, and reserving the judge for what needs judgement keeps both instruments honest.