Evaluating non-deterministic systems

Every test suite ever written rests on one assumption: the same input produces the same output. Break that assumption and the whole apparatus — unit tests, snapshot tests, contract tests — stops meaning anything. This is the position anyone shipping a language model into production is in, and most teams discover it the week after launch.

The first instinct is to set temperature to zero and carry on as before. It does not work, and understanding exactly why it does not work is the beginning of a real evaluation practice.

The first thing to give up#

Greedy decoding removes sampling noise. It does not make an inference stack deterministic. The variance that remains is structural and lives below the API:

• Floating-point addition is not associative. Change the batch composition and a fused GPU kernel reduces partial sums in a different order, shifting a logit by 1e-6 — enough to flip the argmax when the top two tokens are close.
• Continuous batching in vLLM, TensorRT-LLM and every hosted endpoint means your request is padded and scheduled alongside whatever else arrived in that 20ms window. You do not control that.
• Mixture-of-experts routing is computed per batch. Which experts a token is dispatched to can depend on the other sequences in flight.
• Provider-side changes: a silent rollout of a new serving kernel, a quantisation change, a speculative-decoding draft model swapped underneath a stable-looking model name.
• Everything around the model — retrieval over an index that ingested new documents overnight, a tool call that returns live data, a timestamp in the prompt.

So the output distribution is the object under test, not the output. The question stops being “does it return X” and becomes “how often does it return something acceptable, and has that rate moved”. That is a measurement problem, and measurement problems have well-understood machinery.

Build the eval set out of traffic#

Hand-written eval examples encode what the team imagines users do. Production logs encode what they actually do. The gap between those two is where every embarrassing failure lives — the malformed paste, the three-language question, the user who types a single word.

The pipeline we set up on day one of an engagement looks like this:

1. Log every request with full context: the resolved prompt, retrieved chunk ids, tool calls and arguments, model snapshot id, latency, token counts, and any downstream signal (did the user retry, copy the answer, escalate to a human, thumbs-down it).
2. Embed the user turn and cluster with HDBSCAN or simple k-means over normalised embeddings. You are looking for the shape of the traffic, not perfect clusters.
3. Stratify: sample proportionally within each cluster so the eval set mirrors real distribution, then deliberately oversample the tail — the clusters with the worst downstream signal and the smallest volume.
4. Label a calibration slice by hand. Two annotators, disagreements adjudicated by a third, and record the disagreement rate. If humans agree only 78% of the time, no grader can honestly score above that.
5. Split into a development set you may iterate against freely, and a locked holdout you look at on release days only. Store the holdout hashes so leakage is detectable.
6. Rotate. Re-sample from the last 30 days quarterly, and retire examples whose behaviour the product intentionally changed.

Three ways to grade, three ways to be lied to#

Grading is where most eval systems quietly fail. Each method measures something real and is silent about something else, and the silence is what hurts you.

Golden sets

Use them where the answer space is genuinely closed. Invoice field extraction, intent routing, SQL generation checked by executing both queries and comparing result sets — these are golden-set problems and you should not reach for a judge model. Where the answer space is open, a golden set measures conformity to one author’s phrasing.

Rubrics

Programmatic assertions are the cheapest real signal in the system and the most under-used. Every citation resolves to a chunk that was actually retrieved. Every claimed figure appears in the context window. The response parses. No prompt content is echoed. These catch a surprising share of production incidents, run in milliseconds, and never drift.

LLM-as-judge

Judges are indispensable and dishonest. The biases are documented and reproducible — Zheng et al.’s MT-Bench work is the canonical public reference, and the same effects show up on internal sets:

• Position bias — the candidate shown first wins more often than chance. Run both orders; count flips as ties.
• Verbosity bias — longer answers score higher independent of content. Control for length, or include length as an explicit rubric penalty.
• Self-preference — a judge favours text generated by its own model family. Never let the model under test grade itself.
• Scale compression — ask for 1–5 and you get 3s and 4s. Ask for a binary verdict against a specific criterion and you get resolution.
• Sycophancy to the prompt — a judge told “this is our new improved answer” will find it improved.

Treat the judge as a classifier that itself needs evaluating. Score the calibration slice with the judge, compare against human labels, and report Cohen’s kappa. Below about 0.6 the judge is not fit to gate anything. Re-run this whenever the judge model or its prompt changes.

Prefer pairwise to absolute#

Absolute scoring asks a judge to place an answer on a scale it has no anchor for. Pairwise asks which of two answers is better, which is the question a judge can actually answer and the question the release decision reduces to.

• Run A-vs-B and B-vs-A. Agreement across both orders is a preference; disagreement is a tie, and the tie rate is a useful health metric of the judge itself.
• Aggregate with Bradley-Terry or Elo when comparing more than two candidates — it handles incomplete comparison graphs, which you will always have.
• Report the win rate with a confidence interval, not a bare percentage. “62% ± 9” is a decision; “62%” is a number someone will put in a slide.
• Keep a fixed anchor candidate in every run. Drift in the anchor’s win rate against a frozen opponent is drift in the judge, not in the product.

Retrieval and generation are separate experiments#

The single most common mistake in RAG evaluation is one end-to-end number. When it drops, nobody knows whether the retriever stopped finding the document or the generator stopped using it, so the team edits the prompt — the cheapest thing to change — and the real defect survives.

Instrument the two stages independently. Retrieval gets classical IR metrics against labelled relevant chunks: recall@k, nDCG@10, MRR. Generation gets graded only on answers where the correct context was definitely present.

• Recall@k is a ceiling on grounded answers. If the right chunk is in the top-k only 64% of the time, no amount of prompt engineering lifts faithful, cited accuracy above 64% — anything scoring higher is the model answering from memory, which is not what the retriever was built for. Fix the retriever.
• Context precision matters more than teams expect — padding the window with marginal chunks measurably degrades answers, and position matters (the “lost in the middle” effect on long contexts is real and reproducible).
• Grade faithfulness separately from helpfulness. An answer can be perfectly grounded and useless, or useful and partly invented. One score hides both.
• Track the abstention rate. A system that never says “I don’t have that” is not confident, it is unmeasured.

A twenty-example eval tells you nothing#

This is arithmetic, not opinion. A pass rate is a binomial proportion, and its precision scales with the square root of n. At a true 85% pass rate, here is what different eval sizes buy:

Those are unpaired figures and they are pessimistic. Run both candidates over the same examples and compare per-example deltas — a paired design cancels the “this question is just hard” variance and typically recovers a factor of two to three in sensitivity for free. McNemar’s test on the discordant pairs, or a paired bootstrap, is the right statistic.

One more trap: slicing. Evaluate across twelve customer segments at α = 0.05 and nearly half of your clean runs will throw up at least one slice that looks significantly different by chance alone. Apply Benjamini-Hochberg across the slices, or pre-register the two or three slices that actually gate the release.

Gating in CI#

The gate should refuse to merge on evidence, not on a point estimate that happened to land above a round number. Three conditions, all cheap to compute:

import { readFile } from "node:fs/promises";

type EvalResult = { id: string; slice: string; pass: boolean };

const GATE = {
  /** Lower bound of the pass-rate interval — never the point estimate. */
  minPassRate: 0.9,
  /** Examples that passed on main and fail on this branch — a budget, not a licence. A hard zero would make the paired check below unreachable. */
  maxRegressions: 2,
  /** Paired-bootstrap confidence that the branch is not worse overall. */
  minNotWorse: 0.95,
};

/** Wilson score lower bound. Behaves at small n, unlike the normal approximation. */
function wilsonLower(passes: number, n: number, z = 1.96): number {
  if (n === 0) return 0;
  const p = passes / n;
  const denom = 1 + (z * z) / n;
  const centre = p + (z * z) / (2 * n);
  const margin = z * Math.sqrt((p * (1 - p)) / n + (z * z) / (4 * n * n));
  return (centre - margin) / denom;
}

/** Resample per-example deltas — not the two runs independently. */
function pairedNotWorse(deltas: number[], draws = 10_000): number {
  if (deltas.length === 0) return 0;
  let wins = 0;
  for (let d = 0; d < draws; d += 1) {
    let sum = 0;
    for (let i = 0; i < deltas.length; i += 1) {
      sum += deltas[Math.floor(Math.random() * deltas.length)];
    }
    if (sum >= 0) wins += 1;
  }
  return wins / draws;
}

const load = async (path: string): Promise<EvalResult[]> =>
  JSON.parse(await readFile(path, "utf8")) as EvalResult[];

export async function main(): Promise<void> {
  const [baseline, candidate] = await Promise.all([
    load("evals/out/main.json"),
    load("evals/out/branch.json"),
  ]);

  const before = new Map(baseline.map((r) => [r.id, r.pass]));
  const shared = candidate.filter((r) => before.has(r.id));
  const deltas = shared.map((r) => Number(r.pass) - Number(before.get(r.id)));
  const regressions = shared.filter((r) => !r.pass && before.get(r.id) === true);
  const passes = candidate.filter((r) => r.pass).length;

  const lower = wilsonLower(passes, candidate.length);
  const notWorse = pairedNotWorse(deltas);
  const problems: string[] = [];

  if (lower < GATE.minPassRate) {
    problems.push(`pass-rate lower bound ${lower.toFixed(3)} < ${GATE.minPassRate}`);
  }
  if (regressions.length > GATE.maxRegressions) {
    problems.push(`regressed: ${regressions.map((r) => `${r.slice}/${r.id}`).join(", ")}`);
  }
  if (notWorse < GATE.minNotWorse) {
    problems.push(`paired confidence ${notWorse.toFixed(3)} < ${GATE.minNotWorse}`);
  }

  if (problems.length > 0) {
    console.error(problems.join("\n"));
    process.exit(1);
  }
  console.log(`gate passed — n=${candidate.length}, lower bound ${lower.toFixed(3)}`);
}

The regression check is the one that earns its keep. A branch can raise the aggregate pass rate while breaking eleven examples that used to work, and the aggregate will happily hide it. Listing the regressed ids by slice turns an argument about averages into a five-minute code review.

Budget the runs by stage, because a full eval on every push is both slow and expensive:

• Pull request — programmatic assertions plus a 150-example paired subset. Under four minutes, cached judge verdicts where inputs are unchanged.
• Nightly on main — the full development set, all slices, latency and cost percentiles recorded alongside quality.
• Release — the locked holdout, run once, reviewed by a human before the tag is cut.
• Always — record raw per-example outputs as build artefacts. When something regresses in three weeks you will want the diff, not the number.

Drift after a model upgrade#

Quality decays without anyone shipping anything. The provider rotates a snapshot, your corpus grows, user behaviour shifts after a marketing push. Offline evals do not catch any of this, because offline evals run on yesterday’s distribution.

1. Pin model snapshot ids explicitly, never floating aliases, and treat a snapshot change as a code change that goes through the same gate.
2. Pin the judge separately and hold it steady across the comparison. Upgrading the system and the judge in one release makes the result uninterpretable.
3. Shadow the candidate: mirror a slice of live traffic to the new configuration, grade both offline, ship nothing until the paired comparison clears.
4. Monitor input drift, not just output quality — population stability index on query-embedding clusters catches “our users started asking a new kind of question” a week before the complaints arrive.
5. Canary by percentage with an automatic rollback bound to abstention rate, tool-error rate and p95 latency, which move faster than quality signals do.

What good looks like#

None of this is exotic. It is the same discipline as any other measurement system — know the instrument’s error, pair your comparisons, hold one variable still. The teams that ship reliable AI are not the ones with the cleverest prompts. They are the ones who can answer “is this better” with a number and a confidence interval.

• An eval set drawn from real traffic, stratified, with a holdout nobody touches between releases.
• A judge validated against human labels, with its agreement rate published next to every score it produces.
• Retrieval and generation measured apart, so a regression points at a component.
• A CI gate that blocks on interval lower bounds and named regressions, not on a point estimate.
• Production monitoring that watches the input distribution, not only the output.

Build that before the demo goes to a customer, not after. Retrofitting evaluation onto a system already carrying production traffic means grading a moving target with no baseline to compare against — and by then, the only honest answer to “did that change help?” is that nobody knows.