Evals before prompts: a small eval harness for an LLM feature

Every LLM feature I have built started the same way: a prompt that worked on the three examples I had open. On the roof report extraction work, the examples I had open turned out to be the easy ones. Real uploads failed in ways I had not pictured, and the only thing that stopped me from fixing one failure and quietly causing another was turning each failure into a test case.

That is all an eval harness is. A list of inputs you care about, a way to decide whether each output is good, and a number at the end. I now build it before I write the second version of any prompt, because without it every prompt edit is a guess that happens to look right on the case in front of you.

The harness below is small on purpose: a JSONL file, two kinds of checks, a script and a CI job. The examples use the Anthropic TypeScript SDK and a made-up feature, but nothing about the shape is tied to one provider.

The feature under test

To keep it concrete, picture an inbox helper for a roofing contractor. An email comes in, and the feature returns three things: a category (quote request, storm damage, billing, scheduling or spam), an urgency, and a short draft reply that a person reviews before sending.

Two of those outputs have one right answer. The third is free text where many answers are fine and a few are dangerous. That mix is typical, and it is why the harness needs two kinds of checks.

The golden set

The golden set is a file of cases. Each case has the input, the fields that have one right answer, and a few notes in plain English about what a good reply must and must not do. I add tags so I can slice the score later, and I mark a handful of cases critical: the ones where a wrong answer costs real money or trust.

Where do cases come from? Real inputs first, with names and addresses replaced. Then every bug report, word for word. Then the edge cases you can think of: an empty body, a forwarded thread, an email in two languages, a polite message that is actually a complaint. Thirty to fifty cases is enough to start. The set grows every time the feature fails in production, and that habit matters more than the starting size.

I keep the notes short and checkable. "Must not promise a visit date" is a good note. "Should sound professional" is not, because nobody can grade it the same way twice.

{"id":"storm-017","tags":["storm","critical"],"input":{"from":"homeowner@example.com","subject":"Shingles in the yard","body":"Wind last night took a bunch of shingles off the back slope. Water is coming through the bedroom ceiling. Can someone come today?"},"expect":{"category":"storm_damage","urgency":"high"},"notes":"Must acknowledge the leak. Must not promise a time or a price. Should ask for a phone number if none is given."}
{"id":"billing-004","tags":["billing"],"input":{"from":"ap@example.org","subject":"Invoice 2291","body":"We were charged twice for the gutter work. Please advise."},"expect":{"category":"billing","urgency":"normal"},"notes":"Must not admit or deny the double charge. Should say the office will check the invoice."}

import { z } from "zod";

export const Case = z.object({
  id: z.string(),
  tags: z.array(z.string()),
  input: z.object({ from: z.string(), subject: z.string(), body: z.string() }),
  expect: z.object({
    category: z.enum(["quote_request", "storm_damage", "billing", "scheduling", "spam"]),
    urgency: z.enum(["low", "normal", "high"]),
  }),
  notes: z.string(),
});
export type Case = z.infer<typeof Case>;

Exact checks first

Anything that can be checked with code should be checked with code. It is free, it is instant and it never disagrees with itself. For this feature that means the two enum fields, plus a few rules about the reply that a regex can enforce: no dollar amounts, no links, a length cap.

import type { Case } from "./types";
import type { TriageOutput } from "../src/triage";

export function exactChecks(c: Case, out: TriageOutput) {
  return [
    { name: "category", pass: out.category === c.expect.category },
    { name: "urgency", pass: out.urgency === c.expect.urgency },
    { name: "no_prices", pass: !/[$£€]\s?\d/.test(out.reply) },
    { name: "no_links", pass: !/https?:\/\//i.test(out.reply) },
    { name: "short", pass: out.reply.split(/\s+/).length <= 120 },
  ];
}

Exact checks also run first and short-circuit. If the category is wrong, the case has failed, and there is no reason to pay a second model to read the reply. On a large set that cuts judge calls noticeably, and it keeps the judge focused on the part only a model can grade.

An LLM judge with a rubric it cannot wriggle out of

The draft reply needs judgement, so a model grades it. The mistakes I see most with LLM judges are vague questions ("rate this reply from 1 to 10") and judges that see too little context. A score of 7 tells you nothing, and two runs will give you 6 and 8 for the same text.

What works better for me is a short list of named criteria, each pass or fail, each with a one-sentence reason. The judge sees the original email, the case notes and the draft. I use structured outputs so the verdict always parses, and I use a model at least as strong as the one being graded.

import Anthropic from "@anthropic-ai/sdk";
import { z } from "zod";
import { zodOutputFormat } from "@anthropic-ai/sdk/helpers/zod";
import type { Case } from "./types";

const client = new Anthropic();

const Verdict = z.object({
  criteria: z.array(
    z.object({
      id: z.enum(["answers_the_ask", "no_invented_facts", "follows_notes", "tone"]),
      reason: z.string(),
      pass: z.boolean(),
    }),
  ),
});

const RUBRIC = `You grade a draft reply written for a roofing contractor's inbox.
Grade every criterion separately as pass or fail. There is no partial credit.
- answers_the_ask: the reply responds to what the sender actually asked.
- no_invented_facts: no prices, dates, warranties or names that are not in the email.
- follows_notes: the reply satisfies every point in <case_notes>.
- tone: plain and polite, no exclamation marks, no sales pressure.
Write the reason before deciding pass or fail.`;

export async function judge(c: Case, reply: string) {
  const res = await client.messages.parse({
    model: "claude-opus-5-5",
    max_tokens: 16000,
    system: RUBRIC,
    messages: [
      {
        role: "user",
        content: `<email>\n${c.input.subject}\n\n${c.input.body}\n</email>\n<case_notes>\n${c.notes}\n</case_notes>\n<draft_reply>\n${reply}\n</draft_reply>`,
      },
    ],
    output_config: { format: zodOutputFormat(Verdict) },
  });
  if (!res.parsed_output) throw new Error(`judge gave no verdict: ${res.stop_reason}`);
  return res.parsed_output.criteria;
}

A judge is itself a model output, so I check it before I trust it. I grade twenty cases by hand, run the judge on the same twenty, and read every disagreement. Usually the fix is in the rubric: a criterion that mixed two ideas, or a word like "concise" that I understood one way and the judge another. When the judge and I agree on almost all of them, I stop checking every run and only spot-check failures.

Keep the rubric short. Four or five criteria per feature is plenty. If you need more, the feature is probably doing two jobs.

Running it

The run script reads the cases, calls the real feature, runs the checks and writes a result file. Two details matter.

First, outputs vary from run to run. Current Claude models do not accept a temperature setting at all, and even when they did, zero was never fully deterministic. So each case runs three times and gets a pass rate. A case that passes two times out of three is telling you something different from one that never passes, and you want to see both.

Second, concurrency. Firing every case at once hits rate limits; running them one by one is slow. A small worker pool is enough.

import { readFileSync, writeFileSync } from "node:fs";
import { Case } from "./types";
import { exactChecks } from "./checks";
import { judge } from "./judge";
import { triage } from "../src/triage";

const RUNS = 3;
const WORKERS = 6;
const cases = readFileSync("evals/golden.jsonl", "utf8")
  .trim().split("\n").map((line) => Case.parse(JSON.parse(line)));

async function runOnce(c: Case) {
  const out = await triage(c.input);
  const exact = exactChecks(c, out);
  if (exact.some((k) => !k.pass)) return { pass: false, failed: exact.filter((k) => !k.pass).map((k) => k.name) };
  const verdict = await judge(c, out.reply);
  const failed = verdict.filter((v) => !v.pass).map((v) => v.id);
  return { pass: failed.length === 0, failed };
}

async function scoreCase(c: Case) {
  const runs = [];
  for (let i = 0; i < RUNS; i++) runs.push(await runOnce(c));
  return { id: c.id, tags: c.tags, passRate: runs.filter((r) => r.pass).length / RUNS, failed: runs.flatMap((r) => r.failed) };
}

const queue = [...cases];
const results: Awaited<ReturnType<typeof scoreCase>>[] = [];
await Promise.all(
  Array.from({ length: WORKERS }, async () => {
    for (let c = queue.shift(); c; c = queue.shift()) results.push(await scoreCase(c));
  }),
);

const score = (100 * results.reduce((s, r) => s + r.passRate, 0)) / results.length;
writeFileSync("evals/results/latest.json", JSON.stringify({ score, cases: results }, null, 2));
console.log(`score ${score.toFixed(1)} over ${results.length} cases`);

In CI on every prompt change

Prompts live in files in the repository, not in a database or a dashboard, so a prompt change is a pull request like any other. The CI job runs only when prompts, the LLM code or the evals change, because a full run costs real tokens.

The comparison has two rules. The overall score may not drop by more than two points, which leaves room for run-to-run noise. And every case tagged critical must pass on all three runs. If I could keep only one of the two, I would keep the second: an average can hide one very bad regression behind several small wins.

name: evals
on:
  pull_request:
    paths: ["prompts/**", "src/llm/**", "evals/**"]
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      - run: npm ci
      - run: npx tsx evals/run.ts
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
      - run: npx tsx evals/compare.ts

import { readFileSync } from "node:fs";

const base = JSON.parse(readFileSync("evals/baseline.json", "utf8"));
const now = JSON.parse(readFileSync("evals/results/latest.json", "utf8"));

const drop = base.score - now.score;
const critical = now.cases.filter((c: any) => c.tags.includes("critical") && c.passRate < 1);

console.log(`baseline ${base.score.toFixed(1)}, now ${now.score.toFixed(1)}`);
for (const c of critical) console.log(`critical case failed: ${c.id} (${c.failed.join(", ")})`);
if (drop > 2 || critical.length > 0) process.exit(1);

Two practical notes. Pull requests from forks do not get repository secrets on GitHub, so this job will not run for outside contributors without extra setup. And the baseline file is updated in the same pull request that improves the score, so reviewers see the number move next to the prompt diff that moved it.

Tracking the score over time

A single run tells you whether a change is safe. A history tells you whether the feature is getting better. After each run on the main branch I append one line to a JSONL file: date, commit, a hash of the prompt files, the model id, the score and what the run cost.

The model id in that line earns its place on upgrade day. When a new model comes out I change one constant, run the same set, and compare. I do not have to argue about whether the new model "feels better". The cost column matters too: a harness that runs a few hundred model calls per pull request is cheap next to a bad prompt in production, but it is not free, and seeing the number keeps me from adding runs without thinking. For the nightly full run I send the calls through the batch API at half price, which I cover in the post on inference cost.

Where to start

1. Write ten cases today

   Take real inputs, strip personal details, and write down the right answer and one or two notes for each.

2. Check with code what code can check

   Enums, formats, forbidden strings, length. Run these first and stop on failure.

3. Add a judge with named pass/fail criteria

   Four or five criteria, a reason before each verdict, structured output, and a model at least as strong as the one being graded.

4. Calibrate the judge once

   Grade twenty cases yourself and fix the rubric until you and the judge agree.

5. Gate pull requests and keep a history

   Fail on a drop in the average or any failed critical case, and log the score, model and cost on every main-branch run.

None of this is clever. It is the same discipline as unit tests, applied to a part of the system that cannot be unit tested in the usual way. The difference it makes is that a prompt change stops being an opinion and becomes a diff with a number next to it.