What it does
I want a number before and after every change to a prompt, a model or a tool, measured on cases the thing being improved has never seen. The harness gives me that. It runs the same code the chat runs, against two kinds of sets per eval: dev sets that Claude drafts and I review, and held-out sets that I write myself and agents can’t touch.
It covers routing, retrieval over my docs, notes and files, and whole agent answers. The classifier results it produced are on the agent core page; retrieval is below.
Why this way
Decision 01
Measure the code I run, not a copy of it
The classifier and the router live in agent core, and the harness imports them as they are. A test checks that the classifier request the harness sends is byte for byte the one the router sends. The harness changes a single setting: it bypasses the gateway’s response cache, so every case really reaches the model.
Rejected: a copy of the prompt inside the harness
It would drift from the real one, and I’d end up measuring something I no longer run.
Decision 02
Held-out sets belong to me
Every kind of eval has two sets. Claude drafts the dev sets, I review them like code, and models may see them. The held-out sets are mine: agents can’t write, move or delete anything in their folder, and a held-out run prints and stores totals only, with no case IDs, texts or per-case outputs.
The next step goes further: Claude may neither read nor run the held-out sets, and a baseline never changes in the same pull request as the change it measures.
Decision 03
Metrics checked by code
Code, not a model, scores the results: classifier accuracy with a confusion matrix, routing up and down, hit@5 and MRR for retrieval, whether a planted word shows up in the answer, whether the answer invents numbers, and whether its first and last paragraphs are in the question’s language.
Rejected: an LLM judge as the default
That’s a model grading a model, and the judge’s prompt becomes one more thing to tune.
Decision 04
Its own key, budget and stop rules
The harness has its own gateway keys, one for classifier evals and one for agent evals, each with a 30-day budget. It can’t reach the subscription worker at all: it has no way to address it, it isn’t on its network, and the gateway accepts only its roles. A run stops early once it has spent $2, when the kill switch is on, when the money is gone, or after 5 failures in a row. Each eval run is one run record with its cost, and that record never holds case text.
Rejected: running evals on the chat key
Eval spending would look like chat spending, and one bad loop could empty the chat’s budget.
Decision 05
Test data only, and a stop before tuning
Agent evals use a second server token that sees only the repo’s docs, a seed vault of made-up notes and test PDFs, never my real notes. If a dev score comes in under 0.8, the build stops and reports to me before anyone starts tuning.
How it's tested
Retrieval, measured three ways
Docs retrieval reached a hit@5 of 0.967 on the dev set and 0.875 on my held-out set of 16 questions. As the corpus grew, the held-out score fell to 0.75; a ranking fix brought dev back to 1.0, and no held-out set got worse. Files retrieval scored 1.0 on both. On my vault set, searching chunks scored 0.909 and searching notes by title and summary only 0.591, so agents now search chunks first and use note search for an overview.
Before and after, on purpose
Some runs exist only to compare. A planted instruction showed up in 1 of 2 answers before the tool-result rule and in none of 12 after it. The reply-language fix took a deliberately failing run on the old prompt from 0.8 to 1.0. Homelab questions scored 1.0 with tools, and still 1.0 without them in a deliberate run: the model said it couldn’t know instead of inventing a single number.
What it costs to measure
A full agent eval, including the injection repeats, cost between $0.59 and $1.50 at first. With prompt caching the cost per case fell from $0.040 to $0.014 at the same scores, and a full dev pass of 58 to 63 cases now costs $0.77 to $1.03. Building and running the whole first harness item cost $0.004.
| What | Value | n | Date |
|---|---|---|---|
| Docs retrieval hit at 5 held out | 0.875 | 16 | 2026-11-04 |
| Vault retrieval note search | 0.591 | — | 2026-11-04 |
What I learned
Case 01
Result files leaked the names of my notes
What happened
Before one merge I noticed that a docs-retrieval result file listed the real paths of notes in my vault. The docs set searches the whole knowledge scope, and only the vault set’s output had been redacted. No note text got out, but the names were in files headed for publication.
What I changed
Every retrieval set is now redacted the same way: any result outside the eval’s scope shows up as other, and a test holds it there.
Result
I fixed the two affected files and checked all 23 result files; none had anything from outside the scope.
Case 02
The first comparison run failed on every case
What happened
My first run on cheap returned an error for all 40 cases: the provider refuses a temperature setting for those models.
What I changed
The harness now leaves temperature out for cheap and smart only, and every result file records that override.
Result
The output says when the override applied, and local runs stay identical to production.
Case 03
Test the test
What happened
One homelab case had a list of phrases that count as a false all-clear, and one of them was “all 7”. The correct answer contained it, because seven containers really were running, so the case failed a right answer.
What I changed
I cut the list down to real all-clear phrases.
Result
The rerun passed. Word lists in tests need testing too.
A held-out set that wasn’t independent
To save time I let Claude draft my first held-out classifier set, with topics and shapes unlike the dev set, and I reviewed it and copied it in myself. It was still less independent than a held-out set should be. I wrote the retrieval sets’ questions myself, from nothing but a list of file names and headings, never from the builder’s cases.
Case 04
A metric that couldn’t see the problem
What happened
My “right source” metric passed whenever the expected document came back from any tool. In a real run the model found the document I had named, never read it, and answered from search snippets, and the metric would have passed that too.
What I changed
Source cases can now also name the tools that must be called, and the summary reports how often that held.
Result
The new cases exposed a second problem right away, in the knowledge core’s search: a document I named by its short number was never found at all.