An eval that oversells itself is worse than no eval, because it launders a guess into a number. You built the suite in four chapters; this one is about reading it without fooling yourself.
Judge scores vary between identical runs
The judge is an LLM at nonzero temperature, deciding how to split statements, what counts as supported, and how well a transcript met a criterion. The same answer can score 0.75 on one run and 1.0 on the next. A single run near your threshold is noise, not a signal — in either direction. If a criterion flipped and you changed nothing, you have learned something about variance, not about your agent.
The simulator varies on purpose — and so does the agent
Every run, the simulator re-improvises the customer from your stage directions, and the agent re-improvises its handling. This is the honest way to cover a non-scripted system, and it means even the deterministic assertions can flip between runs — not because checking them is fuzzy, but because the behaviour they measure sits near a model’s decision boundary. The assertions are exact; the thing they measure is not.
When a criterion fails because the agent said the right thing in different words, there is a third possibility beside “agent broke” and “variance”: your criterion encodes incidental phrasing rather than a business requirement. Fix the criterion, not the agent.
Small samples say very little
Seven scenarios cannot separate a real regression from chance. The reflex of “we fixed it — the run passes now” after a single green run is exactly the failure mode a harness should protect you from, not create. When a result matters — a model swap, a release decision — raise the run count and look at the distribution, not the last run.
Related, and worth saying at full volume because vendors will not: beware of impressive single numbers. The companion pattern mentions a widely repeated figure from Rasa office hours — roughly 30% better task completion for a controlled variant over about 100 simulated conversations — and immediately labels it unpublished, unverified, and not reproduced. That labelling is the correct handling of every number you cannot rerun yourself, including the ones your own suite produced last month on a judge model that has since been swapped.
Cost decides cadence
Every scenario run bills three LLMs — agent, simulator, judge — and relevance assertions add embedding calls on top. This is not a reason to avoid the instrument; it is a reason to schedule it:
| Activity | Cost | Cadence |
|---|---|---|
| One scenario, run count 1 | cents | While iterating on that behaviour |
| Full suite, run count 1 | 3 LLMs × 7 scenarios | After any skill or config change |
| Full suite, raised run counts | the real bill | Model swaps, release decisions |
A team that runs the full suite at high run counts on every commit soon stops reading the results. A team that runs one scenario tightly while iterating, and the full suite at decision points, finds regressions while they are still cheap.
Ground truth drifts silently
The ground_truth: strings in the FAQ scenarios are copies of the policy
text in skills/faq_support/tools.py. Change the policy in one place only,
and the judge will faithfully score the agent against a policy that no
longer exists — green, and wrong. In a real project, generate ground truth
from the same source the agent reads, or put a check on the pair. (This
failure class — two sources of truth, coupled by nothing — is the same one
that motivates several checks in this site’s own build. It follows eval
suites everywhere.)
The one-paragraph summary of the whole series
Let the simulator play customers you did not script, so your coverage is of the system you actually shipped. Assert every fact the tracker can express — including absence, including order — because those results are immune to judge mood. Send only free text to the judge, pin the judge, pair groundedness with relevance, and read thresholds as fractions of small integers. Then hold the whole thing with a light grip: variance is real on three levels now, seven scenarios prove little, and a number you cannot rerun is an anecdote wearing a decimal point.
That grip — instruments trusted exactly as far as their mechanisms deserve — is the difference between measuring your agent and decorating it.
Where to go next
- The companion pattern’s README goes deeper on every mechanism here, with
engine source citations for each claim:
patterns/evaluation-harness. - The official simulation & evaluation docs cover the MCP runner setup per editor, and the full assertion list.
- New to building the agents themselves? The Mantle starter pack tutorial is this series’ natural prequel — its lint-and-hooks philosophy and this harness are the same idea at two different layers: catch the silent failure with a machine, before it costs you.