Your test suite has produced a result for the banking assistant. Before using it to approve a release, you need to know what passed, what was judged by a model and whether the same outcome repeats.
This chapter shows how to separate a recorded tool action from a judgment about answer quality, plan useful repeats and track their cost. You will also check that the expected answers still match the policy the assistant uses.
Judge scores vary between identical runs
The judge is an LLM deciding how to split statements, what counts as supported, and how well a transcript met a criterion. A model judgment can vary even when the answer stays the same. A score close to your pass/fail cutoff needs repeated observations before you can assess whether the change improved the agent. If a criterion flipped and you changed nothing, you have learned something about variance, not about your agent.
The simulator varies on purpose — and so does the agent
Every run, the simulator re-improvises the customer from your stage directions, and the agent re-improvises its handling. This is the honest way to cover a non-scripted system, and it means even deterministic structural assertions can flip between runs — not because checking them is fuzzy, but because the behaviour they measure sits near a model’s decision boundary. Structural checks are deterministic; the model-generated behaviour they measure can vary.
When a criterion fails because the agent said the right thing in different words, there is a third possibility beside “agent broke” and “variance”: your criterion encodes incidental phrasing rather than a business requirement. Fix the criterion, not the agent.
Small samples say very little
Seven scenarios cannot separate a real regression from chance. The reflex of “we fixed it — the run passes now” after a single green run is exactly the failure mode a harness should protect you from, not create. When a result matters — a model swap, a release decision — raise the run count and look at the distribution, not the last run.
Related, and worth saying at full volume because vendors will not: beware of impressive single numbers. The companion pattern mentions a widely repeated figure from Rasa office hours — roughly 30% better task completion for a controlled variant over about 100 simulated conversations — and immediately labels it unpublished, unverified, and not reproduced. That labelling is the correct handling of every number you cannot rerun yourself, including the ones your own suite produced last month on a judge model that has since been swapped.
Cost decides cadence
A simulated run can make agent, simulator and judge requests. Some relevance checks also use embeddings. The model settings, number of turns and repeat count determine the bill; this tutorial has no measured price per run.
| Activity | Usage to record | Suggested cadence |
|---|---|---|
| One affected scenario | Calls and tokens for each model role | While changing that behaviour |
| Full suite, one repeat | Total model and embedding usage | Before reviewing a broader change |
| More repeats | Additional usage and variation in results | Model comparisons and release decisions |
Keep the original provider usage with each run. Estimate a proposed batch from comparable completed runs, then compare that estimate with the usage actually returned. A team that runs the full suite at high run counts on every commit soon stops reading the results. A team that runs one scenario tightly while iterating, and the full suite at decision points, finds regressions while they are still cheap.
Ground truth drifts silently
The ground_truth: strings in the FAQ scenarios are copies of the policy
text in skills/faq_support/tools.py. Change the policy in one place only,
and the judge will faithfully score the agent against a policy that no
longer exists — green, and wrong. In a real project, generate ground truth
from the same source the agent reads, or put a check on the pair. (This
failure class — two sources of truth, coupled by nothing — is the same one
that motivates several checks in this site’s own build. It follows eval
suites everywhere.)
The one-paragraph summary of the whole series
Let the simulator play customers you did not script, so your coverage is of the system you actually shipped. Assert every fact the tracker can express — including absence, including order — because those results are immune to judge mood. Send only free text to the judge, pin the judge, pair groundedness with relevance, and read thresholds as fractions of small integers. Then hold the whole thing with a light grip: variance is real on three levels now, seven scenarios prove little, and a number you cannot rerun is an anecdote wearing a decimal point.
That grip — instruments trusted exactly as far as their mechanisms deserve — is the difference between measuring your agent and decorating it.
Where to go next
- The companion pattern’s README goes deeper on every mechanism here, with
engine source citations for each claim:
patterns/evaluation-harness. - The official simulation & evaluation docs cover the MCP runner setup per editor, and the full assertion list.
- New to building the agents themselves? The Mantle starter pack tutorial is this series’ natural prequel — its lint-and-hooks philosophy and this harness are the same idea at two different layers: catch the silent failure with a machine, before it costs you.