Set up Rasa simulation tests before reading any score

Clone the companion repository and move into the pattern:

git clone https://github.com/RasaHQ/rasa-community-resources.git
git -C rasa-community-resources checkout 4aa0c4419dc193fef7a969c12d59edcf720f2606
cd rasa-community-resources/patterns/evaluation-harness

Credentials and training

cp .env.example .env    # fill in RASA_LICENSE and OPENAI_API_KEY
uv sync --prerelease=allow
uv run rasa train

The test agent is a retail-banking assistant with two skills:

• check_balance — a task skill: disambiguate which account, call a tool, read back a number. Its tool returns hard-coded balances (skills/check_balance/tools.py): the Everyday Checking account contains $1,284.53 today, tomorrow, and forever, and Rainy Day Savings holds $9,140.00.
• faq_support — a knowledge skill: answer questions about overdraft fees and card replacement from hard-coded policy text (skills/faq_support/tools.py).

Fixed data removes database changes and external API failures from the test. A failure can still come from the agent, the simulator, the judge or the test setup. Inspect the conversation before assigning a cause. This small agent lets you learn the checks without also diagnosing a live banking service.

The suite you just cloned

Everything lives under eval/:

eval/
├── conftest.yml          # the two evaluation LLMs: simulator + judge
└── scenarios/            # one behaviour per file, seven in all

eval/conftest.yml configures the user simulator (which generates the customer’s turns) and the judge (which scores the transcript) independently — different jobs, separately swappable. The agent’s own model lives in the integrations file. Changing that model tests different agent behaviour. Changing the judge changes how the conversation is scored.

The seven scenarios are the standard Mantle behaviour checklist — happy path, out-of-order input, disambiguation, correction, digression-and-resume, refusal instead of fabrication, and a negative case where the right behaviour is to start nothing at all.

Run one scenario

Scenarios run through Rasa’s MCP server, driven from your coding agent (per-editor setup in the simulation docs):

uv run rasa tools run --mode stdio     # loads this project's .env

Then ask your coding agent, in natural language:

Run the balance_named_up_front scenario in eval/scenarios/.

The simulator reads the scenario’s customer instructions and talks to the trained agent. The assertion checker then checks the conversation record. Finally, the judge scores the written criteria.

Each run records its transcript, assertion results, criterion scores and the judge’s explanations. It also includes an Inspector link to replay the conversation. A separate file summarises the scenarios:

Results: eval/results/<timestamp>/
Per-run report: run_N.txt
Scenario summary: summary.txt

A run passes only when every assertion and every criterion passes. Quality metrics (helpfulness, task completion and friends) are recorded but do not gate.

Two observations before the dissection

Record usage for each model role. A simulated run can call the agent, customer simulator and judge. The requests and cost depend on the scenario, models and repeated turns. Start with one affected scenario and record provider usage before planning a larger batch.

The same run produces two kinds of evidence. Structural assertions give a deterministic result for recorded actions and values. Generative assertions use a model judge. The criteria scores are model judgments that can vary. Keeping straight which kind you are reading is most of the skill, and the next three chapters take them one at a time.

Check your understanding

• Why are hard-coded balances a feature of an eval fixture rather than laziness?
• Which model roles can a scenario use, and which role’s swap would change what “pass” means?
• Where do you look first when a run fails: the transcript, the assertion results, or the criteria rationale — and why?

The next chapter explains what a passing assertion establishes.