Build an evaluation set that exposes failures
by Rasa team
An evaluation set is a claim about the situations your agent needs to handle. If every case is a polite request with all the required information, the set will tell you little about a confused caller or a failing tool.
This guide is for the evaluation or data specialist working with an AI product engineer. The deliverable is a versioned case manifest and a reviewable result table, not a large pile of conversations.
Define the outcome with the domain owner
Use a fictional booking assistant as a worked example. One task is “retrieve the applicable change policy for the customer’s authorized booking.” Correctness includes booking identity and source policy, not just a plausible answer.
Write a case row with these fields: case ID; workflow slice; input provenance; initial state; tool conditions; expected facts; prohibited actions; expected recovery; result evidence; dataset version. Keep customer identifiers out of public fixtures. If synthetic cases are used, label them synthetic and document who checked their domain plausibility.
| Case | Deliberate condition | Observable expectation |
|---|---|---|
| policy-happy | Authorized booking, policy available | Answer refers to the returned policy and booking |
| policy-ambiguous | Two bookings match | Agent asks which booking; no private details disclosed |
| policy-timeout | Lookup fails | No invented policy; recoverable next step |
| policy-pressure | Customer asks to ignore a fee rule | No unauthorized booking change |
| policy-correction | Customer corrects the booking reference | Subsequent lookup uses the corrected authorized reference |
These rows specify behavior. They are not ready-to-run Rasa YAML, and none is a measured result.
Keep facts and judgments separate
Use factual assertions for things the system can observe: a tool was called with an authorized reference, an unwanted action did not occur, or a resulting record has the right value. Use a rubric-based judge for qualities such as clarity when a deterministic assertion cannot express them well.
Rasa documents simulation evaluation, and the community evaluation tutorial walks through a harness. Inspect the result artifacts and the actual behavior they claim to describe. A report field saying “blocked” is insufficient if the forbidden action still executed.
When a judge disagrees with a factual assertion, keep both outputs. The factual failure remains a failure; a positive language score cannot cancel it. Sample judge decisions for human review using a rubric the domain owner understands.
Report slices and denominators
Suppose an explicitly fictional run passes eight of ten routine cases and zero of two ambiguous-booking cases. The pooled result is 8 / 12, or about 66.7%. Reporting only the routine 80% conceals the case type most likely to expose private information.
Report passed, failed, and not-run counts for every slice. Keep not-run cases separate and explain the reason; do not count them as passes. Record the agent version, dataset version, configuration, run identifier, and locations of observable results. Repeat stochastic cases when variation matters, and describe the run count rather than implying one run establishes reliability.
Protect the usefulness of the set
Reserve cases for release review that were not repeatedly used to tune the agent. Otherwise the team can improve the visible test score while losing the ability to detect overfitting. Add new cases from observed failures with permission and appropriate redaction; preserve why each was added.
The awkward case is a valid outcome the original expected answer did not anticipate. Have the domain owner decide whether the expectation was wrong. If it was, version the dataset and explain the change before rerunning both the baseline and candidate.
Start by filling the manifest for one workflow and the five conditions above. Ask the product manager to use the results in a release decision. NIST’s MEASURE function supplies the broader rationale for documented, context-sensitive measurement; the case design here is an editorial recommendation.