An assertion reads the conversation tracker — the engine’s own record of what happened — and answers a yes/no question: did this flow start? does this slot hold this value? did this action run? No LLM is involved in checking it, so whatever the simulator improvised and however the judge felt that day, the assertion result is a fact.
That is why assertions are the ground truth inside every scenario: the judged criteria capture quality of handling, but the non-negotiable facts — the right account, the tool actually ran, the right number was spoken — are nailed down here, where variance cannot reach them.
The shape
From eval/scenarios/balance_named_up_front.yml in the companion:
scenario:
name: Customer names the account up front and gets its balance in one pass
simulation_context: >
You are a calm, logged-in customer of a retail bank. Open by asking for
the balance of your Everyday Checking account...
goals:
criteria:
- The agent does not ask which account the customer means...
assertions:
- flow_started: check_balance
- slot_was_set:
- name: check_balance.selected_account_id
value: acc_checking
- action_executed: get_balanceRead the assertions as a claim: however this conversation went, by the end the right skill started, the right account landed in memory, and the tool ran. Three facts, three assertions — and the criteria above them handle the “however it went” part.
The vocabulary
The scenario schema pins these assertion types
(rasa/builder/copilot/mcp_server/schema/scenario_schema.yml in the
installed engine — and the checking machinery is literally the classic e2e
assertion classes: the scenario engine imports rasa.e2e_test.assertions
directly, assertion_engine.py:9):
| Assertion | Checks |
|---|---|
flow_started | a flow with this id started |
flow_completed / flow_cancelled | a flow finished / was cancelled |
slot_was_set / slot_was_not_set | a slot holds (or doesn’t hold) a value |
action_executed | a named action ran |
bot_uttered / bot_did_not_utter | a response, by name, buttons, or regex |
pattern_clarification_contains | the clarification pattern offered these flows |
sequencing | these facts occurred, in this order |
generative_response_is_grounded / ..._is_relevant | the judge — Chapter 4’s subject |
Two naming details that will otherwise cost you an afternoon: memory keys are
scope-qualified — the fixture’s slot is check_balance.selected_account_id,
skill id then entry, never the bare entry name — and a prose skill’s flow id
is simply the skill id (check_balance); the skill__block form belongs
to ordered blocks (rasa/mantle/skills/catalog.py:686-714).
Three habits in the companion’s scenarios separate a suite that catches regressions from one that merely exists:
Habit 1 — assert absence, not just presence
From eval/scenarios/smalltalk_starts_nothing.yml:
goals:
criteria:
- The agent responds politely and briefly without pretending the
customer asked for something
assertions:
- slot_was_not_set:
- name: check_balance.selected_account_idThe cheapest way for an agent to look competent on a positive-only suite is to start a task for everything. Your suite must contain conversations where the correct behaviour is restraint — and assert that restraint happened. An agent that treats “thanks, bye” as a balance enquiry passes every positive scenario you own.
Habit 2 — match the fact, not the phrasing
- bot_uttered:
text_matches: '9,140'text_matches takes a regular expression. The disambiguation scenario pins
"9,140" — the number the fixture tool returns for savings — and lets the
model word the sentence however it likes. Pinning whole sentences gives you a
suite that fails on every prompt tweak; a suite that cries wolf on wording
trains your team to ignore failures, which is worse than no suite — you keep
the cost and lose the signal.
The discipline generalises: assert what must be true (the number, the slot, the action), stay silent about what is allowed to vary (the phrasing).
Habit 3 — express order as sequencing
Some behaviour is an order of events. The correction scenario
(balance_correction.yml) needs “the account was chosen, then changed”:
assertions:
- sequencing:
- slot_was_set: check_balance.selected_account_id # set once...
- slot_was_set: check_balance.selected_account_id # ...then re-set
- slot_was_set:
- name: check_balance.selected_account_id
value: acc_savingsEach child of sequencing must occur, in this order. The same slot set twice
is the correction — a judged criterion could only say the agent “seemed to
handle the change of mind”; this states it as a fact.
What this instrument cannot see
Look again at the vocabulary. Every row is about structure: flows, slots, actions, responses. Nowhere is “was this answer good?” — because that is not a fact about the tracker. Ask the FAQ skill about overdraft fees and it answers in free text: an assertion can prove a response was uttered, even regex-match a fragment, but it cannot judge whether the sentences were accurate, complete, or on-topic.
For that there is no fact to assert — that is judge territory, Chapter 4. But first: who is actually talking to your agent during all of this, and why is it not you?