How to test what a Rasa agent did and did not do

A banking assistant says it has fetched the customer’s balance. You want to check the recorded account choice and tool call, rather than accepting that sentence as proof the work happened.

Rasa keeps a conversation tracker: a record of actions, stored values and messages. In this chapter, you will write structural assertions—yes/no checks of that record—to verify which task started, which account ID was saved and whether the balance tool ran. These checks do not need a model to judge the wording.

Generative assertions are different: they ask an LLM to judge a response. They appear in the same scenario format, but their answers can vary. The table below separates those two types.

The shape

From eval/scenarios/balance_named_up_front.yml in the companion:

scenario:
  name: Customer names the account up front and gets its balance in one pass
  simulation_context: >
    You are a calm, logged-in customer of a retail bank. Open by asking for
    the balance of your Everyday Checking account...
  goals:
    criteria:
      - The agent does not ask which account the customer means...
    assertions:
      - flow_started: check_balance
      - slot_was_set:
          - name: check_balance.selected_account_id
            value: acc_checking
      - action_executed: get_balance

Read the assertions as a claim: however this conversation went, by the end the right skill started, the right account landed in memory, and the tool ran. Three facts, three assertions — and the criteria above them handle the “however it went” part.

The vocabulary

The scenario schema pins these assertion types (rasa/builder/copilot/mcp_server/schema/scenario_schema.yml in the installed engine — and the checking machinery is literally the classic e2e assertion classes: the scenario engine imports rasa.e2e_test.assertions directly, assertion_engine.py:9):

Memory keys include the skill name before the field name. A prose skill uses its skill name as the flow ID. Ordered blocks use a separate naming form:

The installed engine defines the block naming rule here:

rasa/mantle/skills/catalog.py:686-714

Three habits in the companion’s scenarios separate a suite that catches regressions from one that merely exists:

Habit 1 — assert absence, not just presence

From eval/scenarios/smalltalk_starts_nothing.yml:

goals:
  criteria:
    - The agent responds politely and briefly without pretending the
      customer asked for something
  assertions:
    - slot_was_not_set:
        - name: check_balance.selected_account_id

The cheapest way for an agent to look competent on a positive-only suite is to start a task for everything. Your suite must contain conversations where the correct behaviour is restraint — and assert that restraint happened. An agent that treats “thanks, bye” as a balance enquiry passes every positive scenario you own.

Habit 2 — match the fact, not the phrasing

- bot_uttered:
    text_matches: '9,140'

text_matches takes a regular expression. The disambiguation scenario pins "9,140" — the number the fixture tool returns for savings — and lets the model word the sentence however it likes. Matching a whole sentence can cause failures after harmless wording changes. Those failures make it harder to notice a real regression.

The discipline generalises: assert what must be true (the number, the slot, the action), stay silent about what is allowed to vary (the phrasing).

Habit 3 — express order as sequencing

Some behaviour is an order of events. The correction scenario (balance_correction.yml) needs “the account was chosen, then changed”:

assertions:
  - sequencing:
      - slot_was_set: check_balance.selected_account_id # set once...
      - slot_was_set: check_balance.selected_account_id # ...then re-set
  - slot_was_set:
      - name: check_balance.selected_account_id
        value: acc_savings

Each child of sequencing must occur, in this order. Recording two writes to the same slot shows that the value was updated. The final assertion checks that the updated value is the savings account. A judge can assess how the agent handled the correction, but these checks establish the recorded state.

What this instrument cannot see

Structural checks tell you which actions and values the engine recorded. They can establish that an FAQ response was sent or that its text matched a pattern; those checks alone do not establish that its answer was accurate or complete.

Generative assertions and natural-language criteria ask a model to assess that answer. Chapter 4 explains how to read those judgments and their limits. First, Chapter 3 shows how the simulator supplies the customer’s turns.