How to test an agent with a simulated customer

A scripted conversation test fixes the customer’s turns in advance. That is useful for reproducing a known failure, but it covers only the path you wrote.

A Mantle skill is not a scripted system. skill.md gives the model intent and constraints — disambiguate which account, never invent a balance — and the model improvises the path. Two runs of the same conversation can legally take different routes to the same correct outcome. A test with fixed user turns checks one path. A simulated customer can explore other paths while pursuing the same goal. Keep scripted tests for failures that need exact reproduction.

So the framework replaces the script with an actor: an LLM user-simulator that plays your customer, one scenario at a time.

Stage directions, not scripts

The simulator reads one prose block per scenario — simulation_context — and improvises from it. From eval/scenarios/balance_digression_faq_resume.yml:

simulation_context: >
  You are a curious bank customer. Start by asking for an account balance
  without naming an account. When the agent asks which account, do NOT
  answer; instead ask "wait, first — how much is the overdraft fee?". After
  you get the fee answer, say "ok, the checking one" to return to the
  balance question. Accept the balance, thank the agent, and end the
  conversation.

Write these like stage directions for an actor: persona (who the customer is, their temperament), intent (what they want, what they will answer when asked, what they refuse), and a clear end condition. Give the simulator enough to play the part — then let it choose the words. The words being chosen fresh on every run is the entire point: it is what stops your suite from quietly testing only the phrasings you thought of.

What this makes testable

Look at what the checklist scenarios in the companion actually exercise:

• Digression and resume (balance_digression_faq_resume.yml) — the customer interrupts one skill with a question for another, then comes back. The interruption can occur at different points. Assertions check whether the agent resumes (flow_started for both skills, then the balance facts).
• Correction (balance_correction.yml) — “actually, the savings one please”, expressed as a sequencing assertion: the same slot set, then re-set.
• Out-of-order input (balance_named_up_front.yml) — the customer volunteers the account before being asked, and a criterion checks the agent did not re-ask for it.

These are exactly the behaviours a scripted file under-tests, because their interesting part is where in the conversation they happen — the thing you cannot hard-code without deciding it in advance.

The old absence habit survives the move, upgraded: the ambiguous-request scenario used to assert “no slot was set after turn 1” against a scripted turn. A written criterion now asks whether the agent clarified the account before answering. The judge checks the transcript for that behaviour. Separate assertions check the final account, tool call and spoken number.

What happened to dialogue-understanding tests

Earlier versions of this series taught rasa test du here — annotated per-turn command tests for the CALM command generator. They are gone from the companion, for a reason worth reading because it is a lesson in trusting mechanisms over labels:

DU tests score the command generator — CALM’s component that turns each user message into commands like StartFlow(x). The CLI hard-gates them to CALM assistants (rasa/cli/dialogue_understanding_test.py:185-199). A Mantle agent, however, slips that gate: it declares is_calm_assistant = True (rasa/mantle/processor.py:104-107) — and then the tests execute against a component that never runs, because Mantle’s turn loop is an LLM tool-calling orchestrator that imports nothing from rasa.dialogue_understanding (rasa/mantle/orchestration/orchestrator.py). A score for a component your agent does not use cannot establish whether that agent understood the customer.

The question DU tests answered — did the agent understand, separately from whether the conversation ended well? — has not disappeared. It has moved: understanding failures now surface as failed assertions on memory (slot_was_set with the wrong value is a misread) and as criteria rationales that name the turn where the transcript went sideways. The Inspector URL in every run report replays the conversation for exactly this kind of diagnosis.

Variance is a feature with a bill attached

The same scenario can produce different conversations on repeated runs. One passing run is limited evidence. One failing run deserves inspection, especially when a judge score is near its threshold. Chapter 5 explains how to use repeated runs. For now, repeat a surprising scenario before drawing a conclusion. Your coding agent will happily “run balance_correction 5 times” — remembering that each of those runs bills the agent, the simulator, and the judge.

Check your understanding

• Why does hard-coding user turns under-test a Mantle skill specifically, when the same technique served scripted assistants well?
• A digression scenario passes with the interruption landing at turn 2 on one run and turn 4 on the next. What in the scenario file made that robustness testable?
• The old DU suite would still execute against a Mantle agent. Walk the three engine citations that explain why its scores would mean nothing.

The simulator supplies conversations, and assertions check the recorded facts. The next chapter explains how a judge assesses free-text answers.