How to build test fixtures from your agent's own code

Your agent tests pass. But a test is only as good as its input. If you typed the test session by hand, it may hold data the running agent never collects. Then the test passes while the real agent fails.

Here is that case in a sample agent. It hands a caller to a human desk, and two tests check the same thing: the desk never has to ask a question the caller already answered. In a scratch copy, we deleted one line from the list of keys the agent copies into the handoff: the line for the account. Then we ran both tests. This output is trimmed to the result lines; the command, the two tracebacks and the summary are left out:

test_caller_is_never_asked_a_question_they_answered (tests.test_handoff_context.TestPackageSurvivesHandoff.test_caller_is_never_asked_a_question_they_answered)
The teaching claim, as a number rather than a sentence. ... ok
test_the_agent_path_retires_every_desk_question (tests.test_handoff_context.TestTheAgentPathSpecifically.test_the_agent_path_retires_every_desk_question)
THE HEADLINE CLAIM, ON THE PATH THE AGENT ACTUALLY RUNS. ...
  test_the_agent_path_retires_every_desk_question (tests.test_handoff_context.TestTheAgentPathSpecifically.test_the_agent_path_retires_every_desk_question) (key='account_id')
THE HEADLINE CLAIM, ON THE PATH THE AGENT ACTUALLY RUNS. ... FAIL
test_the_agent_path_retires_every_desk_question (tests.test_handoff_context.TestTheAgentPathSpecifically.test_the_agent_path_retires_every_desk_question)
THE HEADLINE CLAIM, ON THE PATH THE AGENT ACTUALLY RUNS. ... FAIL

The first test uses a session typed into the test file. It still passes, although the handoff tool’s key list no longer includes the account. In the tool’s source, a key that is not on that list never reaches the handoff. The second test builds its session from the agent’s own key lists, and it fails.

The fix is to build the test session from the key lists your agent’s code reads and writes. That matters if a green suite is your evidence that a handoff, a summary or a record carries the right data. In the sample, the source records that a green suite once hid a gap like this, and the desk asked which account the call was about. The fix is one more test, beside the hand-typed one, built from the agent’s own lists.

It has two blind spots, and you should know them before you copy it. When we deleted a key the test does not name, all 41 tests stayed green. And a test that loops over a parsed list passes when the list comes back empty. Both are shown below. The second has a simple guard; the first needs you to name the keys that matter.

What you need: Python and the companion pattern patterns/voice-handoff-context from RasaHQ/rasa-community-resources. Its tests need no Rasa install, no licence, no model and no network. Every run on this page is offline. We reran the opening run and the clean suite at the commit the site pins. Our runs were on macOS. To run the tests, open the pattern’s folder and run:

On the unchanged sample, all 41 tests pass.

Why the hand-typed test stays green

A fixture is the input a test sets up before it checks anything. Here it is a session: the facts the agent has gathered about a caller when the caller asks for a person.

The sample agent hands that caller to a human desk with a package. The desk opens with five questions, and the package should answer all of them. One of them is “Which account is this about?”.

The first test builds its session by hand. This excerpt shows the opening lines of that function; it continues in the test file:

def demo_session() -> dict:
    """Session state at the moment the caller asks for a human."""
    session = {
        # --- allowlisted: the state that SHOULD cross -----------------------
        "customer_id": "cust_00417",
        "display_name": "Jordan Rivera",
        "verified_tier": "medium",
        "verified_factors": ["knowledge_passphrase"],
        "channel": "voice:+1-555-0100",
        "goal": "dispute_transaction",
        "goal_label": "Dispute a card transaction",
        "goal_stage": "blocked",
        "account_id": "acc_checking",
        "account_label": "Everyday Checking",
        "card_last_four": "4821",
        "dispute_amount": "$248.00",
        "dispute_merchant": "Northgate Fuel",
        "dispute_date": "2026-08-29",

By the usual standard it is a good fixture. The names match the agent’s memory fields, and the values look real. The test passes this session to the function that builds the package, then checks that no desk question is left.

The running agent does not build its session that way. Its handoff tool copies only the keys on one named list. This excerpt is the loop from the tool’s source:

    # Step 1 — collect indiscriminately. This tool is not the safety boundary.
    session: dict = {}
    for key in _MEMORY_KEYS:
        value = context.memory.get(key)
        if value is not None:
            session[key] = value

A key that is not on that list never reaches the package, whatever the agent’s memory holds. The hand-typed test never reads the list, so it cannot notice when a key is missing.

This is not a made-up risk. The sample once had this exact defect, and the docstring of the second test records it. This excerpt is the docstring’s middle paragraph, unedited and with its source indentation. Left out are its summary line, “THE HEADLINE CLAIM, ON THE PATH THE AGENT ACTUALLY RUNS.”, and its closing paragraph, which begins “The fix is to drive the assertion…”:

        This test exists because the claim once passed while being false. The
        eval suite built its own session dict containing `account_id`, so
        `intent.details` was populated and "which account is this about?" was
        retired — but the agent's own `_MEMORY_KEYS` did not collect
        `account_id`, so a real handoff shipped an empty `details` and the desk
        asked anyway. Green tests, broken claim.

The skill that writes the caller’s profile to memory had the same gap. A comment in its source says it once left out the account and card keys too. The comments put the defect in the key lists, not in anything a model said.

How to build the session from your agent’s own lists

Two lists decide what reaches the package. The skill’s tools write keys to memory, and the handoff tool reads the keys on its list. A key arrives only if both happen. So the fixed test reads both lists and builds its session from the keys they share. The diagram shows the broken copy from the opening: the hand-typed test skips the list, so it never sees the missing account.

In your own project you change two places: your test file, and perhaps the module that holds your key list. The steps below use the sample’s code.

    def _agent_memory_keys(self) -> tuple[str, ...]:
        """Read _MEMORY_KEYS out of the tool module without importing rasa.

        The tool module imports `rasa.mantle`, which the eval suite deliberately
        does not depend on — `make test` runs with no license and no install.
        Parsing the literal keeps this test honest about the real list while
        staying offline.
        """
        source = (
            Path(__file__).resolve().parents[1]
            / "skills" / "human_handoff" / "tools.py"
        ).read_text()
        block = source.split("_MEMORY_KEYS = (", 1)[1].split(")", 1)[0]
        return tuple(
            line.strip().strip(",").strip('"')
            for line in block.splitlines()
            if line.strip().startswith('"')
        )

        keys = set(re.findall(r'memory\.set\(\s*"([^"]+)"', source))
        keys |= set(re.findall(r'_append\(\s*context,\s*"([^"]+)"', source))
        # The `for key in (...)` loop that writes the caller profile.
        for block in re.findall(r"for key in \(([^)]*)\):", source):
            keys |= set(re.findall(r'"([^"]+)"', block))
        return keys

        written = self._skill_memory_writes()
        collected = self._agent_memory_keys()

        # Every detail key the allowlist expects must be both written by a skill
        # and collected by the handoff tool, or it never reaches the package.
        for key in ("account_id", "account_label", "card_last_four"):
            with self.subTest(key=key):
                self.assertIn(key, written, f"no skill writes {key} to memory")
                self.assertIn(key, collected, f"the handoff tool does not collect {key}")

        # Now build a session from ONLY what the agent genuinely produces.
        session = {key: f"value_for_{key}" for key in collected if key in written}

The sample has two more tests that build sessions from the handoff tool’s list. Both use every key on the list, with no intersection. The credentials test names the two credentials the tool collects and checks they are withheld. The every-key test checks that each key is either allowed through or reported as withheld. A hand-picked fixture covers the keys its author thought of; these cover every key on the list.

How to check the fix works

Break the code on purpose and see which test notices.

In a scratch copy of the sample, delete the account line from the handoff tool’s list in skills/human_handoff/tools.py. This is the change as a unified diff, with the file header lines omitted:

@@ -80,5 +80,4 @@
     # is this about?". test_the_agent_path_retires_every_desk_question now
     # exercises THIS list rather than a hand-built one.
-    "account_id",
     "account_label",
     "card_last_four",

Then run the two tests from the opening. These are the two assertion lines from that run, trimmed from their tracebacks:

AssertionError: 'account_id' not found in ('customer_id', 'display_name', 'verified_tier', 'verified_factors', 'channel', 'goal', 'goal_label', 'goal_stage', 'account_label', 'card_last_four', 'dispute_amount', 'dispute_merchant', 'dispute_date', 'attempts_log', 'questions_answered', 'factors_verified', 'confirmed_facts', 'handoff_reason', 'pin_attempt', 'otp_code') : the handoff tool does not collect account_id
AssertionError: None is not true : intent.details is empty on the live path — the desk will re-ask which account this is about

The derived test fails twice. First it fails at the key check and names the missing key. Then it fails at the package, where the account is missing. On the full suite, the other 40 tests pass, including the hand-typed one.

Do the same in your own project. Delete a key your derived test names, in a throwaway copy, and run the suite. If nothing fails, the fixture is not built from the list you think it is.

A key the test does not name can go unnoticed. We deleted the disputed amount from the handoff tool’s list instead, and all 41 tests still passed.

Trade-offs: parse the source or import it

The sample parses source text so its tests run with no Rasa install. You can also import the list, or move it into a small module that does not import Rasa.

This guide recommends importing when your test job can install Rasa. It removes only the reformatting failure on the collected side; the written-keys parser and its comment problem stay. The cost is the install. A licence did not get in the way: with rasa-pro 3.20.0rc1 installed, no licence set and no network, the handoff tool module imported. We have not rerun that import on 3.21.0.dev5, the version the site pins.

Importing covers only the collected side. The skill’s writes are calls inside tool functions, so you still need the regular expressions for them. If your test job cannot install Rasa, parse the source as the sample does.

A fuller test would call the real handoff tool, with a stub memory filled by running the real skill tools. That would also run the tool’s own conversion step, which the derived test skips. It needs Rasa installed, because both tool modules import it. We did not build or run that test, so this guide does not show one. Start with the derived fixture, and treat the end-to-end test as the next step if your test job already installs Rasa.

Limits

• Some values are still typed by hand. After building the session, the test sets four entries itself. One is the list of earlier attempts, in the shape the tool produces after converting it. So that conversion never runs in the test.
• The written-keys parser reads comments. In the sample, the written keys include one entry that is text from a code comment. It does no harm here.
• Keys collected but never written drop out. No dispute tool writes the disputed amount, merchant or date, so the derived session leaves them out. The measurement guide shows that, on the dispute path, the charge the caller picked never reaches the package, and how to fix it.
• None of this tests the model. Every run here is offline. The tests call the sample’s own Python code on sessions built by hand or from parsed lists, and one run imported the handoff tool module with Rasa installed. None of it shows what a model says on a call.
• It catches one kind of drift. It does not catch, for example, a policy string copied into a test and later changed in one place only.

For the wider design of an evaluation set, see the evaluation set guide. For more on breaking code on purpose to check your tests, and on writing the test a surviving break shows is missing, see the mutation testing guide.