Rasa dispatch confirmation test coverage

Your assistant asks the caller to confirm a dispatch. The caller then changes the requested service. Would your tests catch code that accepts the old confirmation?

In the fictional Rasa roadside example, removing the service check left the original guard suite passing. That suite had confirmation tests. They did not independently cover this condition.

Zero failures means the suite missed that removed condition. These are offline guard checks.

The original suite ran 24 tests; the revised suite ran 26. These are offline Python source checks, not agent conversations. No Rasa runtime, model, speech service, real provider, or actual dispatch was involved. The recorded results and original-suite comparison preserve that boundary.

Not claimed: Trained agent, model, voice, vendor, or current global release acceptance.

The separately audited baseline ran 26 guard tests without skips under Python 3.14.6. The lock records Rasa Pro 3.21.0.dev5, but this source check did not import or execute it. The bounded proof records those limits; the earlier comparison and mutation logs remain separate evidence.

Identify what the confirmation actually covers

The guard in the companion example checks four facts. This excerpt omits its signature’s type annotations:

return (
    (memory.get("pending_draft_ref") or "") == draft.tag
    and labels["location"] in question
    and labels["service"] in question
    and conversation.confirmation_answered
)

The draft tag identifies the request and its version. The other checks bind the confirmation to the place, service, and caller’s answer. A test that makes several of these false cannot tell you which one stopped the dispatch.

The original results alone do not identify whether the service check always held or another check blocked first. An independent control resolves that ambiguity.

Run each omission against the published source

The companion’s diagnostic parses the guard and removes one condition in memory. It leaves the file unchanged. This is mutation testing: make one deliberate fault, then ask whether a test detects it.

Use a complete checkout so the guard’s companion-contract check is present:

git clone https://github.com/RasaHQ/rasa-community-resources.git
cd rasa-community-resources
git checkout af57ab393bbffe56a9207afd1ac03a68ef9e423b
cd examples/mantle-voice-insurance-roadside-claude
python3 -I -S -B controls/confirmation_trace.py --tests full

The diagnostic uses Python’s standard library. Python documents isolated mode (-I), disabled site initialization (-S), and disabled bytecode writes (-B). This remains a source test.

The retained intact output ends:

Ran 26 tests in 0.004s

OK

Now remove each condition separately. A nonzero result is expected for a detected mutation, so preserve the exit code rather than hiding it:

for condition in drop_location drop_service drop_tag drop_answer
do
  status=0
  python3 -I -S -B controls/confirmation_trace.py --tests "$condition" || status=$?
  printf '%s exit=%s\n' "$condition" "$status"
done

Each mutation should end with one failing test and exit 1. An import error or a skipped test is a different result. Inspect the assertion before calling the mutation detected.

For example, the service omission fails this control:

test_current_version_does_not_approve_a_different_service (test_guard.GuardTests.test_current_version_does_not_approve_a_different_service) ... FAIL
AssertionError: Tuples differ: ('succeeded', 'verified_fixture_receipt', 1) != ('blocked', 'wrong_incident_location', 0)

The failure shows one fictional dispatch where the test requires zero. The shared reason label is wrong_incident_location; it does not tell you which of the four confirmation facts failed.

The logs were recorded before integration. A byte comparison binds the published diagnostic, guard, and tests to those recorded runs. Their elapsed time is one observation, not a performance claim. Public traceback paths use <example>/ in place of the private host prefix; the export manifest records original and public hashes. Test results are unchanged.

source_root=$(git rev-parse --show-toplevel)
before_root=$(mktemp -d)
git -C "$source_root" worktree add --detach "$before_root" \
  0285d2b86cc3d1fde13881ad68c5288429bce8e4
python3 -I -S -B controls/confirmation_trace.py \
  --project "$before_root/examples/mantle-voice-insurance-roadside-claude" \
  --tests drop_service

Make the missing condition the only possible blocker

The service control starts with an answered tow confirmation. It changes the service to a jump start, then supplies the current request reference while retaining the old question. The place and answer still match. Only the service check can stop that stale confirmation.

The tag control is different. It deliberately puts a different request reference in memory while keeping the place, service, and answer valid. That is a constructed inconsistent-memory control. Showing that the running engine produces this state would require a recorded state-transition trace.

The service control keeps the reference, place, and answer valid. Removing the one false condition permits a fictional dispatch, so the test detects that removal.

Decide what to keep

Do not add a test merely to make every mutation fail. First build a state where all the neighboring conditions hold. Then ask how the application could reach it.

For a normal service edit, the independent test protects a real path through the example. For a hand-edited reference, retaining the check is a defense-in-depth decision with an extra condition and test to maintain. An equivalent condition may instead be removed after reviewing the reachable states.

Take the method to one confirmation rule in your own agent: list its conditions, remove each separately, inspect the surviving mutations, and record why each retained condition deserves its test.