Guide
How Rasa reply guards miss invented transfer confirmations.
Stop an agent from inventing a transfer confirmation even when it makes no tool call.
Key takeaways (3)
- Test invented acceptance text with an empty list of tool calls.
- Let the transaction host own the review and outcome messages.
- A missing reply leaves the transfer outcome unresolved.
Your transfer tool checks consent before moving money. The model cannot bypass it. But can the model tell the caller that money moved without calling that tool at all?
Our SilverLake example had a reply guard that removed model text beside a transfer tool call. That condition left a different path open: the same claim as text alone. The later test supplies this fabricated model answer with no tool calls:
The transfer request was accepted. Accepted does not mean posted.
This is an injected test input, not a recorded bank response or a measured model hallucination. The test checks whether the application lets that sentence through. Guarding the write and guarding what the assistant says about the write are separate jobs.
Inspect the case with no tool call
The earlier reply policy checks for protected transfer calls. When one accompanies model text, it removes the text and keeps the call. The tool can then produce the real review or result.
That makes sense for a prediction spoken before the tool runs. It does not cover a model answer containing only text. With no matching host-delivery marker and no protected tool call, that source returns the payload unchanged. This is a source-level finding from the condition, not a new execution of the earlier version.
A test containing both a tool request and an invented result can therefore pass while missing a text-only false receipt. Add the empty-call-list case to your reply tests, even if your transaction tests already refuse every unauthorised write.
| Model output and delivery state | Earlier source condition | What the later policy does |
|---|---|---|
| Acceptance text beside a transfer tool request | Matches the protected tool name | Removes the text and preserves the requested call. |
| Acceptance text with no tool calls | Does not match that condition | Retries the recognised host-only wording, then uses a bounded fallback. |
| Ordinary text with no tool calls | Does not match that condition | Leaves ordinary replies alone. |
| Acceptance text after a matching host delivery | Delivery marker matches this caller and turn | Removes extra model text and ends the turn with listen. |
The table compares application policy. It does not measure how often a model produces any row.
Make the transaction host own its words
In this example, the host is the application code that holds the transfer review, checks consent and records the operation. Its adapter sends the caller a review or outcome message. A model-written imitation carries none of that provenance.
The merged reply policy adds a second check. Without a matching delivered-host marker, it recognises three wording patterns: a transfer review, a statement that the transfer was accepted, and “accepted does not mean posted”. It requests another model response instead of publishing that text.
The retry instruction tells the model to ask for transfer status when an earlier outcome is unknown. It explicitly forbids requesting a new transfer in that situation. A caller asking what happened must not accidentally authorise another attempt.

This diagram describes the reply policy, not a bank’s transaction state machine. Neither removing a sentence nor ending a conversation rolls back an operation.
Check the negative input and its control
The public guard test supplies two model-written imitations: a transfer review and the acceptance sentence above. Its payload helper sets the tool-call list to empty. Neither fixture asks the host to perform a transfer.
For each sentence, the test expects retries up to the policy’s limit. It then checks the exact fallback. It also asserts that the fallback does not contain “nothing was sent”. That last assertion matters as much as blocking the invented acceptance.
A separate control in the same file leaves an ordinary simulator reply unchanged. It then records a host delivery and checks that the extra model acceptance text is removed. The policy must distinguish a forged receipt from speech that follows a real host message.
The retained native test output includes these exact lines:
test_imitated_review_or_acceptance_is_retried_then_replaced (test_reply_policy_guard.HostOnlyWording.test_imitated_review_or_acceptance_is_retried_then_replaced) ... ok
test_ordinary_text_and_a_real_host_delivery_are_untouched (test_reply_policy_guard.HostOnlyWording.test_ordinary_text_and_a_real_host_delivery_are_untouched) ... ok
The public excerpt also retains the suite’s original total and result. Its export manifest gives the original log hash, selected line numbers, excerpt hash and source-file hashes. The two named tests passed in that retained run; this is not a measurement of model reliability.
The run used Rasa Pro 3.21.0.dev5 on 12 November at 23:12 UTC. The source owner attests that the working-tree source was committed later as c588432c. We compared the complete policy and guard-test files with the upstream merge a373b475: those two files are byte-identical. Other project files changed before merge. The original test therefore remains a working-tree run with an attested source relationship, not an execution of the later merge commit.
You can inspect the copied policy and test alongside their upstream licence. They contain application code, not proprietary SDK code. The test file imports the native SDK; use the pinned example’s own setup to run it. This article adds no new execution or dependency environment.
Refuse certainty in both directions
After the retry limit, the policy’s fallback is:
I cannot confirm a transfer review or outcome from that reply. Check the transfer status before doing anything else.
It does not say that the transfer succeeded. It also does not say that nothing happened.
A missing delivered-message marker records a communication fact: this policy has no matching host delivery for the turn. The transaction outcome still needs the host’s operation record. A request may have crossed the service boundary before the reply was lost. Saying “nothing was sent” could prompt the caller to repeat an operation that already exists.
This is where reply design meets recovery design. Give the caller a status check using the existing operation’s identity. Do not treat a cautious fallback as permission to generate a new operation key. The transfer-outcome guide examines that separate transport and recovery boundary.
Keep delivery ownership inside the current turn
The delivered marker is associated with a sender and the identical runtime turn object. A marker from another caller or a previous turn cannot suppress this caller’s reply. Old records also expire, and the store has a fixed size.
When a matching host message has been delivered, the policy removes extra model text and replaces further calls with listen. In this agentic example, that ends the turn. The caller keeps the host’s wording rather than hearing a second model-generated version of the review or outcome.
This is presentation control. It cannot grant consent, post a transfer or cancel an already submitted request. Those decisions remain inside the host and its operation journal. Keep both kinds of tests: one for what the transaction code permits, another for what the reply lets the caller hear.
A stricter design would prevent the model from speaking about transfer outcomes at all and render every such answer from host state. That avoids guessing which paraphrases to catch, but costs conversational flexibility and requires explicit handling for every supported outcome. The example retains ordinary model speech and tests the narrower boundary above. Choose that trade-off deliberately rather than mistaking a regex for transaction authority.
What should a reply test prove?
A protected tool request with premature result text must preserve the call while removing the prediction. An invented receipt with no calls must not become a host receipt. Ordinary speech must still work. A real delivered host message must not be followed by another acceptance. Finally, every fallback must preserve uncertainty about an unresolved operation.
These are different inputs. Passing one does not imply the others passed. Keep the injected inputs visible in the test report so a green transaction suite cannot conceal an untested speech path.