How to stop an AI agent acting on an outdated confirmation

Put the terms and a version in the question

The confirmation question is a response template. Rasa fills it from memory, the values the agent keeps during the conversation. This is the sample’s skills/subscription_change/responses.yml (excerpt):

responses:
  utter_confirm_subscription_change:
    - text: >
        Please confirm this change to your {change_subscription}:
        {change_label}. It takes effect {change_effective}. What changes for
        you: {change_entitlements}. (Change {change_tag}.) Shall I make this
        change? If you meant a different change, say which.

change_tag joins the subscription, the change type and the record’s version number, for example WS-SUB-3304 stop_renewal r4. In lib/subscriptions.py it is one line:

def change_tag(sub_id: str, change_type: str, revision: int) -> str:
    return f"{sub_id} {change_type} r{revision}"

In your project, put every term the customer is agreeing to into the question. For a payment, a plan or a contract change, that usually means:

• the amount
• any fees
• the date it takes effect
• what the customer loses or gives up
• the account, line or policy it applies to.

Let only a tool write those values

A tool called select_subscription_change asks the subscription service what the change would do. It writes the answers into memory. The fields are never llm_settable, so the model cannot set them. The sample’s skills/subscription_change/memory.yml (excerpt):

# Written only by select_subscription_change (never llm_settable), so what the
# engine reads back for confirmation is always the subscription service's.
...
schema:
  public:
    change_subscription:
      type: text
    change_tag:
      type: text

The tool writes the values through its ToolContext. Rasa passes it in as a keyword argument named context, so the parameter must have that name. From skills/subscription_change/tools.py:

def _write_memory(context: Optional[ToolContext], values: dict) -> None:
    if context is None:
        return
    for key, value in values.items():
        context.memory.set(key, value)

The skill then holds the tool that makes the change. This is the top of skills/subscription_change/skill.md (excerpt):

tool_constraints:
  - apply_subscription_change:
      requires: session.subscription_change.change_tag
      requires_confirmation:
        enabled: true
        utter_for_confirmation: utter_confirm_subscription_change

requires keeps the tool away from the model until a change is selected. requires_confirmation makes Rasa send the question above and wait for the answer.

Compare before you act

The tool that makes the change runs the comparison first. It finds the question Rasa sent by reading the conversation’s events. This is lib/conversation.py (excerpt):

def conversation_from_events(events: Iterable[Any]) -> Conversation:
    users: list[str] = []
    question = None
    answered = False
    for event in events:
        kind = type(event).__name__
        if kind == "UserUttered":
            text = getattr(event, "text", None) or ""
            if text.startswith("/"):
                continue  # /session_start and other intents are not the member's words
            users.append(text)
            if question is not None:
                answered = True
        elif kind == "BotUttered":
            metadata = getattr(event, "metadata", None) or {}
            if metadata.get(UTTER_ACTION_KEY) == CONFIRM_UTTER:
                question = getattr(event, "text", None) or ""
                answered = False
    return Conversation(confirmation_question=question, confirmation_answered=answered, user_messages=tuple(users))

The question is the latest bot message whose utter_action metadata names the confirmation response. It counts as answered only if a customer message follows. Messages that start with /, such as /session_start, are not the customer’s words.

Then it asks the service what the change would do now, and compares. This is the comparison in change_facts, in lib/subscriptions.py (excerpt):

        disclosed = (
            tag_rev == sub["revision"]
            and effective == effective_text(option["effective_at"])
            and entitlements == option["entitlements"]
            and asked
            and effective in question
            and entitlements in question
        )

Line by line, the yes counts only if:

1. the version in the tag is still the record’s version
2. the date read out is still the date the service gives now
3. the line about what the customer keeps and loses is still the service’s line
4. the tagged question was asked and answered
5. the date and that line both appear in the question Rasa sent

If any of these is false, the tool returns blocked before it sends a command. The execution guard chapter of the guarding tutorial explains why this check belongs inside the tool, not in the skill’s configuration.

This check narrows the gap but does not close it. Another system could still change the record between the comparison and the command. In the sample, the command to the service carries the subscription, the change type and a request key, but no expected version or terms. In your own project, let the system that makes the change refuse it too. Send an expected version to your core banking or billing API, use an If-Match header if it supports one, or check the amount inside the same database transaction as the write.

Refuse with the new terms

The sample’s refusal carries the current terms and the reason, so the agent has something to tell the customer. This is the result from the Style Box run (excerpt, trimmed to the fields used here):

{
  "status": "blocked",
  "reason": "benefit_loss_hidden",
  "effects": 0,
  "current": {
    "effective": "2027-01-01 00:00 UTC",
    "entitlements": "October box ships 2026-10-05 (paid); no charge on 2027-01-01; no refund"
  },
  "why": "The October Style Box was charged early, on 2026-09-30, while this chat was open."
}

Step 5 of the skill tells the model what to do with it: “When a result is blocked with benefit_loss_hidden, the subscription changed while you were talking. Say why, call @tool.select_subscription_change again and tell the member the new effect. Apply only after they agree.” Selecting again writes the new terms and version into memory, so Rasa’s next question carries them.

Take no version from the model

The tool that makes the change takes only the subscription and the change type from the model. This is its signature in skills/subscription_change/tools.py:

async def apply_subscription_change(subscription: str, change_type: str, context: ToolContext = None) -> ToolResult:

The version, date and terms come from memory that only the select tool writes. A model that passed the version could pass the wrong one. The sample’s test test_no_tool_takes_dates_amounts_or_facts fails if any tool gains a parameter for a date, an amount or a revision.