Roll out an agent with a working stop button
by Rasa team
Before expanding access to an agent, prove that somebody can stop it and that customers still have a usable route to help. A rollback document is only a proposal until the team exercises it.
This guide is for the platform engineer or operator responsible for a Rasa deployment. It describes an operational contract to adapt to your infrastructure; it does not prescribe an unverified deployment command.
Name the unit you will release
Record the application revision, Rasa version, model configuration, tool configuration, and evaluation dataset version together. A model or tool change can alter behavior even when application code is unchanged. Keep credentials in the deployment’s secret system, not in the release record.
For a fictional travel assistant, the first slice might serve itinerary questions for a small opted-in pilot. The size is your team’s decision based on support capacity and observation; there is no universal safe percentage in this guide.
| Dependency | Check before enabling the slice | If it fails |
|---|---|---|
| Booking lookup | Authorized fixture returns the expected record | Disable the affected workflow |
| Model provider | Request completes within the team’s latency budget | Use the verified fallback or stop the workflow |
| Support queue | Test request appears in the receiving system | Keep human-handoff flows unavailable |
| Secrets and licence | Service can start with the deployed configuration | Hold the release |
| Operator access | Named on-call person can change routing | Hold until an owner is reachable |
Use synthetic or authorized test records for these checks and label them. The results are evidence only if the dependency was actually exercised.
Watch outcomes, not just HTTP status
An agent can return HTTP 200 while giving the wrong answer. Track request counts, task outcomes, tool errors, latency, and handoff delivery separately. Define each denominator and make unresolved outcomes visible.
Connect traces through an interaction identifier, but avoid placing full conversation text or credentials into general-purpose logs by default. Agree retention and access with the data owner. NIST’s AI actor tasks distinguishes deployment and operation from other lifecycle work; our checklist translates that responsibility into practical checks.
Write an alert as an action: “If the booking lookup is unavailable, the on-call operator disables the booking workflow and confirms the alternative support route.” A dashboard with no responsible person is an observation surface, not a response plan.
Rehearse rollback before the pilot
- Save the currently accepted application and configuration versions.
- Enable the candidate in an isolated environment with an authorized test interaction.
- Trigger a representative dependency failure.
- Use the actual routing or deployment control to stop new traffic to the candidate.
- Confirm that the prior version or alternative support route handles a new request.
- Inspect any in-flight action and the receiving system for partial or duplicate work.
Record timestamps and resulting states. Measure recovery time from the drill instead of inventing a target achievement. The target belongs to your service agreement; the measurement tells you whether this mechanism can meet it.
The awkward case is an external action committed immediately before rollback. Reverting application code cannot undo a paid booking change. The rollout plan needs an owner who can reconcile that resulting record, and the agent needs a guarded action boundary before it reaches production.
Make expansion a separate decision
Review the pilot’s task mix against the evaluation set, check fallback demand against the receiving team’s capacity, and list unexplained failures. Use the release decision record to expand, hold, or stop explicitly.
Your next step is one rollback drill with a named observer. If the stop mechanism does not work, fix it before increasing access. For Rasa’s licence configuration details, consult the current licensing documentation; the deployment contract here remains specific to your own infrastructure.