Measure a handoff before you claim it worked

Aurora Home Insurance, a fictional insurer, has shipped a redesigned agent-to-human handoff. Its voice agent now passes a structured context package to the human desk instead of a one-line reason. The launch-review slide says:

Redesigned handoff packages cut average handle time by 30%.

The only evidence behind it is a passing offline test suite. Here is what that evidence supports, and what it does not:

The artefact linked from the slide is the tail of a make test run in the companion pattern:

----------------------------------------------------------------------
Ran 41 tests in 0.004s

OK

No call was recorded and no desk agent was timed. A search of all 25 files in the pattern finds no “30%” and no mention of handle time. The team had a real result and reported a different one.

Aurora is invented; the code is real. It is patterns/voice-handoff-context in RasaHQ/rasa-community-resources at 69e27b6, the revision pinned for Rasa Pro 3.20.0rc1. Its fixture call is a disputed card payment on a bank account, which the fictional team kept for its pilot.

The position this guide takes: at launch, report the count of fixed desk questions the package retires, proven offline, and nothing larger. The cost is a smaller slide, a measurement you still owe, and a script that sees only the questions written into it.

What does the passing test prove?

This section gives you the denominator the test counts against, and the before-and-after it proves.

“Callers repeat themselves less” has no denominator. The pattern fixes one before any caller exists: five questions a desk agent asks when nothing was transferred, each mapped to the package field that makes it unnecessary (handoffpkg/desk.py, lines 53–59):

_QUESTION_RETIRED_BY: dict[str, str] = {
    "Can I take your name?": "identity.display_name",
    "Can you confirm your date of birth?": "identity.verified_tier",
    "Which account is this about?": "intent.details.account_id",
    "What are you calling about today?": "intent.goal",
    "Have you tried anything already?": "attempts",
}

The denominator is the same five for every handoff, whatever the caller or case, so one handoff compares with the next and the old design with the new. The counting function, unanswered_questions, calls itself “The measurable form of the teaching claim” (lines 178–185).

The baseline test builds a package from one free-text handoff_reason and gets all five questions back; the agent-path test gets none back. One offline run on 21 September 2026 passed both, along with the permissions test this guide uses later.

python3 -m unittest -v \
  tests.test_handoff_context.TestPackageSurvivesHandoff.test_the_catalogs_current_handoff_answers_nothing \
  tests.test_handoff_context.TestTheAgentPathSpecifically.test_the_agent_path_retires_every_desk_question \
  tests.test_handoff_context.TestPackageSurvivesHandoff.test_desk_permissions_follow_the_tier_not_the_package

test_the_catalogs_current_handoff_answers_nothing (tests.test_handoff_context.TestPackageSurvivesHandoff.test_the_catalogs_current_handoff_answers_nothing)
Baseline: one free-text reason retires none of the desk's questions. ... ok
test_the_agent_path_retires_every_desk_question (tests.test_handoff_context.TestTheAgentPathSpecifically.test_the_agent_path_retires_every_desk_question)
THE HEADLINE CLAIM, ON THE PATH THE AGENT ACTUALLY RUNS. ... ok
test_desk_permissions_follow_the_tier_not_the_package (tests.test_handoff_context.TestPackageSurvivesHandoff.test_desk_permissions_follow_the_tier_not_the_package)
A medium-tier caller must not be actioned for an irreversible change. ... ok

----------------------------------------------------------------------
Ran 3 tests in 0.001s

OK

The five questions are the pattern author’s, written for a fixture desk. Take yours from your desk’s call guide or QA scorecard, version the list, and date every figure against a version. If the script changes, the number changes, though no caller behaved differently.

Which count belongs on the slide?

The handoff tool returns two counts in the same response, and only one of them has a fixed denominator.

They disagree because the dispute skill appends four script questions (skills/dispute_transaction/tools.py, lines 98–103 and 135) and never “Have you tried anything already?”. The package still carries three attempts, so the scripted check counts that question as answered. Pick one definition and write it into the review.

An offline run of the real dispute and handoff tools printed both counts side by side.

dispute_* keys in memory: ['disputed_txn_id', 'disputed_txn_label']
transferred: {"attempts_carried": 3, "goal": "dispute_transaction", "identity": true, "questions_retired": 4, "verified_tier": "medium"}
desk_still_needs_to_ask: []
hint: Context package delivered. Tell the caller the handoff id and that they will not need to repeat themselves. Do not read back any withheld field.
intent.details: {"account_id": "acc_checking", "account_label": "Everyday Checking", "card_last_four": "4821"}
ASKING FOR line: ASKING FOR  Dispute a card transaction [account_id=acc_checking, account_label=Everyday Checking, card_last_four=4821] (stage: blocked)

Where does the handoff lose the disputed charge?

This section shows the one fact the companion agent drops, and why the count cannot notice.

The handoff tool collects the disputed amount, merchant and date, but nothing on the dispute path writes them. The regression test checks three other keys, so the package names a card dispute on Everyday Checking but not which charge, and the count says the desk needs to ask nothing.

Our inference, which no test checks, is that the desk will have to ask which charge it is. Stop customers repeating themselves after a handoff reaches the same finding from the design side.

Three more limits decide what the count is worth:

session["verified_tier"] = "medium"
session["goal"] = "dispute_transaction"
session["questions_answered"] = "\n".join(DESK_OPENING_SCRIPT)
session["attempts"] = [
    {"action": "send_otp_sms", "outcome": "failed", "code": "delivery_failed"}
]

Can the agent promise callers they won’t repeat themselves?

Not unconditionally, and this section shows the two instructions that contradict each other.

What the desk may do is worked out from the transferred tier when the package is read. The demo caller is hard-coded at medium, and the permissions test asserts that account details may be discussed but irreversible changes may not.

Raising a dispute needs high (_DISPUTE_MIN_TIER in skills/dispute_transaction/tools.py, lines 58–60), and the comment there says the medium caller “is what makes the handoff happen”. So the desk screen tells the human to step the caller up before the dispute can go through. That is correct design, and it hurts a handle-time slide.

tier = package.identity.verified_tier
actions = ["Answer general questions", "Explain what the agent already did"]
if tier_at_least(tier, "medium"):
    actions.append("Discuss account-specific details")
else:
    actions.append("DO NOT discuss account-specific details — identity not established")
if tier_at_least(tier, "high"):
    actions.append("Action irreversible changes (transfers, card reissue, SIM swap)")
else:
    actions.append("DO NOT action irreversible changes — step the caller up first")

How do you measure AI agent human handoff success after launch?

This section gives you the order of work for the numbers only real calls can produce.

The whole suite runs from patterns/voice-handoff-context:

In the saved run on 21 September 2026 (Python 3.14.3), all 41 tests passed over synthetic sessions, with no network, model or licence. None places a call, times a human or reads a production session.

The companion tests none of these steps. Step three depends on what your contact centre records:

What goes on a launch-review slide you can defend?

Here is Aurora’s slide rewritten so that someone in the room can check every line.

The same suite shows the package withholding seven planted sensitive values and a field nobody anticipated. Once the owed changes are decided, that is reasonable to ship on offline evidence. Announcing a business effect before anyone has counted a call is not.

Design a handoff a customer can trust covers consent, the context packet and recovery when the receiving team is unavailable. The tool and memory scope tutorial builds a bank agent that makes the caller verify exactly once.