Test whether your bank assistant uses the right account

Your bank assistant returns a transaction list and its tests pass. Did it read the account the customer asked for? This guide shows how to check that before approving a trial.

Our fictional current and savings accounts return identical transactions. Checking the response and transaction list approves both reads, including a scripted savings read after a request for current. The test data gives that check no way to distinguish the accounts.

Binding the returned account identity to the requested account rejects the same wrong read. The observations stay identical. The acceptance contract changes.

From the pinned companion’s Mantle directory, after its locked install:

This is the literal SDK/application control output:

current: requested=A-CURRENT; actual=A-CURRENT; payload_matches=true; account_bound_matches=true
wrong_eligible: requested=A-CURRENT; actual=A-SAVINGS; payload_matches=true; account_bound_matches=false
identical_transaction_payloads: true
exact_payload_checks: 2/2
account_bound_checks: 1/2
payload_only_decision: APPROVE
account_bound_decision: REJECT
external_provider_requests: 0

The scoring exercise declares the expected current account and its transaction payload before capturing the two SDK/application controls. The wrong reply is deliberately scripted; the shared transaction data comes from the existing account fixture. Its locked read returns the same fixed transaction for either eligible account.

These are deterministic controls with zero external provider requests. The fractions describe exact checks on those two fixtures. Evaluate live models and customer tasks using a separately labelled, representative trial.

Jacaranda Bank and its records are fictional. This exercise requires a named account confirmation before a conversational read; that policy is a teaching assumption. The standalone selector controls bypass that conversation so you can inspect interpretation and permission separately. The running Rasa example uses a saved proposal and the next user reply before reading.

Find the field the exact-match contract omitted

The original example acceptance contract checks status transactions and exact equality with the declared current-account transaction list. It checks every reference, merchant, currency and amount in that list. Both reads pass.

The repaired contract keeps those comparisons and also checks the actual account and recorded read against the declared current account. It rejects the savings read even though the payload matches perfectly.

The counter-tests show why this is a fixture blind spot. Give savings a different amount in the captured test data, and the payload-only contract rejects it. Correct the actual wrong account instead, and the account-bound contract approves it. Duplicated cases are refused; repeated easy inputs cannot pad the panel.

Use distinguishable fixtures when testing account routing. Keep identity checks as well: the counterexample returns equal payloads from different accounts. The choice-quality gate examines the component’s selected account; the customer-task gate must also examine corrections, confirmation and the final reply.

Put the requested account beside the returned account

The positive control returns current-account rows. The deliberately wrong control returns savings-account rows. Both say transactions and both record one SDK request:

{"account_reads": ["A-CURRENT"], "control": "current", "external_provider_requests": 0, "sdk_requests": 1, "status": "transactions"}

{"account_reads": ["A-SAVINGS"], "control": "wrong_eligible", "external_provider_requests": 0, "requested_account": "A-CURRENT", "sdk_requests": 1, "selected_account": "A-SAVINGS", "status": "transactions"}

Ask the implementation team for these fields together: the current user request, the named proposal, any correction or confirmation, the actual read and the final answer. A customer can change their mind. In that case, judge the delivered task against the explicit change; retain the earlier interpretation error separately.

The Rasa walkthrough shows the next step. A local model proposes an account, the tool names it in a question, and a saved proposal binds the next user reply to the read. This gives the customer a chance to correct the choice. Test that recovery instead of assuming every named question is noticed or understood.

Keep permission and task correctness separate

A trial owner needs two answers: did the assistant return the right account, and was it allowed to read that account at that moment? A correct account label can still refer to stale or unauthorised state.

The account store checks current revision, offered membership, owner and active state under its read lock. The stale-state control retires an account while the scripted decision is pending. The resulting read is refused:

{"account_reads": [], "control": "retired_during_decision", "external_provider_requests": 0, "sdk_requests": 1, "status": "stale_options"}

That refusal preserves the exercise’s permission rule. The customer still needs a fresh proposal to obtain an answer. Record both facts; calling it simply “failure” loses the reason the assistant stopped.

The proposal controls check cancellation and single-use consumption. The decorated-tool controls use actual user-event objects through a real loopback worker, including a no reply and a changed-topic yes. The retryable-error control uses a scripted service error to check one attempt and no read. These are native application controls, not rates observed across a customer population.

Approve a trial with labelled outcomes

Give each test case an expected account and expected disposition before the run. Include the account identity alongside the expected transaction payload. The scored exercise above shows how duplicated fixture data can hide a routing error. Include requests that name one account, requests that leave it unclear, corrections, cancellation, stale state and service errors. Keep those labels with the case definition and the pinned model and application versions.

The runtime used in the linked walkthrough starts with one pinned English Laya checkpoint. Hosted Jev is optional and needs separate qualification. A few runnable examples establish that the integration works; a purchase decision needs a representative labelled evaluation for your own customers and languages.

Use this decision record with the team:

Assign an owner for each unresolved item. Advance the trial when the labelled outcomes and controls meet its stated acceptance criteria. A wrong account with returned rows blocks choice-quality acceptance. A correct cancellation belongs in the honoured-cancellation count. A service outage belongs in availability follow-up, even when the application safely returns no data.

Count work before claiming savings

The demo records one SDK request in both the accepted and stale-state cases. It sends zero external provider requests because the replies are scripted. Those counters locate work in the application; they do not measure billing.

In the runnable conversation, each proposal makes one decision request. Confirmation uses the saved proposal without calling the decision model again. Mantle’s ordinary orchestrator and writer remain responsible for the conversation. Their requests need their own accounting; the worker counter does not include them.

Keep billing records and measured runs private. Include the issued Rasa licence, writer-provider setup and local-runtime operation in the trial plan. A local backend needs no TypeSafe key; it still has an operating footprint. The artifact manifest and dependency locks in the companion make that footprint rebuildable.