Guide · AI product engineer
Fiserv Banking Hub testing: agent bug or API gap?
Prove why Fiserv Banking Hub's history definition rejects a transaction before changing your assistant, then repeat the example from a printed seed.
Key takeaways (3)
- Test the vendor's definition before changing an assistant that says transaction history is unavailable.
- Prove with a test that a gap is in the vendor's definition, not in your code.
- A printed seed repeats the starting data, not the model's wording.
You are building a Fiserv Banking Hub assistant that must list recent transactions. It says history is unavailable because of a schema defect. Before changing the assistant, test whether the vendor’s definition can represent the request and reply you need.
In this release, it cannot. The history definition makes an account key and a transaction key satisfy two incompatible types: string and object. A test against the downloaded release rejects a request naming an account and a reply containing one transaction. The defect is in that published definition, not in the assistant’s phrasing.
The contradiction to check before debugging the assistant
| What the caller needs | What the release requires | Result checked by the example |
|---|---|---|
| A history request naming an account | Its account key must be both a string and an object | The validator rejects the request |
| A reply containing one transaction | Its transaction key must be both a string and an object | The validator rejects the reply |
Step 6 shows the two type requirements in the source and the three tests that pin the defect. You run those tests against the same downloaded release file. A green result there means the tests reproduced the contradiction; it does not mean history is working.
The recorded typed run at seed 295185 gave this answer:
You: What did I spend recently on DDA-A?
Assistant: Recent transaction history for your DDA-A account is not available in this banking release due to a schema defect.
Other unavailable answers have different causes. An SDA account has no sourced balance example, and this release has no payoff code. The same run replied:
You: What is the balance of SDA-A?
Assistant: The balance information for your SDA-A account is not available in this banking release.
You: What is my payoff on LOAN-A?
Assistant: The payoff information for your LOAN-A account is not available in this banking release.
Those two replies omit the host’s specific reasons, and their saved records contain no matching tool entry. Record completeness was not tested. Step 5 treats these as missing evidence to investigate, separately from the schema contradiction that Step 6 reproduces.
Suppose you are building an assistant that has to answer balances, a loan payment, recent transactions, and a loan payoff from a bank’s public API definitions. The API cannot supply all of that. The question is not whether your assistant can say “not available”. It is who says it, and whether you can tell.
In this tutorial you run the example against a small simulated bank. It is not Fiserv’s software. It is a mock written for this example, built from Fiserv’s own release files while the example starts, and every message it sends or receives is checked against them. Nothing talks to a real bank, and nothing here contacts Fiserv after you download the release file. All people, accounts, and amounts are fictional.
You ask the assistant what the release can answer, then what it cannot. You compare each reply with the host results in the saved record. You prove with a test that one gap sits in Fiserv’s definition rather than in the example’s code. Then you repeat the whole conversation from a printed seed, and again from a second seed, to see what a seed may change and what it must never change.
Not claimed: Trained agent, model, voice, vendor, or current global release acceptance.
What you will learn
- How to compare a “not available” answer with saved host results and identify gaps in the evidence for that conversation.
- How to read an API definition for what it does not promise: a field described without its values, and a definition that contradicts itself.
- How to answer “not available” with a reason and never fill a gap with a nearby number, such as another balance or a sum of parts.
- Who can say “not available” in this design: the host (the example’s own code, which the model reaches only through tools) returns a closed reason code, and the model words the sentence. The model can also decline without asking the host.
- How to show, with a test, that the vendor’s own definition rejects a history reply that contains a transaction.
- What a seed is: a number that repeats the starting fictional data. Which answers a seed may change, and which it must never change.
- Why a reply that is valid against a schema is not the same as a success in this release.
What you need
- Three keys: the keys page explains how to get each one: a free Rasa Developer Edition license (
RASA_LICENSE), an OpenAI API key (OPENAI_API_KEY), and a Deepgram API key (DEEPGRAM_API_KEY). OpenAI and Deepgram usage can cost money, so check both accounts first. - A GitHub account with Codespaces: this example runs only in GitHub Codespaces, a development environment in your browser. Running it on your own computer is not tested or supported.
- No Fiserv key or password: in step 1 you download Fiserv’s public API definitions. The file is free and public, and you do not need to sign in to get it (
README.md:49). - A microphone is optional: you type every step in this tutorial, and the README says typing does not test microphone input or spoken replies.
You should know a little Python and Git, and have built a first assistant on a bank mock, for example the SilverLake voice agent tutorial. This tutorial does not walk through a transfer.
The code is in the examples/fiserv-banking-hub-11.0.0.2026.4.1/ folder of RasaHQ/rasa-community-resources.
Line numbers below refer to commit 84a145aae7360e7d8d8b4d6db6ec05ca7bee6104. Every recorded run in this tutorial used that commit.
The README’s Codespaces button opens the main branch, so what you run from it can differ from the recordings. Step 1 gives a link that names the recorded commit. We have not tested that link in a browser. The README at that commit is the one this tutorial followed. The current README may differ.
Step 1. Put Fiserv’s release file in place
Save the three keys as personal Codespaces secrets, with this repository chosen under Repository access (README.md:6, 48).
Open the Codespace at the recorded commit, not from the README badge. The README’s badge names the main branch.
Use this link. It names the commit and this example’s environment. We have not opened it in a browser to a running Codespace ourselves. We have no record of what GitHub does with it when it asks you to sign in.
We did create a Codespace at this commit with the GitHub command line. In its terminal, HEAD was this commit and the source tree was clean. The setup configuration, the lock file, and the Makefile matched the pinned versions (pinned-route receipt):
gh codespace create --repo RasaHQ/rasa-community-resources --branch 84a145aae7360e7d8d8b4d6db6ec05ca7bee6104 --devcontainer-path .devcontainer/examples-fiserv-banking-hub-11.0.0.2026.4.1/devcontainer.json
Whichever way you open it, check where you are. In the terminal, run git rev-parse HEAD. It should print 84a145aae7360e7d8d8b4d6db6ec05ca7bee6104. The terminal should start in the Banking Hub folder (README.md:6, 86).
Then get Fiserv’s release file:
- On your own computer, open Fiserv’s Banking Hub Resources page: https://developer.fiserv.com/product/BankingHub/docs/resources/resources.md?branch=main
- Under API Specification, click Download API Specifications (v11.0.0.2026.4.1).
- A file named
esf-service-swagger-release-11.0.0.2026.4.1.zipappears in your Downloads folder. Do not open or unzip it. - In the Codespace’s left panel, right-click the folder
fiserv-banking-hub-11.0.0.2026.4.1, choose New Folder, and name itvendor-inputs. Right-click that folder, choose Upload…, and pick the ZIP.
Checkpoint: the file is about 1.1 MB (the recorded download was 1,144,551 bytes) and sits inside vendor-inputs/ in the example folder (README.md:61-82). This example is pinned to that one release. It checks the file when it starts and refuses any other file, even one that carries the right name (README.md:226; VENDOR-FILES.md:5-14, 19-21; the test InputHandling::test_wrong_bytes_fail_even_when_named_correctly in the recorded run). The recorded download had the pinned SHA-256 hash, 844bb524…f2fd3, and is described in the download receipt.
That strictness is a choice. The example does not yet check whether a different archive would still work. It says so (VENDOR-FILES.md:19-21). If you build on a vendor’s release file, decide early whether a changed file stops you or warns you.
Step 2. Start a run and write down the seed
In the terminal at the bottom of the Codespace, run:
make run
Leave this terminal running. It keeps printing while the server runs, and you will read it again in step 5.
The example starts with fictional balances and loan payments, drawn from a number called the seed. The first line the example itself prints gives that number:
Demo data seed: <n>; set DEMO_SEED=<n> to get the same data again.
The launcher prints it before it reads the release file (bh_demo/launcher.py:82), so it is the first line. Write it down. If you restart the server with another seed below, write that seed down as well. The seed of the run you use for the seven questions is the one that counts.
Checkpoint: look for these lines, in this order. Warning and log lines appear between them. That is expected.
-
The seed line above.
-
This line, which means the download was found and checked against its pinned hash (
bh_demo/launcher.py:158):Vendor definitions: found and hash-verified. Mode: local mock built from them. -
A line starting
Open the Inspector:, followed by a link. -
Later,
Rasa server is up and running.
The link prints before the server is ready. Wait for the last line before you open it.
The numbers in this tutorial come from a run with seed 295185. Your run may draw a different seed. To get the same starting data as the recorded run, stop the server with Ctrl+C and start it again with that seed set:
DEMO_SEED=295185 make run
The seed line now shows the number you set. Write this seed down in place of the first one. The recorded run printed:
uv run --locked python run_demo.py --port 5005
Demo data seed: 295185; set DEMO_SEED=295185 to get the same data again.
...
Rasa server is up and running.
A seed is the starting number of a random number generator. The example uses it to draw balances, loan figures, and a due date. The same seed always gives the same draw. Any seed from 0 to 999,999 is accepted, and anything else stops the start with a plain message (bh_demo/demo_data.py:76-84; bh_demo/launcher.py:78-80).
Open the Inspector: hold Ctrl (Command on a Mac) and click the link. If it does not open, use the Ports tab, find port 5005, and click its globe icon. Click Type your message to type instead of speaking.
Step 3. Ask what the release can answer
Use the server run whose seed you wrote down last. In the Inspector, type these seven questions one at a time, in this order, and wait for each reply. This is the order of both recordings:
- “What accounts do I have?”
- “What is the balance of DDA-A?”
- “What is the balance of SDA-A?”
- “When is my next loan payment on LOAN-A?”
- “What did I spend recently on DDA-A?”
- “What is my payoff on LOAN-A?”
- “What is the balance of DDA-B?”
This step explains the answers to questions 1, 2, and 4. Step 4 explains questions 3, 5, 6, and 7.
Order note. The recorded pattern of tool calls is for these seven questions, in this order, with no other turns between them. If you ask them in another order, add a question, or chat in between, you are running a different, unrecorded conversation. The recorded pattern does not cover that case, so the tool calls may differ. In that case compare facts (the balances, the date, and the three “not available” answers) and not the tool calls.
The account names use the release’s own codes: DDA, SDA, and LOAN. The release declares these codes but does not say which products they name, so the assistant never calls them checking or savings (README.md:143; skills/banking/skill.md:18-19). One fictional caller owns DDA-A, SDA-A, and LOAN-A (bh_demo/permissions.py:14-17).
Checkpoint:
- The accounts are
DDA-A,SDA-A, andLOAN-A. - The balance is an available balance, not a ledger balance.
- The loan answer gives two facts: the payment and its due date.
- The amounts and the date match your seed.
Here are the replies from the recorded run with seed 295185:
You: What accounts do I have?
Assistant: You have the following accounts:
- DDA-A
- LOAN-A
- SDA-A
You: What is the balance of DDA-A?
Assistant: The available balance of your DDA-A account is 702.20 USD.
You: When is my next loan payment on LOAN-A?
Assistant: Your next loan payment on LOAN-A is due on November 27, 2026, for an amount of 155.51 USD.
Where does each value come from? The README answers that for each one.
Fiserv’s release files describe the shape of each reply. For some fields they also list the allowed values. For others they leave the values to each bank.
The release labels balances by a field called BalType and declares no values for it. Fiserv’s official Premier sample collection (release R2026.3.3) labels the available balance Avail, in a sample that updates a DDA account.
So the example reads the single Avail entry for a DDA account and ignores every other balance (README.md:197).
A loan payment is similar. The due date is a plain field. The amount sits in a list of parts labeled by CompositeCurAmtType, which declares no values. The same sample collection labels the full payment Total, so the example reads the single Total entry and never a part or a sum of parts (README.md:198).
Both labels come from an older sample collection, not from this release. The README says so, and says that no live bank has confirmed either label (README.md:207, 217). Wherever you show a balance or a payment from this example, say that too.
The tests pin these rules, one per bullet:
- A balance is the
Availentry, never the ledger balance:tests/test_flow.py::WireTests::test_balance_is_the_avail_entry_never_ledger. - A missing or repeated
Availentry makes the balance unavailable instead of picking one:::test_a_missing_or_repeated_avail_entry_is_unavailable. - A payment is the
Totalentry, never a part or a sum:::test_loan_payment_is_the_total_entry_never_a_part_or_a_sum. - A missing or repeated
Totalentry makes the payment unavailable:::test_a_missing_or_repeated_total_entry_is_unavailable.
All four appear in the recorded test run.
You can run the first one on its own, from the example folder. It needs the release file in place:
uv run --locked --group models python -m unittest discover -s tests -v -k test_balance_is_the_avail_entry_never_ledger
A separate run of that command at the pinned commit printed these result lines: the test id and result, the summary, and the exit status. Log lines that unittest printed between them are left out, as the single-test runs file says:
test_balance_is_the_avail_entry_never_ledger (test_flow.WireTests.test_balance_is_the_avail_entry_never_ledger) ... ok
----------------------------------------------------------------------
Ran 1 test in 1.132s
OK
If a read says you need to verify
A read needs a simulated verification that is less than 300 seconds old (bh_demo/host.py:28, 210-213). If you pause for more than five minutes during this step or the next, a question can be refused with the reason assurance_required. That is a refresh, not an API gap and not real authentication. The check runs before the account is looked up (bh_demo/host.py:215-216), so it also comes before the reason for a history or payoff gap can be returned (bh_demo/host.py:300-310).
def _read_ok(self, context):
self._bound(context)
if self._age(context) >= READ_GRANT_SECONDS:
raise Refused("assurance_required")
The skill tells the model what to do: “If a refusal reason is assurance_required, tell the caller to say verify me and then ask again” (skills/banking/skill.md:42-43). The steps are in the box below.
No recorded run in this tutorial hit this case, so the steps above come from the code and from a test. The test moves a fake clock to 299 seconds, reads, moves it one second further, expects a refusal, refreshes the verification, and reads again (tests/test_flow.py:265-273). Run only this test:
uv run --locked --group models python -m unittest discover -s tests -v -k test_read_grant_expires_at_300_seconds
Checkpoint: the result lines are:
test_read_grant_expires_at_300_seconds (test_flow.HostTests.test_read_grant_expires_at_300_seconds) ... ok
test_read_grant_expires_at_300_seconds (test_host_ci.HostWithoutVendorFile.test_read_grant_expires_at_300_seconds) ... ok
----------------------------------------------------------------------
Ran 2 tests in 1.130s
OK
Those are the result lines of a separate run of this one test at the same commit, saved in the single-test runs, which leaves out the log lines printed between results. The -k filter selects every test whose name contains the text, so both copies of the test ran.
Step 4. Ask what it cannot answer
You asked these four in Step 3, as questions 3, 5, 6, and 7. If you have not asked all seven yet, go back to Step 3 and do that first, in order. Look at the answers to these four now:
- Question 3: “What is the balance of SDA-A?”
- Question 5: “What did I spend recently on DDA-A?”
- Question 6: “What is my payoff on LOAN-A?”
- Question 7: “What is the balance of DDA-B?”
Question 7 asks about an account that belongs to a different fictional caller.
Checkpoint: each answer says plainly that the information is not available, and none of them gives an amount. If the assistant ever states an amount for one of these, stop: that is a bug in the example or in your setup.
If it does, treat the answer as wrong and do this. Stop the run. Write down the seed and the exact question. Check the terminal for a mantle.tool.dispatched line for that turn. We did not test whether every setup logs every dispatch, so treat a missing line as a strong clue, not as proof. We recommend not releasing a build that behaves this way. Fixing it is work in your own code, for example checking the reply before it is spoken (see the end of Step 5). In the example’s code that we read, nothing does it for you.
The recorded run with seed 295185 replied:
You: What is the balance of SDA-A?
Assistant: The balance information for your SDA-A account is not available in this banking release.
You: What did I spend recently on DDA-A?
Assistant: Recent transaction history for your DDA-A account is not available in this banking release due to a schema defect.
You: What is my payoff on LOAN-A?
Assistant: The payoff information for your LOAN-A account is not available in this banking release.
You: What is the balance of DDA-B?
Assistant: That account is not available to you.
Look closely at the differences. The history answer gives a reason: a schema defect. The SDA-A balance and the payoff answers give none. The README gives a different reason for each gap (README.md:149-153, 199), and the skill tells the model to give one short reason for each closed reason code (skills/banking/skill.md:35-41). Yet only the history answer carries a reason. Step 5 compares these replies with the saved tool results.
The three gaps are different kinds of gap, and it helps to name them:
- The
SDAbalance: Fiserv publishes an example of an available balance only for aDDAaccount. Nothing sourced shows one forSDA. - The payoff: this release has no payoff code. A separate payoff service exists, but its definition has its own errors, and this example does not use it (
README.md:199). - The history: the release’s own transaction definition contradicts itself. Step 6 proves that.

Step 5. Compare replies with saved host results
The example has two places that can say “not available”. The host is the example’s own code, which the model reaches only through tools. For each gap it holds a closed reason, in a table called UNSOURCED (bh_demo/host.py:34-39). When the model asks for one of these facts, the host checks that the account is the caller’s and then returns the reason without calling the bank (bh_demo/host.py:291-315):
UNSOURCED = {
"history": "release_history_schema_defect",
"payoff": "payoff_code_not_in_release",
# Fiserv's "Avail" sample is a DDA record; nothing sourced shows it for SDA.
"sda_balance": "available_balance_sample_is_dda_only",
}
def balance(self, context, account):
with self._lock:
kinds = {name: kind for name, _, kind in context.accounts}
if kinds.get(account) == "SDA":
return self._unavailable(context, account, {"SDA"}, "sda_balance")
return self._read(context, account, {"DDA"}, lambda ident: {
"amount": format(self.client.available_balance(ident), "f"), "currency": "USD",
"balance_type": "available"})
def _unavailable(self, context, alias, kinds, fact):
"""Check the alias is the caller's, then say plainly the fact is missing."""
with self._lock:
self._account(context, alias, kinds)
return {"state": "unavailable", "reason": UNSOURCED[fact]}
The model is the second place. The skill asks it to call the matching tool even when it expects the answer to be unavailable, and for an SDA alias it says “answers unavailable; say so” (skills/banking/skill.md:12-13, 20-21):
Always call the matching tool, even when you expect it to be unavailable, and
say only what it returns.
...
Use @tool.banking_balance for the available balance of a DDA alias. For an SDA
alias it answers unavailable; say so.
In the code we read, nothing makes the model call banking_balance for an SDA alias; the skill text above is an instruction to the model, and the reply policy we read covers transfers. The saved records contain no matching call for those questions.
A call is a tool call: the model asks, the host answers. A mantle.tool.dispatched line records a dispatch and names the tool. In the short live-terminal run at seed 0, the terminal itself printed this line for the request “List my accounts.” (the id is shown as ...):
2026-11-16 17:20:42 INFO rasa.mantle.orchestration.orchestrator - [info ] mantle.tool.dispatched conversation_id=... iteration=0 tool_name=banking_accounts
The two typed runs at seeds 0 and 295185 saved their server output to a file, where each line is in a JSON form. For the first question about the loan payment, the saved output at seed 295185 logged:
2026-11-16 06:10:41 INFO rasa.mantle.orchestration.orchestrator - {"tool_name": "banking_loan_payment", "iteration": 0, "event": "mantle.tool.dispatched", "conversation_id": "...", "level": "info"}
The saved dispatch lines for both runs show the same pattern: no line for the SDA-A balance, the payoff, or DDA-B.
Now look at your terminal after each of the seven questions from steps 3 and 4. For the SDA-A balance, the payoff, and the other caller’s account, the saved output has no matching mantle.tool.dispatched line. Record the tool name and result you actually find, and leave unmatched turns marked as missing evidence.
The typed runs contain the following tool entries and results for the seven questions. This table describes the saved record, excluding the setup tools activate and listen:
| Question | Saved tool entry | Host result in the record |
|---|---|---|
| What accounts do I have? | banking_accounts | The host listed them; the model worded the list |
Balance of DDA-A | banking_balance | The host supplied the amount; the model worded it |
Balance of SDA-A | no entry | No matching host result |
Next loan payment on LOAN-A | banking_loan_payment | The host supplied it; the model worded it |
Recent transactions on DDA-A | banking_history | The host returned release_history_schema_defect |
Payoff on LOAN-A | no entry | No matching host result |
Balance of DDA-B | no entry | No matching host result |
The saved tool results provide a second view of the record. Both runs show exactly one banking_balance call, with the argument {"account": "DDA-A"}. Its result was {"state": "supplied", "amount": "702.20", ...} at seed 295185 and "1250.00" at seed 0. Both runs also show one banking_history call with the result {"state": "unavailable", "reason": "release_history_schema_defect"}. No saved call names SDA-A, LOAN-A for a payoff, or DDA-B.
The record does not explain those missing entries. The source shows two inputs to the model:
- The host adds a note to every model request. The note lists the caller’s aliases with their codes,
DDA-A (DDA), SDA-A (SDA), LOAN-A (LOAN), and never mentionsDDA-B(lib/conversation_policy.py:24-29). The testAliasContext::test_the_caller_aliases_are_in_every_request_oncechecks that the note carries the three aliases and notDDA-B. - The skill says “For an SDA alias it answers unavailable; say so” (
skills/banking/skill.md:20-21).
The two gap replies were right, but neither included the host’s reason: available_balance_sample_is_dda_only or payoff_code_not_in_release. The history reply did include a reason, matching a saved host result. That is the observable difference to record.
The example also has a separate offline component proof, run on November 17, 2026 at this commit. Four test suites (tests.test_contract, tests.test_flow, tests.test_reply_adapter, and tests.test_demo_data) ran 72 tests in 33.793 seconds with no skips, on the locked Rasa 3.21.0.dev5 wheel. It is a different run from the 14 contract tests and the 140 tests of make test described in this tutorial, and from the two typed conversations. It does not show a trained agent, a model, a voice call, a vendor, or a bank. The component proof receipt lists its decisive tests and these limits, and the execution record holds the exact command.
This matters because of what you would otherwise believe. If you read only the host’s code and its tests, you would say “every gap returns a fixed reason and never a nearby number”. That is true of the host. The two recorded gap replies gave no invented amount, but their saved records contain no matching host result. The host tests cannot settle whether those conversations used the refusal path. To test that, you need a complete per-turn record as well as the spoken or typed reply.
If you want a gap to hold no matter what the model decides, move the refusal out of the model’s hands. For example, let the code that sends the model’s reply check it, the way this example’s reply policy already checks the model’s text for host-only wording about a transfer review (lib/reply_policy.py:18-21, 205-211). Or accept the model’s judgment for the harmless gaps, and say so in your own documentation. Whichever policy you choose, test the complete conversation path, not just the host function.
Step 6. Prove the history gap is Fiserv’s, not yours
The history answer gave a reason: a schema defect. How do you know the defect is in Fiserv’s file and not in the example?
Read the README row for transactions (README.md:200). In Fiserv’s AcctTrnService 11.0.0 definition, the type BaseKeysType is declared as a string. The account key type and the transaction key type are built as “a string and an object”. Nothing can be both a string and an object, so no request that names an account can be valid, and no reply with a transaction in it can be valid either.
That is a claim about someone else’s file, so test it. In a second terminal, run:
make contract-test
(Makefile:9-10). This runs only the contract tests, which check messages against the release file you uploaded. In the recorded run, 14 tests passed in about 6.6 seconds with no skips, and the recorded test run lists all 14. Three of them pin the defect (tests/test_contract.py:107-136):
ReleaseDefect::test_base_keys_type_is_a_string_inside_object_key_typesreads the definition and asserts thatBaseKeysTypehas typestringand that both key types include it.ReleaseDefect::test_a_history_request_naming_an_account_cannot_validatebuilds the request a caller would need, naming account5550010001, and asserts that the validator rejects it.ReleaseDefect::test_a_history_reply_with_one_row_cannot_validatebuilds a reply with one transaction and asserts that the validator rejects it.
Here are the first two, as written in the example:
def test_base_keys_type_is_a_string_inside_object_key_types(self):
self.assertEqual(self.contract.schema("trn", "BaseKeysType").get("type"), "string")
for name in ("AcctKeysType", "AcctTrnKeysType"):
refs = [part.get("$ref") for part in self.contract.schema("trn", name)["allOf"]]
self.assertIn("#/components/schemas/BaseKeysType", refs)
def test_a_history_request_naming_an_account_cannot_validate(self):
with self.assertRaises(ContractError):
self.contract.validate_request("trn", {"AcctTrnSel": {
"AcctKeys": {"AcctId": "5550010001", "AcctType": "DDA"}, "TrnPostType": "ExcludeMemoPost"}})
You can run the second one on its own, with the release file in place:
uv run --locked --group models python -m unittest discover -s tests -v -k test_a_history_request_naming_an_account_cannot_validate
A separate run of that command at the pinned commit printed these result lines: the test id and result, the summary, and the exit status. Log lines that unittest printed between them are left out, as the single-test runs file says:
test_a_history_request_naming_an_account_cannot_validate (test_contract.ReleaseDefect.test_a_history_request_naming_an_account_cannot_validate) ... ok
----------------------------------------------------------------------
Ran 1 test in 0.634s
OK
Checkpoint: the run ends with OK and reports no skipped tests. The contract tests need the release file. Without it they are skipped, not failed, so a run with a missing file can look green (README.md:293). Check the file is in place before you trust the result.
A second test in the same file shows the opposite problem. A reply that contains nothing but a success status is valid against the schema (tests/test_contract.py:66-69):
def test_a_status_alone_is_schema_valid_so_validity_is_not_success(self):
for key in ("party", "acct"):
self.contract.validate_response(key, 200, {"Status": {
"StatusCode": "0", "StatusDesc": "Success", "Severity": "Info", "SvcProviderName": "x"}})
Valid is not the same as success. The README says the example counts a read as a success only with HTTP 200, status 0, and severity Info, and a transfer only with HTTP 201, status 0, Info, and Valid (README.md:203). The recorded run shows this test as ok, and so does a separate run of it at the pinned commit.
What you can claim after this step is narrow, and the README keeps to it. The example proves its conversation design and its code against the shapes Fiserv published in this release. Whether it works with a real Fiserv system was not tested here (README.md:24, 213).
Step 7. Repeat the run from its seed
Stop the server with Ctrl+C. Start it again with the seed of the run you used for the seven questions, the last seed you wrote down in step 2:
DEMO_SEED=<your seed> make run
Wait for Rasa server is up and running, open the Inspector again, and ask the seven questions in the recorded order (the order note in Step 3), the same order you used the first time.
Checkpoint:
- The seed line shows the seed you set.
- The amounts and the date are the same as the first time. The recorded run with seed 295185 gave
DDA-AUSD 702.20, andLOAN-AUSD 155.51 due on November 27, 2026. - The three “not available” answers match the recorded runs. We did not record a same-seed repeat of the model conversation, so if one differs, write it down.
- The wording of the replies may be different. The seed repeats the data, not the model’s sentences.
Now stop the server again with Ctrl+C and wait for the terminal prompt to come back. Then run the reference seed:
DEMO_SEED=0 make run
Wait for Rasa server is up and running and open the Inspector again. The first line now ends with (reference data, as in the README's checked sequence) (bh_demo/demo_data.py:87-91). Seed 0 does not draw anything. It returns a copy of the fixed reference data (bh_demo/demo_data.py:51-52), the data the README’s checked sequence uses: DDA-A USD 1,250.00, and LOAN-A USD 182.40 due on December 1, 2026 (README.md:145-153).
The recorded run with seed 0 replied (the full transcript has all seven answers):
You: What accounts do I have?
Assistant: You have the following accounts:
- D D A account A
- L O A N account A
- S D A account A
How can I assist you further?
You: What is the balance of DDA-A?
Assistant: The available balance of your D D A account A is 1,250.00 USD. What would you like to do next?
You: When is my next loan payment on LOAN-A?
Assistant: Your next loan payment on L O A N account A is due on December 1, 2026, for an amount of 182.40 USD. Is there anything else you would like to know?
Notice what changed besides the amounts. This run spelled the codes out (“D D A account A”) and added a follow-up question. The seed 295185 run wrote “DDA-A” and added nothing. The skill tells the model to “say each alias as it is, for example ‘D D A account A’” (skills/banking/skill.md:16-17), and the two runs followed that differently. Same skill, same model, same questions, different seed, and different wording. That is why this tutorial compares facts and not sentences.
Step 8. Deterministic tests: what they check and what they leave open
Now run the full suite, in the second terminal:
make test
(Makefile:6-7). The README describes the suite as offline: the manual live check, which does call a model, is not part of it (README.md:293, 303-306). That is why it cannot see a model skip a tool, and it is also why it is repeatable.
Checkpoint: the recorded run of make test passed 140 tests in about 56 seconds, with no skipped tests and the release file in place. The run ended:
Ran 140 tests in 56.185s
OK
Start with tests/test_demo_data.py, which tests the seed itself:
- Seed 0 is exactly the reference data:
DemoDataTests::test_reference_seed_is_the_reference_data. - The same seed always gives the same data:
::test_same_seed_same_data. Seeds 1 and 2 give different data:::test_different_seeds_differ. - Drawing data does not disturb the program’s global random state:
::test_does_not_touch_global_random_state. - For each of 200 seeds (1 to 200),
::test_invariants_hold_for_200_seeds(tests/test_demo_data.py:39-68) checks these, one at a time:- the account ids, parties, and types never change;
- the type of every field never changes;
- available balances, principal, and interest stay inside their ranges;
- the available balance never exceeds the ledger balance;
- a loan’s total equals principal plus interest.
- The first line is the seed line, in exactly the wording you saw:
::test_seed_line_wordingandWiringTests::test_launcher_prints_the_seed_first. - Every seeded reply for seeds 0 to 200 validates against the pinned release schemas:
SeededRepliesFitThePinnedRelease::test_every_seeded_reply_validates_against_the_pinned_schemas.
Each of these printed ok in the recorded test run. Two of them were also run on their own at the pinned commit, with the command shown in step 3 and the test name changed:
test_invariants_hold_for_200_seeds (test_demo_data.DemoDataTests.test_invariants_hold_for_200_seeds) ... ok
test_reference_seed_is_the_reference_data (test_demo_data.DemoDataTests.test_reference_seed_is_the_reference_data) ... ok
Then the test that guards the gaps. HostTests::test_unsourced_facts_are_unavailable_with_a_reason_and_no_bank_call (tests/test_flow.py:246-264) asks the host for an SDA balance, a history, and a payoff:
def test_unsourced_facts_are_unavailable_with_a_reason_and_no_bank_call(self):
context = self.build()
calls = getattr(self.fx.client, "calls", None)
if calls is not None:
calls.clear()
self.assertEqual(self.service.balance(context, "SDA-A"),
{"state": "unavailable", "reason": UNSOURCED["sda_balance"]})
self.assertEqual(self.service.history(context, "DDA-A"),
{"state": "unavailable", "reason": UNSOURCED["history"]})
self.assertEqual(self.service.payoff(context, "LOAN-A"),
{"state": "unavailable", "reason": UNSOURCED["payoff"]})
if calls is not None:
self.assertEqual(calls, [])
with self.assertRaises(Refused):
self.service.payoff(context, "LOAN-B") # the alias is still checked first
It asserts that each call returns the reason from the UNSOURCED table, that no call reached the bank when the test’s fake bank records calls, and that the alias is still checked first: a payoff for LOAN-B is refused. The test also tries four bad page values for a history request, which the host refuses. A separate run of this test at the pinned commit printed:
test_unsourced_facts_are_unavailable_with_a_reason_and_no_bank_call (test_flow.HostTests.test_unsourced_facts_are_unavailable_with_a_reason_and_no_bank_call) ... ok
test_unsourced_facts_are_unavailable_with_a_reason_and_no_bank_call (test_host_ci.HostWithoutVendorFile.test_unsourced_facts_are_unavailable_with_a_reason_and_no_bank_call) ... ok
----------------------------------------------------------------------
Ran 2 tests in 1.214s
OK
The command selects by name, so it also ran the copy of the test that runs without the release file.
The same file pins that the host reads only the caller’s permitted aliases and the release’s own codes (HostTests::test_reads_use_only_permitted_aliases_and_the_release_codes). In tests/test_wire.py, EndToEnd::test_a_caller_gets_sourced_facts_and_honest_unavailable_answers runs the host service against the mock bank and asserts that the three unavailable answers carry the host’s reasons and never reach the bank. Put side by side with step 5, this is the full picture. The host path is tested. The model’s decision to use the host path is not.
The README’s checked sequence also covers a transfer: verify, review, then “Confirm transfer.” (README.md:157-168). This tutorial does not walk through it. The release declares TransferType as an optional field with no published values. This example leaves it out, as Fiserv’s own Add Transfer samples do (README.md:201).
Exercise: two seeds, one table
The seeded tests in Step 8 check that the data and the reply shapes stay valid across seeds. They do not show that the assistant gives the same answers for every seed. Compare two seeds in a table to see what your runs do.
- Make a table with a column for your recorded seed and a column for a second seed. Stop the first server with Ctrl+C and wait for the terminal prompt before you start the second one with
DEMO_SEED=<another number> make run; then wait forRasa server is up and runningand open the Inspector again. - For each seed, ask the seven questions in the recorded order (the order note in Step 3). Fill in the answers.
- In each run, look at the terminal and write down which questions produced a
mantle.tool.dispatchedline, and which tool it named.
Expected result, for you to compare after you fill in your table
Expected result, in three parts.
- The same in both recorded runs: the account list,
DDA-A,LOAN-A, andSDA-A; and the three gaps, each a plain “not available”. The two runs used two seeds and two separate conversations. - Different in the two recorded runs: the
DDA-Abalance, theLOAN-Apayment, and its due date. The code can vary them, within the ranges below; the two runs do not show how often.- The balance is a draw, rounded to whole cents and clipped to a fixed range. The loan payment is the sum of two draws, principal and interest. Each is rounded to whole cents and clipped to its own range, and the interest range is also capped at the principal (
bh_demo/demo_data.py:28-31, 40-43, 64-72). - The due date is the reference date moved by a whole number of days, at most 10 either way (
bh_demo/demo_data.py:32, 69-70). So a nonzero seed has 21 possible dates. - Two seeds can give the same date. A value that lands on the edge of its range can match too. A match is not a bug. Record it.
- The balance is a draw, rounded to whole cents and clipped to a fixed range. The loan payment is the sum of two draws, principal and interest. Each is rounded to whole cents and clipped to its own range, and the interest range is also capped at the principal (
- Something to compare: the pattern of tool calls. The saved records contain matching tool entries for the same questions in both runs, and the table below shows the answers. Compare yours with them, and write down any difference. If one of your runs does call a tool for
SDA-Aor the payoff, the host was asked. Whether the spoken answer carries the host’s reason depends on the model, because the skill is an instruction and not a guarantee, so check it.
The recorded runs gave this table:
| Seed 0 | Seed 295185 | |
|---|---|---|
DDA-A available balance | USD 1,250.00 | USD 702.20 |
LOAN-A next payment | USD 182.40, due December 1, 2026 | USD 155.51, due November 27, 2026 |
SDA-A balance | not available (no saved tool entry) | not available (no saved tool entry) |
DDA-A history | not available, the host’s reason | not available, the host’s reason |
LOAN-A payoff | not available (no saved tool entry) | not available (no saved tool entry) |
DDA-B balance | not available to you (no saved tool entry) | not available to you (no saved tool entry) |
Two different seeds changed two amounts and a date. The gap answers and pattern of saved tool entries matched. The wording of the replies differed in other ways (see step 7). The seed 0 and seed 295185 transcripts contain the saved replies and tool entries.
Stop and clean up
Click the terminal running the server and press Ctrl+C (README.md:114-118).
Before you delete the Codespace, keep what you need:
- the table and notes you made in the two-seed exercise;
- any source change you made;
- the seed lines you wrote down;
- the release file, if you want to repeat a run later.
The README asks you to preserve source changes and unique receipts or private state before deleting, and to keep vendor inputs private (README.md:119-125). Keep the release file somewhere private. Do not put it in a public repository or in anything you share.
The README also asks you to delete the Codespace when you are done (README.md:120-125). One that you stop but do not delete keeps using your storage. Open your Codespaces, find this one, and delete it.
Recap
- Use a value only where the vendor publishes it, and say where it comes from. Here the balance label and the payment label come from an older Fiserv sample collection, not from this release, and the README says no live bank has confirmed either.
- Everywhere else, the host holds a fixed reason code and the answer is “not available”. Whether the caller hears the host’s reason depends on whether the model asks the host. The two saved runs contain a history result, but no matching tool entry for the
SDA-Abalance, payoff, or other caller’s account. Keep those turns marked as missing evidence. - A test against the vendor’s own file can prove a gap is theirs: here, the tests show that no request naming an account and no reply with a transaction passes the v11 definition. A reply that passes a schema is still not a success.
- A printed seed repeats the starting data, not the model’s wording. Different seeds can give different amounts, and the two recorded seeds did. The tests check that the seeded replies for seeds 0 to 200 still validate against the vendor’s definitions.
- Host tests check the host path. A complete per-turn record lets you compare that path with the reply the caller received.
To go further, open the other bank examples in the companion repository and compare their READMEs with this one. The Fiserv Premier example uses Fiserv’s Premier-specific definitions, which supply both history and a payoff.