# Check old billing replies before trusting the latest one

Source: https://rasa.community/library/casebook/insurance-billing/
Author: Rod Rivera
Published: 2026-11-05T12:00:00Z

A customer asks whether their payment arrived and whether their insurance is
active. Your billing assistant has the latest policy reply, so it discards
older replies. An older reply arrives twice with different states. The state
you display is still the latest one, but you have lost evidence that the
source contradicted itself.

You will run two Python reducers that select the same latest policy state.
One checks for contradictions before filtering old replies; the other filters
first. Across all 5,040 orders of seven invented observations, filtering first
misses the old contradiction in 3,360 orders. The difference is the evidence
retained, not which version wins.

The stricter check has a cost. A policy conflict can also make a combined report
withhold an otherwise unconflicted payment reply. You will compare that choice
with reporting each source separately, while keeping the policy answer unknown.
These are finite teaching comparisons, not insurer error rates or a Rasa model
conversation. Neither script contacts a service or submits a payment.

## Run the comparison

You need Git and Python 3.11 or later. The example uses the standard library,
with no license, model key, or customer records. In a new directory, run:

```bash
git clone https://github.com/RasaHQ/rasa-community-resources.git
cd rasa-community-resources
git checkout 21fbc5cb60a87049dcc141636e78972df32fd28b
cd tutorials/rasa-ai-team-casebook
python3 -B billing_history.py
python3 -B -m unittest discover -s tests -p 'test_billing*.py' -v
```

The billing script's JavaScript Object Notation (JSON) output includes:

```text
"delivery_orders": 5040,
"latest_first_missed_old_conflict": 3360,
"retained_history_detected_old_conflict": 5040
```

The `unittest` command must report 20 tests and `OK`: 9 reconciliation tests
and 11 history tests. Zero tests is not success. Check that your working
directory ends in `rasa-ai-team-casebook` and that you kept the quoted filename
pattern. The recorded run on November 4, 2026 used Python 3.14.3. All 39 tests
in this pinned casebook project passed too.

If a file is missing, check `git rev-parse HEAD` against the checkout above.
Do not substitute a different branch and assume the output still qualifies it.
The [comparison source](https://github.com/RasaHQ/rasa-community-resources/blob/21fbc5cb60a87049dcc141636e78972df32fd28b/tutorials/rasa-ai-team-casebook/billing_history.py)
and [history tests](https://github.com/RasaHQ/rasa-community-resources/blob/21fbc5cb60a87049dcc141636e78972df32fd28b/tutorials/rasa-ai-team-casebook/tests/test_billing_history.py)
are the files you will inspect next.

## Make the revision contract explicit

The invented account has independent payment, invoice, and policy sources:

| Source  | Earlier observation  | Latest observation   | What it describes |
| ------- | -------------------- | -------------------- | ----------------- |
| Payment | Revision 40: pending | Revision 41: settled | This payment      |
| Invoice | Revision 90: overdue | Revision 91: paid    | This invoice      |
| Policy  | Revision 5: lapsed   | Revision 6: active   | This policy       |

Revision 91 is not newer than revision 6 across services. Each source assigns
its own numbers. The reducer selects a maximum within each source, keeping
payment, invoice, and policy observations separate.

For this comparison, a source also promises that a revision identifies
immutable state. Two different states under the same source and revision
break that contract. This is a stated exercise assumption. Before applying the
check to a real service, establish whether its revision identifies an immutable
record, a mutable projection, or something else. A sequence number alone does
not answer that question.

## Keep the contradiction that a fast path skips

The original six observations contain no contradictory revision. Add one policy
reply saying revision 5 was active. It conflicts with the existing revision 5
reply saying lapsed. Here is the smallest policy trace that exposes the difference:

| Arrival | Policy reply       | Latest-first filter  | Retained-history check             |
| ------- | ------------------ | -------------------- | ---------------------------------- |
| First   | Revision 6: active | Selects active at 6  | Selects active at 6                |
| Second  | Revision 5: lapsed | Discards it as older | Records the revision 5 observation |
| Third   | Revision 5: active | Discards it as older | Detects two states for revision 5  |

Both reducers still select revision 6: active. Checking only their selected
value would miss the difference. Check their conflict records as well.

In `LatestFirst.receive`, the extra fast path is:

```python
old = self.current.get(observation.source)
if old and observation.revision < old.revision:
    return "older"
return super().receive(observation)
```

The comparison preserves account, source, revision-type, and state validation
before this branch. It also retains detection of contradictory replies for the
current revision. The changed order is specific: obsolete replies return
before the history check can see them.

`RetainedHistory` also keeps both validated contradictory observations under
their source and revision. The output's `retained_conflict_evidence` field prints
that pair. A conflict flag alone would not supply the evidence requested by the
investigation step below. Duplicate delivery does not multiply those entries.

Run the focused test:

```bash
python3 -B -m unittest discover -s tests -p test_billing_history.py   -k test_same_latest_policy_hides_a_different_conflict_observation -v
```

Expect one passing test. It checks that the two selected observations are equal,
then checks that only retained-history detection records the policy conflict.
This is a counterexample to validating the reducer solely by its final state.

The exhaustive run repeats the comparison over all seven observations. The
latest-first reducer notices the contradiction when both revision 5 replies
arrive before revision 6. In the other 3,360 orders, the fast path hides it.
The history check notices it in all 5,040 orders. These counts describe this
fixture and contract, not the frequency of contradictions in a service.

:::checkpoint{id="billing-old-conflict" question="A revision 6 reply is already selected. Two revision 5 replies then disagree. What does keeping revision 6 establish?" options="The older replies can safely be ignored|Which version is selected, but not whether the source contradicted itself|That the policy is currently active in production" answer="1"}
Latest selection and evidence validation are different checks. Under the
exercise's immutable-revision contract, the older pair must still be examined.
:::

## Choose how much an investigation blocks

The original `Reconciliation.report` requires all three sources before reporting
public states. A policy contradiction makes every public state unknown. That
is conservative for a joined question, but costly for a separate payment question:

| Source  | Whole-report contract after the policy conflict | Separate-source report       |
| ------- | ----------------------------------------------- | ---------------------------- |
| Invoice | Unknown                                         | Paid observation retained    |
| Payment | Unknown                                         | Settled observation retained |
| Policy  | Unknown                                         | Unknown; investigate         |

The `separate_sources` function quarantines the conflicted source. It reports
the other observations without turning them into a policy decision. A settled
payment still cannot answer whether coverage is active.

Run this test:

```bash
python3 -B -m unittest discover -s tests -p test_billing_history.py   -k test_separate_reports_keep_unconflicted_evidence_but_not_policy -v
```

Expect one passing test. It checks both retained states and the withheld policy
state. The output label `observed` is deliberate: the example has no freshness
clock or source authentication. It describes received fixture evidence, not a
verified live answer for a customer.

Choose the reporting contract before connecting a language model. If a tool must
answer a combined payment-and-coverage question, both parts need their own
usable evidence. If a tool only reports a payment observation, a policy conflict
need not erase that observation. The interface must keep that distinction
visible instead of compressing every result into one success flag.

## Keep the recovery cost visible

Send a later policy revision 7 saying active. Retained-history detection still
records the contradiction at revision 5; a newer value does not explain why
that key disagreed with itself. The demonstration prints `investigate` and
unknown public states before and after revision 7 under the whole-report contract.

:::callout{type="pitfall" title="A stricter check can keep the answer unavailable"}
The example has no automatic conflict reset. Its history grows with the source
revisions it receives and disappears when the process ends. Retaining every
old key catches this trace; forgetting old keys needs an explicit service
retention and recovery contract, not a shorter cache chosen just to turn the
report green.
:::

:::solution{title="What to inspect when a later reply does not clear the conflict"}
Inspect the two observations under the same source and revision, the source's
revision guarantee, and the retained conflict record. Ask the service owner
which evidence can explain or supersede the contradiction. The example has no
administrative command that claims to perform that investigation. A later reply
alone is not such a record.
:::

## Carry the check into your adapter

The earlier [reconciliation exercise](https://github.com/RasaHQ/rasa-community-resources/blob/21fbc5cb60a87049dcc141636e78972df32fd28b/tutorials/rasa-ai-team-casebook/billing_reconciliation.py)
also compares all 720 orders of the original six observations. Its deliberately
faulty first-invoice inference disagrees with the final policy state in 360.
Use it to learn source separation; the old-conflict comparison above tests a
more plausible optimization that still selects the latest state correctly.

Before using either pattern in an assistant, establish the account and payment
references, authenticated source, revision meaning, observation age, and
recovery record. Final agreement after every message arrives does not make an
intermediate reply current or establish a consistent snapshot across services.
Retain the selected values and the contradictions that affected the answer.

:::cta{href="/library/tutorials/guarding-irreversible-actions/" label="Put the check inside a tool that can change something"}
This exercise only reports observations. If your next tool changes an account or
submits a payment, build its authorization and retry checks before exposing it.
:::