Check old billing replies before trusting the latest one

A customer asks whether their payment arrived and whether their insurance is active. Your billing assistant has the latest policy reply, so it discards older replies. An older reply arrives twice with different states. The state you display is still the latest one, but you have lost evidence that the source contradicted itself.

You will run two Python reducers that select the same latest policy state. One checks for contradictions before filtering old replies; the other filters first. Across all 5,040 orders of seven invented observations, filtering first misses the old contradiction in 3,360 orders. The difference is the evidence retained, not which version wins.

The stricter check has a cost. A policy conflict can also make a combined report withhold an otherwise unconflicted payment reply. You will compare that choice with reporting each source separately, while keeping the policy answer unknown. These are finite teaching comparisons, not insurer error rates or a Rasa model conversation. Neither script contacts a service or submits a payment.

Run the comparison

You need Git and Python 3.11 or later. The example uses the standard library, with no license, model key, or customer records. In a new directory, run:

git clone https://github.com/RasaHQ/rasa-community-resources.git
cd rasa-community-resources
git checkout 21fbc5cb60a87049dcc141636e78972df32fd28b
cd tutorials/rasa-ai-team-casebook
python3 -B billing_history.py
python3 -B -m unittest discover -s tests -p 'test_billing*.py' -v

The billing script’s JavaScript Object Notation (JSON) output includes:

"delivery_orders": 5040,
"latest_first_missed_old_conflict": 3360,
"retained_history_detected_old_conflict": 5040

The unittest command must report 20 tests and OK: 9 reconciliation tests and 11 history tests. Zero tests is not success. Check that your working directory ends in rasa-ai-team-casebook and that you kept the quoted filename pattern. The recorded run on November 4, 2026 used Python 3.14.3. All 39 tests in this pinned casebook project passed too.

If a file is missing, check git rev-parse HEAD against the checkout above. Do not substitute a different branch and assume the output still qualifies it. The comparison source and history tests are the files you will inspect next.

Make the revision contract explicit

The invented account has independent payment, invoice, and policy sources:

Revision 91 is not newer than revision 6 across services. Each source assigns its own numbers. The reducer selects a maximum within each source, keeping payment, invoice, and policy observations separate.

For this comparison, a source also promises that a revision identifies immutable state. Two different states under the same source and revision break that contract. This is a stated exercise assumption. Before applying the check to a real service, establish whether its revision identifies an immutable record, a mutable projection, or something else. A sequence number alone does not answer that question.

Keep the contradiction that a fast path skips

The original six observations contain no contradictory revision. Add one policy reply saying revision 5 was active. It conflicts with the existing revision 5 reply saying lapsed. Here is the smallest policy trace that exposes the difference:

Both reducers still select revision 6: active. Checking only their selected value would miss the difference. Check their conflict records as well.

In LatestFirst.receive, the extra fast path is:

old = self.current.get(observation.source)
if old and observation.revision < old.revision:
    return "older"
return super().receive(observation)

The comparison preserves account, source, revision-type, and state validation before this branch. It also retains detection of contradictory replies for the current revision. The changed order is specific: obsolete replies return before the history check can see them.

RetainedHistory also keeps both validated contradictory observations under their source and revision. The output’s retained_conflict_evidence field prints that pair. A conflict flag alone would not supply the evidence requested by the investigation step below. Duplicate delivery does not multiply those entries.

Run the focused test:

python3 -B -m unittest discover -s tests -p test_billing_history.py   -k test_same_latest_policy_hides_a_different_conflict_observation -v

Expect one passing test. It checks that the two selected observations are equal, then checks that only retained-history detection records the policy conflict. This is a counterexample to validating the reducer solely by its final state.

The exhaustive run repeats the comparison over all seven observations. The latest-first reducer notices the contradiction when both revision 5 replies arrive before revision 6. In the other 3,360 orders, the fast path hides it. The history check notices it in all 5,040 orders. These counts describe this fixture and contract, not the frequency of contradictions in a service.

Choose how much an investigation blocks

The original Reconciliation.report requires all three sources before reporting public states. A policy contradiction makes every public state unknown. That is conservative for a joined question, but costly for a separate payment question:

The separate_sources function quarantines the conflicted source. It reports the other observations without turning them into a policy decision. A settled payment still cannot answer whether coverage is active.

Run this test:

python3 -B -m unittest discover -s tests -p test_billing_history.py   -k test_separate_reports_keep_unconflicted_evidence_but_not_policy -v

Expect one passing test. It checks both retained states and the withheld policy state. The output label observed is deliberate: the example has no freshness clock or source authentication. It describes received fixture evidence, not a verified live answer for a customer.

Choose the reporting contract before connecting a language model. If a tool must answer a combined payment-and-coverage question, both parts need their own usable evidence. If a tool only reports a payment observation, a policy conflict need not erase that observation. The interface must keep that distinction visible instead of compressing every result into one success flag.

Keep the recovery cost visible

Send a later policy revision 7 saying active. Retained-history detection still records the contradiction at revision 5; a newer value does not explain why that key disagreed with itself. The demonstration prints investigate and unknown public states before and after revision 7 under the whole-report contract.

What to inspect when a later reply does not clear the conflict

Inspect the two observations under the same source and revision, the source’s revision guarantee, and the retained conflict record. Ask the service owner which evidence can explain or supersede the contradiction. The example has no administrative command that claims to perform that investigation. A later reply alone is not such a record.

Carry the check into your adapter

The earlier reconciliation exercise also compares all 720 orders of the original six observations. Its deliberately faulty first-invoice inference disagrees with the final policy state in 360. Use it to learn source separation; the old-conflict comparison above tests a more plausible optimization that still selects the latest state correctly.

Before using either pattern in an assistant, establish the account and payment references, authenticated source, revision meaning, observation age, and recovery record. Final agreement after every message arrives does not make an intermediate reply current or establish a consistent snapshot across services. Retain the selected values and the contradictions that affected the answer.