Check for missing audio in a saved Rasa voice test

You are reviewing a voice assistant that asks before recording a disputed card charge. Your test checks that it selects the charge and records the dispute after the caller says yes. Before accepting that result, you also need to know whether the caller received the question.

In this guide, you will inspect one saved call from Northgate Bank, a fictional bank, then run an offline audit. You will finish with a review decision: which parts of the passing test you can accept, and which need more evidence.

The example uses synthetic callers and invented transactions. Its recorded calls used Rasa Mantle on 29 September 2026, with Rasa Pro 3.21.0.dev5, Claude Sonnet 5.5, Speechmatics and Rime. You will read those historical results, not place another call or test today’s versions of that stack.

Follow the question and the scripted answer

Open the saved results and find normal-date-said-numerically.

The assistant’s tracker contains this question:

I can record that you do not recognise the charge of 1,850 rupees from Lakeview Fuel on 19 September, on your debit card ending 7 3 1 9, for review. This is not a refund decision. Would you like to confirm this transaction?

That is recorded text. It is not a transcript of audio someone listened to.

Follow the call in order. Turn numbers below start at one.

The caller script does not listen to the assistant and decide how to respond. It sends the next line from the scenario. A filing after that yes can therefore pass without establishing a completed spoken exchange.

The matching server event is a summary extracted by the harness from its server log. The public report retains it; this guide does not inspect the unpublished raw log independently. Keep that distinction when recording your conclusion.

Compare the text with the client observations

In the same turn, inspect extra. These are fields from the saved record:

"eos_to_first_marker_ms": 11361.2,
"eos_to_first_audio_ms": null,
"eos_to_first_audible_ms": null,
"end_markers": [],
"bot_audio_s": 0.0,
"interrupts": 0,
"ended_by": "tracker",
"markers_pending": 2

The first marker arrived even though no audio was captured. A marker timestamp cannot stand in for a first-audio timestamp.

The client receiver handles audio and markers in different branches. It counts received PCM samples as audio. It records a marker separately, along with any latency figures supplied in that message.

The completion loop explains how the test continued. Its tracker branch can finish when the tracker says the bot turn ended, the emulated speaker queue is empty and the connection has settled. That branch does not require a first audio frame. An empty queue can mean there was nothing to play.

An empty queue can follow completed playback or zero received audio. The separate first-audio and sample fields distinguish those cases.

This is a gap in what the pass/fail decision checks. The harness already recorded the audio fields and event; they were available to a reviewer.

Those meanings come from the speaker queue and receiver, the returned observation fields, and the report’s marker and transcription processing. They describe this client’s bookkeeping, not a general server guarantee about what a person heard.

Run the saved-record audit

You need Git, Python 3.12 and a terminal. The audit uses Python’s standard library. You do not need to install Rasa, supply a licence or configure speech providers.

Clone the public example at the revision below so the audit and saved input stay together:

git clone https://github.com/RasaHQ/rasa-community-resources.git voice-test-review
cd voice-test-review
git checkout --detach 5f6fa63c5a987c42461bf77ce6cc61576f940fef
cd examples/mantle-voice-banking-dispute-claude

First, run the audit’s regression tests from that project directory:

python3 -m unittest discover -s tests -p 'test_confirmation_audio_audit.py' -v

The retained offline run ended with this summary. Its qualification receipt links the unchanged test log and audit output and identifies the exact executed source files. Those files match the canonical revision used above. The receipt retains the original run’s source identity; this link update is not a new test run.

Ran 20 tests in 0.011s

OK

These tests check the saved cases and the audit’s handling of missing, duplicated and conflicting evidence. Passing them means the audit behaves as tested. It does not mean every saved call delivered its question.

Now inspect the calls:

For this saved input, exit code 1 is expected: it means the audit found evidence needing review. An import error or missing file is a different failure. If the test file or saved result cannot be found, check your directory and revision before proceeding; do not change the expected labels to obtain a pass.

The command prints JSON. This is an unedited row from the retained output:

    {
      "case": "normal-date-said-numerically",
      "conversation_id": "banking-dispute-claude-voice-normal-date-said-numerically-20260929T230650",
      "confirmation_turn": 1,
      "finding": "missing_audio_corroborated",
      "original_checks_passed": true,
      "bot_audio_s": 0.0,
      "first_audio_ms": null,
      "missing_audio_utterance_ids": [
        "d3e7c782"
      ]
    },

The other corroborated finding is recovery-ambiguous-then-amount, turn 2. Across this input, the audit recognises ten confirmation rows: two have corroborated missing audio and eight record audio presence. Those counts describe this saved input, not a failure rate.

Read a row’s finding before deciding what to accept:

Also inspect unbound_events. The audit keeps review open for any missing-audio event it cannot bind to one confirmation, even an event for another reply. It does not silently discard that evidence because one turn contains audio.

Make three review decisions

For each row, assess two claims separately:

• A: the report records client audio captured in this turn.
• B: the full confirmation question reached the client before the scripted yes.

Choose accept, reject or hold for more evidence. Name the observation that would change your decision.

Compare your decisions and name the next evidence

Row 1: reject A and B for the retained captured test. Its zero audio and matching event support the missing-audio finding. More tracker text, marker timing or the scripted yes cannot repair that record. A separately captured complete exchange would be new evidence.

Row 2: reject A for this reported window and hold B. A missing event is not proof of successful playback. Check the capture window and seek correctly bound delivery evidence. Finding a matching missing-audio event would strengthen the negative finding, rather than turn it into a pass.

Row 3: accept A and hold B. To accept complete question delivery, you need a complete, ordered client capture spanning the question and response window, with trustworthy timing. Evidence that the bytes belonged to another turn would invalidate A. Even a complete capture would not establish human understanding.

The third row is a useful positive result. You can accept it without making the stronger claim. Row 2 is a teaching construction; it is not a new experiment or additional audit-test coverage.

Carry the distinction into your own evaluation

Keep backend completion and spoken delivery as separate findings. A successful filing can remain a successful filing while the delivery claim fails review. Otherwise a correct tool result can conceal the exact evidence your voice evaluation needs.

Keep the evidence sources separate too. Tracker text identifies the intended words. Client observations tell you about received audio in a capture window. An extracted server event can corroborate or conflict with that observation. The fixed next caller line records how the test continued. Establishing what a person heard requires separate listening evidence.

This audit recognises the example’s exact English confirmation sentence and reads only results.json. It binds an event using conversation identity, matching text and a unique question occurrence. Repeated identical questions cannot be assigned to one turn from text alone. The implemented guard for two different questions within one turn has no direct regression test yet. These limits matter when adapting the audit; do not use its labels unchanged for a different report format.

The latency guide asks when the answer starts after a filler. This exercise asks whether a passing call supports a delivery claim. The revision-bound confirmation guide checks whether the question still matches the terms. Playback requires its own evidence. The short-reply transcription guide concerns incoming caller speech, rather than the assistant’s outgoing audio.

Source and reproduction limits

The driver and harness explain the saved observations. The audit and its regression tests supply the offline check.

The evidence here consists of historical saved records and offline audit checks. The selected audit tests run without the SDK; the wider example proof has an SDK-dependent skip in the retained source check. Neither result qualifies the historical stack as today’s production behaviour.