How to check whether an answer is supported and relevant

Some questions have no fact to assert. “Is this answer actually supported by our overdraft policy?” is not a tracker fact. For free text, the framework sends the answer to a second LLM — the judge — and compares a returned score to a threshold.

The judge holds two jobs in a scenario. It scores your criteria — the natural-language goals like “the agent asks exactly one clarifying question” — against the whole transcript, with a written rationale per criterion. And it powers the two generative assertions, which score a specific answer against a specific standard. Criteria assess the conversation against your written goals. The two generative assertions assess a particular answer. One checks support from the reference text. The other checks whether the answer addresses the question. They measure different properties.

(The machinery, for the record, is the same code the classic e2e tests used — the scenario engine imports rasa.e2e_test.assertions directly (rasa/builder/copilot/mcp_server/tools/assertion_engine.py:9), so every citation below reads from the engine you have installed.)

Groundedness: a ratio of statements

From eval/scenarios/faq_grounded_answer.yml:

assertions:
  - generative_response_is_grounded:
      threshold: 0.8
      ground_truth: >-
        A replacement debit card is issued free of charge once per
        calendar year. Additional replacements cost $12 each. Standard
        delivery takes five to seven business days.

Mechanism: the judge splits the answer into atomic statements and marks each one supported or unsupported against your ground truth (rasa/e2e_test/llm_judge_prompts/groundedness_prompt_template.jinja2:7-12). Then the score is plain arithmetic — no embeddings involved (rasa/e2e_test/utils/generative_assertions.py:156-166):

score = supported statements / total statements

This is the metric for “did the agent make something up?” — and the companion’s most valuable scenario turns it inside out: faq_unknown_topic_refused.yml asks about a policy the source does not contain, with a ground_truth that describes the refusal. An agent that fills the gap from general banking knowledge produces fluent, helpful, unsupported statements — and scores badly against a truth that says “I don’t have that.”

The denominator insight — the detail that changes how you read every threshold: the judge chooses how many statements the answer splits into, not you. A terse two-statement answer can only score 0, 0.5, or 1.0. Against that answer, threshold: 0.8 does not mean “80% good” — it means “both statements must be supported”. Read your thresholds as fractions of a small integer, not as percentages.

Relevance: cosine similarity of invented questions

assertions:
  - generative_response_is_relevant:
      threshold: 0.7

Completely different mechanism. The judge is shown only the answer — not the ground truth, not your policy — and invents three questions that answer would address (answer_relevance_prompt_template.jinja2:1-9; the count is num_variations = 3 at assertions.py:1595). Those invented questions and the user’s real question are embedded, and the score is their mean cosine similarity (generative_assertions.py:123-153).

Relevance never sees your reference text. A wrong answer can still score highly if it addresses the customer’s question.

Relevance detects evasion and topic drift — an agent that answers the question it wished you’d asked. It cannot detect fabrication. Which is why faq_grounded_answer.yml asserts both metrics on the same turn: groundedness catches the confident lie, relevance catches the accurate dodge. Neither is sufficient alone.

(Side effect of the mechanism: relevance needs an embedding model as well as a judge — default OpenAI text-embedding-3-small, generative_assertions.py:28-31. Groundedness never embeds anything.)

Pin your judge

The judge — and, separately, the simulator — is configured in eval/conftest.yml:

simulation:
  llm: # drives the user simulator
    provider: openai
    model: gpt-5.1
evaluation:
  llm: # the judge: criteria scores + quality metrics
    provider: openai
    model: gpt-5.1

Record both model choices. Changing the simulator changes which conversations are generated. Changing the judge changes how those conversations are scored. Results from different judges need separate labels, even when the scenarios are unchanged. Keep the model names in the reviewed configuration above.

The trap: a judge with nothing to score

This one trips everyone, and the error message doesn’t help. Generative assertions do not look at every bot message. With no utter_source: given, they consider only messages whose source metadata is one of:

EnterpriseSearchPolicy · ContextualResponseRephraser · IntentlessPolicy

(rasa/e2e_test/utils/generative_assertions.py:34-38.) A plain templated response is invisible to them. Responses become eligible when they are generated or passed through the rephraser — and the rephraser only touches responses that explicitly set rephrase: true (rasa/core/nlg/contextual_response_rephraser.py:122-124). Otherwise a response’s source is simply its action name — which you can target, by naming it in utter_source:.

So: if a generative assertion behaves as though it never ran, check the source of the message you think it is judging. Before the threshold, before the prompt, before blaming the judge.

Thresholds are not portable

A last habit before the honesty chapter: the thresholds in the companion are illustrative, not engine defaults, and any threshold you tune is tuned against three things at once — a specific judge model, a specific embedding model, and a specific set of ground-truth strings. Change any of the three and the old threshold is a number from a different experiment. (The engine’s own fallback, for the record, is DEFAULT_THRESHOLD = 0.5, assertions.py:70 — treat that as a placeholder, not a recommendation.)

Check your understanding

• Your FAQ agent starts answering overdraft questions with confident, fluent, wrong numbers. Which metric catches it, and why does the other one score it high?
• Why does the refusal scenario’s ground_truth describe what the agent should say it cannot do, rather than containing the right answer?
• Why does changing your embedding model invalidate relevance thresholds but leave groundedness scores untouched?

The last chapter explains what a passing result can and cannot establish.