We said Rasa never expands api_key: ${VAR}. Its client does

On 3 September our companion repository, RasaHQ/rasa-community-resources, told its readers in docs/MIGRATING.md that for model-group credentials in Rasa Pro “the ${VAR} form has never worked”: a line such as api_key: ${OPENAI_API_KEY} would hand the provider the placeholder text as its key. The same commit, 432d416, changed 67 credential lines across 46 files, 63 of them to api_key_env: NAME. Its evidence was a reproduction against Rasa Pro 3.20.0.dev6, recorded in the commit message:

read_yaml("api_key: ${MY_KEY}")                 -> {'api_key': '${MY_KEY}'}
_resolve_api_key_env({'api_key_env': 'MY_KEY'}) -> {'api_key': 'sk-REAL-…'}
_resolve_api_key_env({'api_key': '${MY_KEY}'})  -> {'api_key': '${MY_KEY}'}

The commit concluded “There is no rescue path”. Follow the same value the rest of the way to the call into litellm, on the released 3.20.0 wheel, with litellm’s acompletion replaced by a stub that records its arguments and raises before any network call, and this is what the stub records:

rasa-pro 3.20.0
stopped: ProviderClientAPIException
api_key handed to litellm.acompletion: sk-probe-real-value

The real key reached litellm. Replace the one Rasa function that expands it with the identity function, and the same stub records ${PROBE_KEY} instead. Run the quickstart live on 3.20.0 with api_key: ${ANTHROPIC_API_KEY}, and it passes its acceptance script against Anthropic. Meanwhile the spelling our readers were told to adopt is the one Rasa Pro 3.21.0.dev3, a prerelease, refuses. The companion’s release run against it on 28 September logged 22 lines like this one:

- [mantle.validation.config.invalid_model_group_credentials] Model group at index 0 in 'integrations.yml' has invalid credentials: Model group 'orchestrator' uses 'api_key_env', which is no longer supported. Replace it with 'api_key: ${ENV_VAR_NAME}'.

Anyone who followed the note moved off the form 3.21.0.dev3 asks for and onto the form it rejects. The error does name the replacement, so the fix is one edit per line; what it cannot tell a reader is that the note which sent them there was wrong. The mistake was ours, and it was one of method rather than of reading. Each line of that reproduction is true. It stopped two layers before the network, and a repository-wide rewrite followed it.

The position this guide takes: a secret that is still a placeholder after your config loader runs has been deferred, not dropped. The only proof of what a provider receives is the arguments captured at the call site. The cost is a probe per credential path, written against a private module attribute that can move between releases.

If you are testing a Rasa Pro project against 3.21.0.dev3, the last section lists which credential lines must change. The sections before it follow the value through four stops, so you can check your own.

The loader defers api_key on purpose

The YAML loader expands ${VAR} in _env_var_constructor in rasa/shared/utils/yaml.py. This is lines 140 to 168 of the 3.20.0 wheel (sha256 a68fa7ef…e9b5), unedited:

def _env_var_constructor(loader: BaseConstructor, node: ScalarNode) -> str:
    """Process environment variables found in the YAML."""
    value = loader.construct_scalar(node)
    expanded_vars = os.path.expandvars(value)
    not_expanded = [
        w for w in expanded_vars.split() if w.startswith("$") and w in value
    ]

    constructed = loader.constructed_objects
    key_node = next(reversed(constructed)) if constructed else None

    if not_expanded:
        if (
            isinstance(key_node, ScalarNode)
            and key_node.value in DEFERRED_RESOLUTION_KEYS
        ):
            # These fields (e.g. OAuth client_id, token_url, Langfuse keys) are
            # resolved at runtime, so an unset env var is not an error at load time.
            return value
        raise RasaException(
            f"Error when trying to expand the "
            f"environment variables in '{value}'. "
            f"Please make sure to also set these "
            f"environment variables: '{not_expanded}'."
        )

    if isinstance(key_node, ScalarNode) and key_node.value in SENSITIVE_DATA:
        return value
    return expanded_vars

Read it in order. The constructor runs on the scalars the loader tags as !env_var, which are unquoted ${VAR} values, and expands each one first. If a variable is unset, it raises, unless the key is on DEFERRED_RESOLUTION_KEYS. Only once every variable has a value does it reach the last branch. For a key on SENSITIVE_DATA it then returns value, the original text, and throws expanded_vars away.

SENSITIVE_DATA in rasa/shared/constants.py (lines 361 to 373) opens with API_KEY, and line 185 sets API_KEY = "api_key"; nine other constants follow, among them the AWS and Langfuse credentials. DEFERRED_RESOLUTION_KEYS, the keys allowed to stay unset at load, does not include API_KEY in 3.20.0 or 3.21.0.dev3. So api_key: ${OPENAI_API_KEY} comes back from read_yaml as the string ${OPENAI_API_KEY} when the variable is set. When it is unset, read_yaml raises RasaException and names the variable. model: ${VAR} comes back expanded.

Why throw away a value the loader already has? The wheel says so in its own comments. The fallback constructor’s docstring in the same file (lines 104 to 124) calls SENSITIVE_DATA and DEFERRED_RESOLUTION_KEYS “protections”, and warns that a parser which always expanded “would risk exposing secrets to any bare-parser consumer in the process”. A parsed config that holds ${OPENAI_API_KEY} can be logged, dumped or passed around without carrying the key.

The placeholder also lets Rasa check how a secret was written. In the same 3.20.0 wheel, validate_model_group_configuration_setup in rasa/engine/validation.py validates the model groups in endpoints.yml and calls a sensitive-keys check (lines 1425 to 1446). That check tests every sensitive value against the ${...} pattern at lines 1394 to 1396:

                if key in SENSITIVE_DATA:
                    if isinstance(value, str):
                        if not re.match(SECRET_DATA_FORMAT_PATTERN, value):

A value that fails raises “must be set as an environment variable”. re.match anchors only at the start, so the check refuses an api_key string that does not begin with ${. It does not require api_key, and api_key_env is not on the list it walks. What it does accept is a reference, which it can only see if the loader has left the reference in place. On 3.20.0, then, the one kind of api_key string this check lets through in an endpoints.yml model group is the spelling our note said had never worked.

Our commit named the reason correctly, “a secret-leak guard”, and then assumed that a guard at the loader means nothing downstream expands the value. A loader built this way hands out references and leaves the dereferencing to later code. The job is to find that code. Rasa’s redaction module states the trap from the other side (rasa/utils/secret_redaction.py, lines 21 to 26): keys on the list “survive in memory as the literal "${VAR}" placeholder”, while keys off it, such as token, are already expanded. The 3.21.0.dev3 wheel keeps the loader branch unchanged, at lines 166 and 167 of its yaml.py.

Follow the value to the call site

One probe, run on three wheels: 3.20.0.dev6 (the version the commit names), 3.20.0 and 3.21.0.dev3. It sets a fake key, loads a model group and prints the key at each stop. No provider is called. The notes give what the probe printed at each numbered line.

import os, sys
os.environ["PROBE_KEY"] = "sk-probe-real-value"
import rasa
from importlib.metadata import version
print("rasa-pro", version("rasa-pro"))
from rasa.shared.utils.yaml import read_yaml
cfg = read_yaml("""
model_groups:
  - id: orchestrator
    models:
      - provider: anthropic
        model: claude-haiku-4-5
        api_key: ${PROBE_KEY}
""")
group = cfg["model_groups"][0]
print("after read_yaml:", group["models"][0]["api_key"])
from rasa.mantle.llm import client as mc
from rasa.shared.utils.io import resolve_environment_variables
if hasattr(mc, "_resolve_api_key_env"):
    print("after _resolve_api_key_env:", mc._resolve_api_key_env(group)["models"][0]["api_key"])
from rasa.shared.utils.llm import llm_factory
c = llm_factory(dict(group), mc._DEFAULT_LLM_CONFIG)
print("client:", type(c).__name__)
print("stored api_key:", c._completion_fn_args.get("api_key"))
print("api_key passed to litellm at call time:", resolve_environment_variables(c._completion_fn_args).get("api_key"))

Stop 2 explains the commit’s “no rescue path”. In the 3.20.0 wheel, _resolve_api_key_env in rasa/mantle/llm/client.py looks for one key and nothing else (lines 142 to 147):

            env_var_name = value.get(_API_KEY_ENV_CONFIG_KEY)
            if isinstance(env_var_name, str):
                value.pop(_API_KEY_ENV_CONFIG_KEY)
                api_key = os.environ.get(env_var_name)
                if api_key:
                    value[API_KEY] = api_key

It substitutes a value when a mapping carries api_key_env, and leaves an api_key: ${VAR} string alone. We read “this resolver does not expand it” as “nothing expands it”.

Stop 4 is where the value becomes real. The client’s calls go through _BaseLiteLLMClient in rasa/shared/providers/llm/_base_litellm_client.py. In the 3.20.0 wheel it has three completion methods: completion (line 130), acompletion (165) and acompletion_stream (228). Each wraps the stored arguments in resolve_environment_variables, imported at line 28, before it calls litellm (lines 155, 190 and 235). Lines 153 to 159, from completion:

            formatted_messages = self._get_formatted_messages(messages)
            arguments = cast(
                Dict[str, Any], resolve_environment_variables(self._completion_fn_args)
            )
            response = completion(
                messages=formatted_messages, **{**arguments, **kwargs}
            )

The function itself, in rasa/shared/utils/io.py (lines 518 to 525), applies os.path.expandvars to every string in a string, list or dict:

    if isinstance(value, str):
        return os.path.expandvars(value)
    elif isinstance(value, list):
        return [resolve_environment_variables(item) for item in value]
    elif isinstance(value, dict):
        return {key: resolve_environment_variables(val) for key, val in value.items()}
    else:
        return value

So the value travels as ${PROBE_KEY} through the loader, the resolver and the constructed client, and becomes sk-probe-real-value in the last Rasa frame before litellm. Two credential spellings resolved in two places, one at bootstrap and one inside every completion call, is a design a reader could fairly call a trap. It caught us.

Put a spy on the last function before the network

Stop 4 in probe.py has a weakness a sceptic would name: it calls resolve_environment_variables itself, so it shows what the function returns, not that the client calls it. The second probe removes the shortcut. It installs a test spy, the test double that records how it was called: a replacement for acompletion in the base client’s module that stores its arguments and raises. It builds the client with LLMClient.from_bootstrap (the constructor that reads the integrations.yml model group) and lets the client make the call. The lines that do the work, excerpted; the full script is folded below:

import rasa.shared.providers.llm._base_litellm_client as base
seen = {}
async def fake_acompletion(**kw):
    seen.update(kw); raise RuntimeError("stopped before network")
base.acompletion = fake_acompletion
# ... read_yaml of the same model group as probe.py ...
c = LLMClient.from_bootstrap(B())
try:
    asyncio.run(c._client.acompletion("hi"))
except Exception as e:
    print("stopped:", type(e).__name__)
print("api_key handed to litellm.acompletion:", seen.get("api_key"))

import os, asyncio
os.environ["PROBE_KEY"] = "sk-probe-real-value"
from importlib.metadata import version
print("rasa-pro", version("rasa-pro"))
from rasa.shared.utils.yaml import read_yaml
import rasa.shared.providers.llm._base_litellm_client as base
from rasa.mantle.llm import client as mc
from rasa.mantle.llm.client import LLMClient
seen = {}
async def fake_acompletion(**kw):
    seen.update(kw); raise RuntimeError("stopped before network")
base.acompletion = fake_acompletion
cfg = read_yaml("""
model_groups:
  - id: orchestrator
    models:
      - provider: anthropic
        model: claude-haiku-4-5
        api_key: ${PROBE_KEY}
""")
group = cfg["model_groups"][0]
class B: llm = group
c = LLMClient.from_bootstrap(B())
try:
    asyncio.run(c._client.acompletion("hi"))
except Exception as e:
    print("stopped:", type(e).__name__)
print("api_key handed to litellm.acompletion:", seen.get("api_key"))

Run with uv run --no-project --python 3.12 --prerelease allow --with rasa-pro==3.20.0 python probe_call.py, then again with rasa-pro==3.21.0.dev3. The output, unedited:

rasa-pro 3.20.0
stopped: ProviderClientAPIException
api_key handed to litellm.acompletion: sk-probe-real-value

rasa-pro 3.21.0.dev3
stopped: ProviderClientAPIException
api_key handed to litellm.acompletion: sk-probe-real-value

The stub raised RuntimeError, and the probe caught ProviderClientAPIException, so the client’s own acompletion made the call and wrapped the stub’s error. On 3.20.0, from_bootstrap also runs _resolve_api_key_env first; on 3.21.0.dev3 it runs the credential validator, and the placeholder passes it.

That shows the real key arrives. It does not yet show why. The counter-test, probe_call_counter.py, is probe_call.py with one line added after the stub is installed:

base.resolve_environment_variables = lambda value: value  # counter-test: call-time expansion disabled

The base client imports the function by name, so replacing the module attribute replaces what its completion methods call. The command differs from the first probe’s in two ways: an env -u prefix that keeps any real key out of the environment, and a grep that drops two structlog lines about SSL certificates:

env -u ANTHROPIC_API_KEY -u OPENAI_API_KEY -u LITELLM_MODIFY_PARAMS uv run --no-project --python 3.12 --prerelease allow --with rasa-pro==3.20.0 python probe_call_counter.py | grep -v '\[debug'

Then again with rasa-pro==3.21.0.dev3. The output, unedited:

rasa-pro 3.20.0
stopped: ProviderClientAPIException
api_key handed to litellm.acompletion: ${PROBE_KEY}

rasa-pro 3.21.0.dev3
stopped: ProviderClientAPIException
api_key handed to litellm.acompletion: ${PROBE_KEY}

With that one call disabled, the placeholder reaches litellm on both versions. Nothing else on this path expands it.

The same method carries to any stack that interpolates environment variables into config. Find the last function your code calls before bytes leave the process. Put a spy there that records its arguments and raises. Drive the real code path into it and assert on what it recorded. Then disable the step you think is responsible and check that the assertion fails. The loader, the parsed config and the constructed client are all intermediate states, and each of them can legitimately hold a placeholder.

Here is everything the probes show, in one place:

The probes are the mechanism evidence; the live runs are the outcome. Runs recorded 29 September 2026. On the released 3.20.0 wheel, the quickstart with its orchestrator model group set to Anthropic, claude-haiku-4-5 and api_key: ${ANTHROPIC_API_KEY} trained and passed its acceptance script: four conversations, no ERROR lines in the server log. On 3.21.0.dev3 the quickstart as it ships, with the same model group, passed too; its log holds two failed turns in which a request with no non-system message was refused, and neither concerns the key.

A live pass shows that a provider accepted the key. It does not show which line produced it, and it would look the same if some resolver we had not found did the work. The counter-test is what ties the result to resolve_environment_variables. Neither kind of evidence is enough alone.

Rasa 3.21.0.dev3: api_key_env is no longer supported

Here is what the wrong premise cost. Of the lines commit 432d416 moved to api_key_env, 30 went into endpoints.yml model groups; one of those 30 is a commented-out line, and the 30 sit in 15 files. Four more lines it changed sat in speech blocks and were deleted, because by the commit’s account asr: and tts: entries take no credential key at all. Commit 432d416 also added a lint check that enforced api_key_env, in the repository’s gate and in the starter pack, and it put the “has never worked” note in docs/MIGRATING.md. This guide’s probes, live runs and table cover the integrations.yml group Mantle builds, not endpoints.yml model groups.

The commit was careful about the check. It records that the check was “demonstrated FAILING on a deliberately broken fixture” and that its unit tests were mutation-tested, and it states the rule it was following: “A check only ever seen green is the defect this change exists to fix.” The premise the check enforced got no such test. The commit predicts the symptom a literal ${VAR} key would cause, “a provider auth error at runtime”, and quotes no such error. For an unquoted ${VAR} on the path traced above, the placeholder text never reaches litellm: a set variable is expanded at the call, and an unset one stops read_yaml. The literal placeholder does reach the model call in one case, which the last section shows: a quoted placeholder whose variable is unset.

Rasa Pro 3.21.0.dev3 then turned the question from a belief into a validation error. The companion’s release workflow ran make ci KEEP_GOING=1 against it on 28 September and logged 22 lines like the one at the top of this guide: 21 for the model group at index 0 of integrations.yml, one at index 1, all for api_key_env on orchestrator. Here is where each spelling ends up on each version:

The 22 CI lines are a separate observation from the diagram’s InvalidConfigException. They carry the code mantle.validation.config.invalid_model_group_credentials and the prefix “Model group at index 0 in ‘integrations.yml’” (21 lines) or “at index 1” (one), which come from a caller we did not open. Their message text matches validate_model_group_credentials in rasa/shared/providers/model_group_validation.py of the 3.21.0.dev3 wheel (sha256 8b7a1977…3495), and the attribution rests on that match. The validator raises api_key_env_not_supported when a model entry contains api_key_env. Its docstring then requires every SENSITIVE_DATA value that is present to be exactly ${ENV_VAR_NAME}, and the code checks each string with re.fullmatch against SECRET_DATA_FORMAT_PATTERN, which is \${(\w+)} (line 421 of rasa/shared/constants.py in both wheels). A string that fails raises sensitive_key_string_value_must_be_set_as_env_var. In the same wheel, LLMClient.from_bootstrap no longer calls _resolve_api_key_env; it runs this validator and raises InvalidConfigException before llm_factory builds anything.

The companion’s fix, commit ededc1c, converts the api_key_env entries back to api_key: ${NAME} and flips both lint checks to reject api_key_env and any api_key that is not exactly ${NAME}. Its message still says “On 3.20 the rule was the reverse”. The probes and the live run contradict the half of that which matters: api_key: ${NAME} reached the litellm call as the real key on 3.20.0 as well, and a provider accepted it.

The companion’s main branch no longer says “has never worked”. Pull request #50 merged as 5206372, and commit 1e5101b in it rewrites the 3.20 history in docs/MIGRATING.md and says why the old advice existed: “This catalog once advised api_key_env because a reproduction stopped at the YAML loader, which returns sensitive values unexpanded and defers expansion to the call.” Two sentences in the same file at that commit still contradict the probes. Lines 240 and 241 say “The provider client expands that reference from the environment when it is built”; stop 3 shows the built client still holding the placeholder, and the expansion happens at each call. Line 330 says “On 3.20 it had to be api_key_env: OPENAI_API_KEY”.

Which credential lines to change for 3.21.0.dev3

The table covers the entries under models: in the integrations.yml model group that LLMClient.from_bootstrap builds, which is what the 3.21.0.dev3 validator checks. It does not cover endpoints.yml model groups, where 30 of the companion’s rewritten lines went. Each cell names the layer that accepts or refuses the line. The results come from an offline probe, offline-credential-spellings-probe.txt, that loads each spelling with read_yaml, builds it with from_bootstrap and calls it through the spy, plus the live runs for the unquoted ${NAME}.

env -u ANTHROPIC_API_KEY -u OPENAI_API_KEY -u LITELLM_MODIFY_PARAMS -u UNSET_PROBE uv run --no-project --python 3.12 --prerelease allow --with rasa-pro==3.20.0 python probe_spellings.py 2>/dev/null | grep -v '\[debug'

import os, asyncio
os.environ["PROBE_KEY"] = "sk-probe-real-value"
os.environ.pop("UNSET_PROBE", None)
from importlib.metadata import version
print("rasa-pro", version("rasa-pro"))
from rasa.shared.utils.yaml import read_yaml
import rasa.shared.providers.llm._base_litellm_client as base
from rasa.mantle.llm.client import LLMClient
seen = {}
async def fake_acompletion(**kw):
    seen.update(kw); raise RuntimeError("stopped before network")
base.acompletion = fake_acompletion
def probe(line):
    seen.clear()
    try:
        cfg = read_yaml("model_groups:\n  - id: orchestrator\n    models:\n      - provider: anthropic\n        model: claude-haiku-4-5\n        " + line + "\n")
    except Exception as e:
        return f"read_yaml raised {type(e).__name__}"
    group = cfg["model_groups"][0]
    class B: llm = group
    try:
        c = LLMClient.from_bootstrap(B())
    except Exception as e:
        return f"from_bootstrap raised {type(e).__name__}"
    try:
        asyncio.run(c._client.acompletion("hi"))
    except Exception:
        pass
    return f"spy on litellm.acompletion recorded api_key={seen.get('api_key')!r}"
for line in ["api_key: ${PROBE_KEY}", "api_key: ${PROBE_KEY:-fallback}", "api_key: $PROBE_KEY",
             "api_key: sk-literal-probe", "api_key_env: PROBE_KEY",
             "api_key: ${UNSET_PROBE}", 'api_key: "${PROBE_KEY}"', 'api_key: "${UNSET_PROBE}"']:
    print(f"{line!r:36} {probe(line)}")
base.resolve_environment_variables = lambda value: value  # counter-test: call-time expansion disabled
for line in ["api_key: $PROBE_KEY", 'api_key: "${PROBE_KEY}"']:
    print(f"{line!r:36} {probe(line)}  [call-time expansion disabled]")

rasa-pro 3.20.0
'api_key: ${PROBE_KEY}'              spy on litellm.acompletion recorded api_key='sk-probe-real-value'
'api_key: ${PROBE_KEY:-fallback}'    read_yaml raised RasaException
'api_key: $PROBE_KEY'                spy on litellm.acompletion recorded api_key='sk-probe-real-value'
'api_key: sk-literal-probe'          spy on litellm.acompletion recorded api_key='sk-literal-probe'
'api_key_env: PROBE_KEY'             spy on litellm.acompletion recorded api_key='sk-probe-real-value'
'api_key: ${UNSET_PROBE}'            read_yaml raised RasaException
'api_key: "${PROBE_KEY}"'            spy on litellm.acompletion recorded api_key='sk-probe-real-value'
'api_key: "${UNSET_PROBE}"'          spy on litellm.acompletion recorded api_key='${UNSET_PROBE}'
'api_key: $PROBE_KEY'                spy on litellm.acompletion recorded api_key='$PROBE_KEY'  [call-time expansion disabled]
'api_key: "${PROBE_KEY}"'            spy on litellm.acompletion recorded api_key='${PROBE_KEY}'  [call-time expansion disabled]

rasa-pro 3.21.0.dev3
'api_key: ${PROBE_KEY}'              spy on litellm.acompletion recorded api_key='sk-probe-real-value'
'api_key: ${PROBE_KEY:-fallback}'    read_yaml raised RasaException
'api_key: $PROBE_KEY'                from_bootstrap raised InvalidConfigException
'api_key: sk-literal-probe'          from_bootstrap raised InvalidConfigException
'api_key_env: PROBE_KEY'             from_bootstrap raised InvalidConfigException
'api_key: ${UNSET_PROBE}'            read_yaml raised RasaException
'api_key: "${PROBE_KEY}"'            spy on litellm.acompletion recorded api_key='sk-probe-real-value'
'api_key: "${UNSET_PROBE}"'          spy on litellm.acompletion recorded api_key='${UNSET_PROBE}'
'api_key: $PROBE_KEY'                from_bootstrap raised InvalidConfigException  [call-time expansion disabled]
'api_key: "${PROBE_KEY}"'            spy on litellm.acompletion recorded api_key='${PROBE_KEY}'  [call-time expansion disabled]

The validator’s docstring says nested mappings, for example Azure oauth, get the same rule, so check the other keys on SENSITIVE_DATA too, not only api_key.

If you upgrade, write api_key: ${NAME}, the spelling the evidence here supports on both versions; the quickstart agent already does. Before you change a whole repository, put a spy on the call and read what it recorded.