Handling Config Files and Env Vars in CLIs

A real CLI reads settings from several places at once: a command-line flag, an environment variable, a project config file checked into the repo, a user config in your home directory, and hard-coded defaults. The hard part is not reading any one source — it is deciding which wins when two of them disagree. This page gives you a single, deterministic precedence chain and a runnable merge function that resolves it.

TL;DR

• Fix the precedence once and document it: CLI flags > env vars > project config > user config > defaults.
• Merge low-to-high into one plain dict, then validate the result with a single Pydantic v2 model so types and unknown keys are checked in one place.
• Put user config under the XDG base directory (~/.config/<app>/config.yaml), and look for a project config in the current tree.
• Coerce strings (env vars are always strings) by letting Pydantic do the work — "5432" becomes 5432, "true" becomes True.

The precedence chain

The single most important decision is the order. Higher-priority sources overwrite lower ones key by key. The rule of thumb: the closer a value is to the moment of invocation, the more it should win. A flag you typed this second beats an env var in your shell, which beats a file someone committed last month, which beats a file in your home directory, which beats the built-in default.

Where config files live

Don't invent paths. On Linux and macOS, follow the XDG Base Directory spec: user config lives in $XDG_CONFIG_HOME (default ~/.config). The project config is whatever file you find walking up from the working directory. A small resolver keeps this honest:

from __future__ import annotations
from pathlib import Path
import os

APP = "myapp"

def user_config_path() -> Path:
    base = os.environ.get("XDG_CONFIG_HOME") or str(Path.home() / ".config")
    return Path(base) / APP / "config.yaml"

def project_config_path(start: Path | None = None) -> Path:
    return (start or Path.cwd()) / f"{APP}.yaml"

A runnable merge function

Merge each source into one dict in priority order (lowest first so higher overwrites), then validate once. Keeping validation at the end means every source — file, env, or flag — is checked against the same schema and coerced to the same types. This snippet runs as-is:

from __future__ import annotations
from pathlib import Path
import yaml
from pydantic import BaseModel, ConfigDict, ValidationError


class AppConfig(BaseModel):
    model_config = ConfigDict(extra="forbid")
    host: str = "localhost"
    port: int = 8000
    timeout: int = 10
    verbose: bool = False


DEFAULTS = {"host": "localhost", "port": 8000, "timeout": 10, "verbose": False}
ENV_MAP = {
    "MYCLI_HOST": "host",
    "MYCLI_PORT": "port",
    "MYCLI_TIMEOUT": "timeout",
    "MYCLI_VERBOSE": "verbose",
}


def _read_yaml(path: Path) -> dict:
    if not path.is_file():
        return {}
    data = yaml.safe_load(path.read_text(encoding="utf-8"))
    if data is None:
        return {}
    if not isinstance(data, dict):
        raise ValueError(f"{path}: top-level YAML must be a mapping")
    return data


def _from_env(environ: dict) -> dict:
    return {key: environ[name] for name, key in ENV_MAP.items() if name in environ}


def merge_config(user_file: Path, project_file: Path,
                 environ: dict, cli_flags: dict) -> AppConfig:
    """Precedence (low -> high): defaults < user < project < env < CLI."""
    merged: dict = {}
    merged.update(DEFAULTS)
    merged.update(_read_yaml(user_file))
    merged.update(_read_yaml(project_file))
    merged.update(_from_env(environ))
    merged.update({k: v for k, v in cli_flags.items() if v is not None})
    try:
        return AppConfig.model_validate(merged)  # coerces "3333" -> 3333, "true" -> True
    except ValidationError as exc:
        raise SystemExit(f"Bad merged config: {exc}")

Run it with a user file (host, port, timeout), a project file (host, port), env vars (MYCLI_PORT=3333, MYCLI_VERBOSE=true), and a single --host flag-host flag, and you get:

Final config: {'host': 'flag-host', 'port': 3333, 'timeout': 99, 'verbose': True}

host came from the flag, port from the env (overriding both files), timeout from the user file (no higher source set it), and verbose from the env — exactly the precedence table above.

Why merge-then-validate

Two patterns compete here. You could validate each source separately and then merge typed objects, but that forces every source to be complete and duplicates the schema. The merge-then-validate approach treats every layer as a partial dict, lets dict.update express precedence with zero ceremony, and runs one schema check on the final result. That single check is where type coercion and unknown-key rejection happen — see Advanced argument validation strategies for the validation patterns this leans on.

The one subtlety is type coercion. Environment variables are always strings, so MYCLI_PORT=3333 arrives as "3333". Pydantic v2 coerces it to int during model_validate, and "true"/"false" to bool. Because coercion happens after the merge, you never have to parse types by hand per-source. Set extra="forbid" so a typo'd key fails loudly instead of being silently ignored.

We deliberately don't use pydantic-settings here. Building the merge by hand keeps the precedence explicit and testable, and avoids a dependency you may not want in a small CLI.

Production notes

• Deep merge for nested config. dict.update is shallow — a nested table in the project file replaces the whole nested table from the user file. If you need per-key merging inside nested mappings, recurse.
• Boolean env vars. Pydantic accepts true/false/1/0/yes/no for bools. Document which strings your users should set.
• Testing precedence. Make merge_config take environ and cli_flags as arguments (as above) rather than reading os.environ directly — that makes precedence trivial to unit-test with pytest.mark.parametrize.
• TOML too. The pattern is identical for TOML; swap yaml.safe_load for tomllib.load (stdlib since 3.11). The merge and validation layers don't change.

Reading the file, whatever the format

The format matters less than the layering, but the mechanics are worth pinning down because they differ in one important respect: TOML is in the standard library, YAML is not.

import tomllib                      # stdlib since Python 3.11
from pathlib import Path

def load_toml(path: Path) -> dict:
    with path.open("rb") as handle:  # tomllib needs binary mode
        return tomllib.load(handle)

import yaml                         # pip install pyyaml

def load_yaml(path: Path) -> dict:
    return yaml.safe_load(path.read_text(encoding="utf-8")) or {}

Two details that bite. tomllib.load requires a binary file object — passing a text handle raises a confusing error. And yaml.safe_load returns None for an empty document, which is why the or {} is there; without it the next line raises AttributeError on a file somebody left blank.

For a CLI, TOML is usually the better default: no dependency, one obvious way to write a value, and it is already the format of pyproject.toml, so users have seen it. YAML earns its place when your audience already writes it — Kubernetes, CI pipelines, Ansible — or when documents are long enough that anchors genuinely help. The YAML guide covers the safety rules in full.

Reading pyproject.toml deserves a mention because it is free real estate for a developer tool:

def project_config(root: Path) -> dict:
    pyproject = root / "pyproject.toml"
    if not pyproject.is_file():
        return {}
    with pyproject.open("rb") as handle:
        return tomllib.load(handle).get("tool", {}).get("mytool", {})

A [tool.mytool] table means users configure your tool in a file they already have, and it is version-controlled alongside the code it applies to.

Environment variables without surprises

Environment variables are the layer people get wrong most often, because they arrive as strings and are read implicitly.

import os

PREFIX = "MYTOOL_"

def from_env(environ: dict[str, str]) -> dict:
    """MYTOOL_RETRIES=5 -> {"retries": "5"}. Values stay strings; coercion happens later."""
    return {
        key.removeprefix(PREFIX).lower(): value
        for key, value in environ.items()
        if key.startswith(PREFIX)
    }

Three rules keep this predictable. Namespace everything with a prefix, so your tool cannot be influenced by an unrelated DEBUG or TIMEOUT somebody exported. Do not coerce here — return strings and let the single coercion pass handle types, or you end up with two conversion paths and two different error messages for the same mistake. And document them, ideally in --help, because an undocumented variable is indistinguishable from a bug when it changes behaviour.

Booleans deserve a deliberate rule, since MYTOOL_DEBUG=false is a string that Python considers true:

TRUTHY = {"1", "true", "yes", "on"}
FALSY = {"0", "false", "no", "off"}

def as_bool(raw: str, *, key: str) -> bool:
    lowered = raw.strip().lower()
    if lowered in TRUTHY:
        return True
    if lowered in FALSY:
        return False
    raise ConfigError(f"{key}: expected true or false, got {raw!r}")

Click can wire the prefix up for you with auto_envvar_prefix, which is worth using for flat options. Anything with nested structure still wants an explicit reader like the one above.

Secrets do not belong in the config file

Configuration and credentials are different problems that share a mechanism, and conflating them is how tokens end up in version control.

The order that works: read a secret from an environment variable, or from a file whose path is configured, or from the platform keyring — never from the config file that lives in the repository.

def resolve_token(settings: Settings) -> str:
    if settings.token_file:
        return settings.token_file.read_text(encoding="utf-8").strip()
    token = os.environ.get("MYTOOL_TOKEN")
    if not token:
        raise ConfigError(
            "no credential found: set MYTOOL_TOKEN or point token_file at a file"
        )
    return token

Two further habits. Never accept a secret as a command-line flag if you can avoid it — arguments are visible in ps output and land in shell history. And make sure your own diagnostics cannot leak one: a config show command should print token: **** rather than the value, and any structured logging should redact by key name rather than by hoping nobody logs the object.

From merged dictionary to typed settings

Merging produces a plain mapping of strings and values from four sources. One more step turns it into something the rest of the program can rely on.

from dataclasses import dataclass
from pathlib import Path

@dataclass(frozen=True, slots=True)
class Settings:
    retries: int = 3
    timeout: float = 30.0
    region: str = "eu-west-1"
    colour: bool = True
    token_file: Path | None = None

def build_settings(merged: dict[str, str | int | float | bool]) -> Settings:
    try:
        return Settings(
            retries=int(merged.get("retries", 3)),
            timeout=float(merged.get("timeout", 30.0)),
            region=str(merged.get("region", "eu-west-1")),
            colour=as_bool(str(merged.get("colour", "true")), key="colour"),
            token_file=Path(merged["token_file"]) if merged.get("token_file") else None,
        )
    except (TypeError, ValueError) as exc:
        raise ConfigError(f"invalid configuration: {exc}") from exc

Frozen, because a command that mutates settings changes behaviour for every command after it in the same process — which is exactly the bug that makes test order matter. Typed, because the alternative is settings["retries"] returning a string from the environment and an integer from a flag depending on how the tool was invoked.

If your configuration has nested structure, swap the dataclass for a Pydantic model and get the coercion, the nested validation and the error paths for free. The shape of the pipeline does not change: sources in, one mapping, one validation step, one immutable object out.

Store the result on the context in the group callback, and every command receives it without reaching for a global:

@app.callback()
def main(ctx: typer.Context, config: Path | None = None) -> None:
    ctx.obj = build_settings(resolve(cli={}, env=os.environ, file=load_file(config), defaults=DEFAULTS))

Making precedence testable

The reason to write the resolver as a function that takes its inputs is that it makes the whole system testable without monkeypatching, temporary directories or subprocesses.

import pytest

DEFAULTS = {"retries": 3, "region": "eu-west-1"}

def test_flag_beats_environment_and_file():
    merged = resolve(
        cli={"retries": 5},
        env={"retries": "9"},
        file={"retries": 7},
        defaults=DEFAULTS,
    )
    assert merged["retries"] == 5

def test_unset_flag_does_not_clobber_the_file():
    merged = resolve(cli={"retries": None}, env={}, file={"retries": 7}, defaults=DEFAULTS)
    assert merged["retries"] == 7

@pytest.mark.parametrize("raw,expected", [("true", True), ("0", False), ("On", True)])
def test_boolean_coercion(raw, expected):
    assert as_bool(raw, key="colour") is expected

Three tests, no filesystem, no environment mutation, and they cover the rules people actually get wrong. Contrast that with testing precedence through the CLI: every case needs monkeypatch.setenv, a temporary config file and a runner invocation, which is slow enough that the cases stop being written.

The one thing worth testing through the CLI is that the wiring is correct — that --retries ends up in cli, that MYTOOL_RETRIES is read, that --config is honoured. One invocation-level test per source, and the rest at the function level.

Deprecating and renaming settings

Configuration keys are as much a public interface as flags are, and they need the same care when they change.

The pattern that avoids breaking anyone: accept both names for a release or two, apply the old one only when the new one is absent, and say something once.

RENAMED = {"retry_count": "retries", "aws_region": "region"}

def apply_renames(file_config: dict) -> dict:
    out = dict(file_config)
    for old, new in RENAMED.items():
        if old in out:
            if new not in out:
                out[new] = out[old]
                warn(f"config key {old!r} is deprecated; rename it to {new!r}")
            del out[old]
    return out

Three properties matter. The warning goes to stderr, so it cannot contaminate results. It fires once per run, not once per lookup. And the new key wins when both are present, so a half-migrated file behaves predictably rather than depending on dictionary order.

Record the removal version in the message and the changelog — "will be removed in 2.0" gives people a deadline they can act on, where a bare "deprecated" gets ignored until it breaks.

The same reasoning applies to changing a value's meaning, which is more dangerous than renaming a key because nothing looks wrong. If timeout used to mean milliseconds and now means seconds, do not reuse the key: add timeout_seconds, deprecate the old one loudly, and let the two coexist until the major release that removes it.

Frequently asked questions

Related

• Up: Advanced Input Parsing for Python CLIs
• Down: Loading YAML configs safely in CLI apps
• Sideways: Advanced argument validation strategies