Python CLI Toolcraft

Architecture

Testing Click Commands with CliRunner

Drive Click and Typer commands in-process with CliRunner - separate streams, isolated filesystems, exit codes, and readable failures in pytest.

Updated

CliRunner invokes a command the way a user would — a real argument list, real parsing, real exit code — without spawning a process. It is the single most useful testing tool in either framework, and most of the friction people hit with it comes from three or four behaviours that are easy to miss.

TL;DR

  • runner.invoke(app, ["sync", "--dry-run", "data"]) runs in-process; assert on result.exit_code first.
  • Construct with CliRunner(mix_stderr=False) so result.stdout and result.stderr are separate.
  • The runner never raises: an escaped exception becomes result.exception with exit code 1.
  • Use catch_exceptions=False when debugging to get the real traceback.
  • runner.isolated_filesystem() for commands that write relative paths; tmp_path otherwise.

Prerequisites

A Click or Typer application with at least one command, and pytest. Typer re-exports the runner, so the import differs but nothing else does:

from typer.testing import CliRunner    # Typer
from click.testing import CliRunner    # Click

Everything in this guide applies to both.

The result object

from typer.testing import CliRunner

from mytool.cli import app

runner = CliRunner(mix_stderr=False)

def test_sync_reports_uploads(tmp_path):
    (tmp_path / "a.txt").write_text("x")

    result = runner.invoke(app, ["sync", str(tmp_path)])

    assert result.exit_code == 0
    assert "1 uploaded" in result.stdout
    assert result.stderr == ""
What comes back from invoke() The fields of a runner Result object: the exit code, captured standard output and error, the exception that escaped, and the exception info for debugging. What comes back from invoke() Result returned by runner.invoke(...) exit_code assert on this first stdout / stderr separate when mix_stderr=False exception what escaped, if anything exc_info the traceback, for re-raising An escaped exception does not fail the test by itself exit_code is 1 when an exception escaped Re-raise result.exception to see the real traceback The runner never raises: an unexpected exception is data on the result, which is why tests must check the code.

The property that surprises people: the runner does not raise. If your command throws an unhandled KeyError, the test does not error out with a traceback — it gets a result whose exit_code is 1 and whose exception holds the KeyError. A test that only asserts on output will fail with a confusing message about a missing string, and the actual cause is one attribute away.

That is why the exit-code assertion goes first, and why this helper is worth three lines in conftest.py:

def invoke(runner, app, argv, **kwargs):
    result = runner.invoke(app, argv, **kwargs)
    if result.exception and not isinstance(result.exception, SystemExit):
        raise result.exception          # surface the real traceback
    return result

When you are actively debugging, catch_exceptions=False does the same thing for one call:

result = runner.invoke(app, ["sync", "data"], catch_exceptions=False)

Streams, and why they must be separate

By default the runner mixes stderr into stdout, which makes it impossible to assert that an error message went to the right place. Since that is a real property scripts depend on, turn it off:

runner = CliRunner(mix_stderr=False)

def test_missing_source_reports_on_stderr():
    result = runner.invoke(app, ["sync", "/does/not/exist"])

    assert result.exit_code == 66
    assert result.stdout == ""                      # nothing contaminated the results stream
    assert "does not exist" in result.stderr

The result.stdout == "" assertion is the valuable one. It catches the stray print() that someone added while debugging and never removed — the kind of change that silently breaks mytool export > data.json for every user.

Runner options worth knowing A table of CliRunner constructor and invoke options: separating streams, supplying input, setting environment variables and catching exceptions. Runner options worth knowing Option Does Use when mix_stderr=False splits stdout and stderr asserting a message went to the right stream input="y\n" feeds stdin the test is about a prompt env={...} sets variables for the run testing environment precedence catch_exceptions=False lets errors propagate debugging an unexpected failure catch_exceptions=False is the fastest way to turn "exit code was 1" into an actual traceback.

Isolated filesystems

Commands that take explicit paths are easiest to test with tmp_path. Commands that write relative to the working directory want the runner's own context manager:

def test_init_creates_a_config_in_the_current_directory():
    with runner.isolated_filesystem():
        result = runner.invoke(app, ["init"])

        assert result.exit_code == 0
        assert Path(".mytool.toml").is_file()
What isolated_filesystem() changes The isolated filesystem context manager creates a temporary directory, changes into it for the duration of the block, and removes it afterwards. What isolated_filesystem() changes Enter the block temp dir created cwd changes relative paths land here Command runs writes what it likes Exit directory removed create invoke cleanup Use it for commands that write relative paths; use tmp_path when the test needs the directory afterwards.

It creates a temporary directory, changes into it for the duration of the block, and removes it afterwards. Two caveats worth knowing: the directory is gone once the block exits, so assert inside it; and it changes the process working directory, so it does not compose with tests that run commands in parallel threads.

For most projects a fixture combining both is the comfortable middle ground:

@pytest.fixture
def cli(tmp_path, monkeypatch):
    monkeypatch.chdir(tmp_path)                     # relative paths land in tmp_path
    monkeypatch.delenv("MYTOOL_RETRIES", raising=False)
    return CliRunner(mix_stderr=False)

monkeypatch.chdir is undone automatically, the directory survives for post-test assertions, and tmp_path is printed in the failure output when something goes wrong.

Environment and input

Both are constructor or call arguments rather than something to patch:

def test_environment_variable_is_read(cli):
    result = cli.invoke(app, ["config", "show"], env={"MYTOOL_RETRIES": "9"})
    assert "retries 9" in result.stdout

def test_confirmation_can_be_answered(cli):
    result = cli.invoke(app, ["destroy", "prod"], input="y\n")
    assert result.exit_code == 0

Supplying env sets those variables for the invocation only. It does not clear the rest of the environment, which is why a fixture that deletes your tool's variables is still worth having — otherwise a developer with MYTOOL_REGION exported gets different results from CI.

Testing a group's callback

Global options live on the callback, and it runs on every invocation. The way to test what it produced is to look at what a command received:

def test_config_flag_populates_the_context(cli, tmp_path):
    (tmp_path / "custom.toml").write_text('region = "us-east-1"\n')
    seen = {}

    @app.command()
    def _probe(ctx: typer.Context) -> None:          # a test-only command
        seen["region"] = ctx.obj.region

    cli.invoke(app, ["--config", str(tmp_path / "custom.toml"), "_probe"])

    assert seen["region"] == "us-east-1"

Adding a probe command inside a test is slightly unusual and much clearer than reaching into Click's internals. The alternative — asserting on the output of a real command that happens to print the region — couples the test to that command's formatting.

UX considerations

Two behaviours are worth pinning because they are what users meet first.

--help must work with no configuration present. It is the most common first invocation, and a tool that needs a valid config file to print help is frustrating in exactly the moment someone is trying to learn it:

def test_help_works_without_config(cli):
    result = cli.invoke(app, ["--help"])
    assert result.exit_code == 0
    assert "sync" in result.stdout

Usage errors must exit 2. Both frameworks do this by default; the test exists to catch the day someone wraps the app in a try that swallows it:

@pytest.mark.parametrize("argv", [["sync"], ["sync", "data", "--retries", "x"]])
def test_usage_errors_exit_2(cli, argv):
    assert cli.invoke(app, argv).exit_code == 2

Testing the behaviour

Keep runner tests to the interface and push the rules down. A useful heuristic: if a test's name describes a domain rule ("skips files that are already current"), it should not need invoke. If it describes an interface fact ("--dry-run reaches the core", "a missing file exits 66"), the runner is the right tool.

def test_dry_run_flag_reaches_the_core(cli, monkeypatch):
    seen = {}
    monkeypatch.setattr(
        "mytool.commands.sync.core.sync_directory",
        lambda source, **kwargs: seen.update(kwargs) or SyncResult(0, 0),
    )

    assert cli.invoke(app, ["sync", "data", "--dry-run"]).exit_code == 0
    assert seen["dry_run"] is True

Patch the name as imported by the command module, not where it is defined. mytool.commands. sync.core.sync_directory works because the module holds a reference to core; patching mytool.core.sync.sync_directory after the command module has already bound the function is the classic reason a patch appears to do nothing.

Conclusion

CliRunner gives you the full parsing and dispatch path at roughly a millisecond per invocation. Assert on the exit code, split the streams, re-raise unexpected exceptions so failures are readable, and keep the tests focused on the interface — the behaviour belongs in cheaper tests one layer down.

Frequently asked questions

Why does my test pass when the command clearly failed?

Because the runner captured the exception instead of raising it, and your assertion happened to match text in the error output. Assert on result.exit_code first in every test, and add the re-raising helper so unexpected exceptions surface with their traceback.

How do I test a command that reads from stdin?

Pass input= to invoke. It works both for prompts and for commands that read sys.stdin.read() directly, since the runner replaces the stream for the duration of the call. For binary input, input= accepts bytes.

Can I invoke a subcommand object directly?

You can — runner.invoke(sync, ["data"]) invokes a single command without going through the group — but prefer invoking the root app with the full argument list. That is what users type, and it exercises the group callback, which is where global options and shared setup live.

Does the runner work with async commands?

Typer and Click commands are synchronous; if your command body runs an event loop with asyncio.run, the runner tests it like any other command. Testing the async functions themselves is better done directly with pytest-asyncio, leaving the runner to cover the command wrapper.

Why is result.output empty when I know the command printed something?

Most often because the output went to stderr and you constructed the runner with mix_stderr=False — check result.stderr. Failing that, the command wrote to a stream it opened itself rather than to sys.stdout, which the runner cannot capture; route all output through the framework's echo or a shared console object.

How do I test a command that calls sys.exit directly?

The runner catches the resulting SystemExit and reports its code as result.exit_code, so the test looks the same as any other. That said, a command calling sys.exit itself is usually a design smell — raising a domain exception and mapping it in one boundary keeps the codes reviewable in a single place and makes the command testable as a plain function.

Is it worth asserting on help text for every command?

No. Assert once that the root --help lists every command, which catches a registration that was removed or never added. Beyond that, pin help output with a snapshot only if you have committed to keeping it stable — otherwise every wording improvement becomes a test failure with no defect behind it.