Python CLI Toolcraft

Architecture

Testing Python CLI Applications

Test Python command-line tools with CliRunner, pytest fixtures, snapshots, and isolation - fast suites that cover flags, exit codes, and output.

Updated

A command-line tool is unusually easy to test badly. Everything it does is observable from outside — an exit code, some text on a stream, a file that appeared — which tempts people into testing it entirely through subprocesses. That works, and it produces a suite so slow that nobody runs it.

This section is about the other approach: a small number of invocation-level tests that pin the interface, sitting on top of ordinary unit tests that prove the behaviour.

TL;DR

  • Invoke commands with CliRunner in-process, never through a subprocess.
  • Assert on exit_code first, output second.
  • Keep the interesting assertions in unit tests of plain functions; runner tests should prove that flags map to arguments and that failures produce the documented code.
  • Use tmp_path for real files, a mock transport for HTTP, and injected values for time.
  • Add exactly one test that installs the built wheel and runs the command.
What this section covers The testing section covers the command runner, snapshot testing of output, isolating the filesystem and network, interactive prompts, and coverage. What this section covers Testing a CLI fast tests you will actually keep The runner invoke commands in-process Snapshots pin output that must not drift Isolation files, network, time Coverage and what it misses every guide here assumes the logic already lives outside the command functions The layering from the architecture track is what makes all of this cheap.

Where the tests should sit

The layering from structuring multi-command CLIs is what makes a fast suite possible. When the logic lives in plain functions, the tests that cover it need no parsing, no runner and no filesystem — and the tests that do need a runner become few and cheap.

What to test, and how much of it A testing pyramid for a command line tool: many unit tests of the logic layer, fewer runner tests of the commands, and a single installed smoke test. What to test, and how much of it slowest, fewest Installed smoke test 1 per project Install the built wheel into an empty environment and run the command once. Runner tests ~1 per command Flags map to arguments, defaults match the help, failures produce the documented exit code. Unit tests of core/ as many as the logic needs Plain functions, no parsing, milliseconds each. This is where behaviour is proven. fastest, most numerous If an assertion needs a runner to reach the logic, the logic is in the wrong layer.

The shape is not an aesthetic preference; it falls out of the cost of each kind of test.

Where a slow CLI test suite goes A bar chart comparing the cost of a unit test, a runner test and a subprocess test of the same behaviour. Where a slow CLI test suite goes unit test of core 2 ms CliRunner invocation 18 ms subprocess call 240 ms Indicative per-test cost for a small CLI; the subprocess figure includes interpreter start-up. Two hundred subprocess tests is a minute of waiting; the same coverage as unit tests is under a second.

Two hundred subprocess tests is a minute of waiting before you learn anything. The same coverage through unit tests is under a second, and the difference decides whether people run the suite before pushing or after CI complains.

The runner, in one page

Both Click and Typer ship the same runner; Typer simply re-exports it. It replaces argv, captures the streams and calls your application in the current process.

from typer.testing import CliRunner      # or: from click.testing import CliRunner

from mytool.cli import app

runner = CliRunner()

def test_sync_reports_what_it_uploaded(tmp_path):
    (tmp_path / "a.txt").write_text("x")

    result = runner.invoke(app, ["sync", str(tmp_path)])

    assert result.exit_code == 0
    assert "1 uploaded" in result.stdout
What CliRunner actually does The runner replaces argv, captures the output streams, invokes the application in process and returns a result carrying the exit code and captured text. What CliRunner actually does invoke(app, argv) a real argument list Streams captured stdout, stderr, stdin App runs in-process no subprocess Result exit_code, output, exception as a user would type redirected returns Because it runs in the same process, a failure gives you a real traceback instead of a captured stderr blob.

Three habits make runner tests useful rather than incidental.

Assert on the exit code first. A test that only checks output passes cheerfully when the command failed for an unrelated reason — a missing fixture, a changed default — because the string happened to appear in an error message.

Separate the streams. Construct the runner with CliRunner(mix_stderr=False) and assert against result.stdout and result.stderr explicitly. Whether a message went to the right stream is a real property that scripts depend on, and mixing them hides regressions.

Pass real argument lists. ["sync", "--dry-run", str(path)] is what a user types. Building the list programmatically from your own constants means a renamed flag updates the test automatically, which is precisely the failure you wanted the test to catch.

A short table of invocations is worth writing once per project:

The invocations worth pinning A table of command line invocations every CLI test suite should cover, with the exit code each should produce. The invocations worth pinning Invocation Expect Proves no arguments 0 or 2, deliberately the empty case is designed --help 0 help works without config --version 0 metadata is readable a missing required argument 2 usage errors are usage errors a valid run 0 the happy path a known failure the documented code the boundary maps it Six tests, written once, that catch most of what breaks when a command tree is refactored. 
import pytest

@pytest.mark.parametrize("argv,expected", [
    ([], 2),                                 # no subcommand: deliberate, documented
    (["--help"], 0),
    (["--version"], 0),
    (["sync"], 2),                           # missing required argument
    (["sync", "./missing"], 66),             # documented "no input" code
])
def test_documented_exit_codes(argv, expected, tmp_path, monkeypatch):
    monkeypatch.chdir(tmp_path)
    assert runner.invoke(app, argv).exit_code == expected

Isolating the world

A test is only fast and repeatable if it does not touch anything shared. Three sources of non-determinism cover almost every flaky CLI test.

Choosing a test double A comparison of temporary directories, fake objects, monkeypatching and recorded HTTP responses for isolating a command line test. Choosing a test double For Reach for Because Filesystem tmp_path real files, thrown away after HTTP a transport mock no network, real client code Time and randomness inject the value deterministic without patching A slow collaborator a small fake class readable failures, no mock DSL Environment monkeypatch.setenv undone automatically Prefer a real temporary file over a mocked open(): the code under test then behaves the way it will in production.

The filesystem is the easy one: tmp_path gives each test its own directory with absolute paths, and pytest removes it afterwards.

A temporary project a test can trust A pytest temporary directory containing a config file, a source directory with sample files and an output directory, all discarded after the test. A temporary project a test can trust tmp_path/ (unique per test) removed automatically .mytool.toml the config under test data/ three small input files out/ where the command writes No test can see another test's files Absolute paths, so the working directory does not matter Real filesystem behaviour, including permissions A real temporary directory beats a mocked open(): the code behaves the way it will in production.

Prefer a real temporary file over a mocked open(). The code under test then behaves the way it will in production, including the encoding, the permissions and the error you get when the file is missing.

The network should be intercepted at the transport, not at your own functions:

import httpx

def test_deploy_sends_the_right_payload():
    seen = {}

    def handler(request: httpx.Request) -> httpx.Response:
        seen["url"] = str(request.url)
        seen["json"] = json.loads(request.content)
        return httpx.Response(200, json={"id": "dep_123"})

    client = httpx.Client(transport=httpx.MockTransport(handler))
    result = core.deploy(client, environment="prod", replicas=3)

    assert seen["url"].endswith("/deployments")
    assert seen["json"]["replicas"] == 3
    assert result.id == "dep_123"
Keeping the network out of tests A test replaces the HTTP transport rather than the client, so request building and response parsing are still exercised while nothing leaves the machine. Keeping the network out of tests Your code builds a real request httpx client unchanged in the test Mock transport returns a canned response Your parsing runs for real calls intercepted response Patching at the transport keeps your request-building and parsing code under test; patching your own function does not.

Patching your own deploy_to_remote function would make the test pass while proving nothing about the request you actually build. Patching the transport keeps the request construction and the response parsing under test, which is where the bugs are.

Time and randomness are best injected rather than patched. A function that takes now: datetime is trivially testable; one that calls datetime.now() internally needs monkeypatching in every test that touches it.

Fixtures that compose

Most of the setup in a CLI suite is shared, and a handful of small fixtures keep individual tests to three lines.

Fixtures that compose A layered set of pytest fixtures: a temporary project directory, a fake remote client, a settings object and a runner that uses all three. Fixtures that compose conftest.py shared by every test module tmp_project a directory with a config file fake_remote in-memory stand-in for the API settings built from tmp_project cli a runner wired to the above Each fixture does one thing Tests request only what they need No global state to reset between runs Composable fixtures are what keep a command test to three lines of setup. 
# tests/conftest.py
import pytest
from typer.testing import CliRunner

from mytool.cli import app

@pytest.fixture
def tmp_project(tmp_path):
    (tmp_path / ".mytool.toml").write_text('region = "eu-west-1"\nretries = 2\n')
    (tmp_path / "data").mkdir()
    (tmp_path / "data" / "a.txt").write_text("x")
    return tmp_path

@pytest.fixture
def cli(tmp_project, monkeypatch):
    monkeypatch.chdir(tmp_project)
    monkeypatch.delenv("MYTOOL_RETRIES", raising=False)   # a developer's shell must not leak in
    return CliRunner(mix_stderr=False)

That last line matters more than it looks. A suite that reads real environment variables passes on your machine and fails on someone else's, and the failure is baffling because nothing in the test mentions the variable.

The one test that is worth a subprocess

Everything above runs in-process, which means none of it exercises the console script, the packaging metadata or the entry point. Exactly one test should:

python -m build
python -m venv /tmp/smoke
/tmp/smoke/bin/pip install dist/*.whl
/tmp/smoke/bin/mytool --version
Test stages in a CI run The order of testing stages in continuous integration: fast unit tests, then command runner tests, then a packaged smoke test. Test stages in a CI run Unit core logic, no CLI ~1 s Runner flags and exit codes ~5 s Packaged install the wheel, run it ~30 s ordered so the fastest signal arrives first Failing fast on unit tests keeps the expensive packaging stage for changes that deserve it.

It belongs in CI rather than in the pytest suite, and it catches a class of failure nothing else can see: a package missing from the wheel, an unlisted runtime dependency, a typo in [project.scripts]. Those are precisely the bugs that reach users while a green test suite says everything is fine.

Fakes, mocks and what to reach for

Fakes age better than mocks A comparison of hand-written fake collaborators and mock objects for testing command line tools. Fakes age better than mocks A small fake class ✓ Fails with a readable AttributeError when the interface changes ✓ Can assert on state afterwards, not on call sequences ✓ Reads like the real thing, so tests document the contract ✓ Reusable across a whole test module A patched mock ✗ Passes happily after the real method is renamed ✗ Couples the test to how, not what ✗ Patch targets break when a module moves ✗ Needs a new patch for every call site Use mocks where you must assert an interaction happened; use fakes everywhere else.

A small hand-written fake is usually the better tool:

class FakeBucket:
    def __init__(self) -> None:
        self.uploaded: list[str] = []
        self.current: set[str] = set()

    def is_current(self, path) -> bool:
        return str(path) in self.current

    def upload(self, path, *, retries: int = 3) -> None:
        self.uploaded.append(str(path))

It fails loudly with an AttributeError when the real interface gains a method, which is exactly when you want to know. A MagicMock accepts anything, so a renamed method leaves the test green and the code broken. Reach for unittest.mock when the interaction itself is the thing under test — "did we call commit exactly once" — and for a fake everywhere else.

Testing the layers underneath

Runner tests prove the wiring. Everything about behaviour belongs one level down, where tests are an order of magnitude cheaper and the failures point at the right place.

# tests/core/test_sync.py — no runner, no parsing, no filesystem beyond tmp_path
def test_skips_files_that_are_already_current(tmp_path):
    (tmp_path / "a.txt").write_text("x")
    (tmp_path / "b.txt").write_text("y")
    bucket = FakeBucket()
    bucket.current.add(str(tmp_path / "a.txt"))

    result = sync_directory(tmp_path, bucket=bucket)

    assert result == SyncResult(uploaded=1, skipped=1)
    assert bucket.uploaded == [str(tmp_path / "b.txt")]

Notice what this test does not need: no app, no argument list, no captured output, no exit code. It states a rule about the domain and fails when that rule breaks. Parametrising it over a dozen edge cases — empty directory, nested directories, a symlink, a file that disappears mid-run — costs almost nothing.

The corresponding runner test is then deliberately thin:

def test_dry_run_flag_reaches_the_core(cli, monkeypatch):
    seen = {}
    monkeypatch.setattr(
        "mytool.commands.sync.core.sync_directory",
        lambda source, **kwargs: seen.update(kwargs) or SyncResult(0, 0),
    )

    result = cli.invoke(app, ["sync", "data", "--dry-run"])

    assert result.exit_code == 0
    assert seen["dry_run"] is True

One assertion about the flag, one about the exit code, and nothing about what syncing means. When someone renames --dry-run, this fails; when someone changes how skipping works, it does not — and that separation is what stops a suite becoming a wall of red for a single change.

Testing configuration and precedence

Configuration is the other place where CLI test suites go wrong, usually by testing precedence through the command line. Written as a function that takes its inputs, the whole precedence system is testable without a runner at all:

def test_flag_beats_environment_beats_file():
    merged = resolve(
        cli={"retries": 5},
        env={"retries": "9"},
        file={"retries": 7},
        defaults={"retries": 3},
    )
    assert merged["retries"] == 5

def test_unset_flag_does_not_clobber_the_file():
    merged = resolve(cli={"retries": None}, env={}, file={"retries": 7}, defaults={"retries": 3})
    assert merged["retries"] == 7

Then one runner test per source proves the wiring: that --retries lands in cli, that MYTOOL_RETRIES is read, that --config is honoured. Three tests, rather than a combinatorial matrix run through the CLI.

The trap to avoid is a test that leaves an environment variable set. monkeypatch.setenv undoes itself; os.environ[...] = ... does not, and the resulting failure appears in an unrelated test that happens to run afterwards.

Making failures readable

A test suite is a diagnostic tool, and a few habits make the diagnosis immediate.

Assert on structured values, not on rendered text, wherever you have the choice. assert result == SyncResult(uploaded=1, skipped=1) prints both sides on failure; assert "1 uploaded" in output prints a wall of captured text and leaves you searching.

Give the runner's result a chance to explain itself. When a runner test fails unexpectedly, the exception is on the result object rather than raised:

def invoke(cli, argv):
    result = cli.invoke(app, argv)
    if result.exception and not isinstance(result.exception, SystemExit):
        raise result.exception          # surface the real traceback in the test output
    return result

That three-line helper turns "exit code was 1, expected 0" into the actual KeyError with its traceback, which is usually the difference between a minute and twenty.

Name tests after the rule, not the function. test_missing_config_file_is_a_usage_error tells you what broke from the summary line alone; test_main_2 requires opening the file.

What to do about slow suites

If a CLI suite is slow, the cause is nearly always one of three things, and each has a direct fix.

Subprocess tests. Replace them with runner tests, keeping one packaged smoke test in CI. This is usually the entire problem.

Heavy imports at collection time. pytest imports every test module before running anything, so a module-level import pandas in a test file costs on every run, including pytest -k something_else. Import inside the test, or inside a fixture.

Real sleeps and timeouts. A retry test that waits two seconds three times is six seconds of nothing. Inject the sleep function, or pass a zero backoff in tests — the retry logic is what you are testing, not the waiting.

Measure before optimising: pytest --durations=10 prints the ten slowest tests, and the answer is usually obvious from that list alone.

A suite worth copying

For a tool of any size, the following shape covers the ground without ceremony:

tests/
  conftest.py                 # tmp_project, cli, fake collaborators
  core/
    test_sync.py              # the rules, parametrised
    test_config.py            # precedence, coercion, error messages
  commands/
    test_sync_cmd.py          # flags, defaults, exit codes
    test_status_cmd.py
  test_cli_contract.py        # the invocation table: --help, --version, usage errors

Four properties are doing the work. The tree mirrors the source, so finding a file's tests is mechanical. Rules live in core/ tests where they are cheap to parametrise. Command tests are thin and there is roughly one per command. And test_cli_contract.py holds the table of invocations that must keep working — the closest thing a CLI has to a public API test.

Add to that a pyproject.toml section that makes the defaults sensible:

[tool.pytest.ini_options]
addopts = "-q --strict-markers --strict-config"
testpaths = ["tests"]
filterwarnings = ["error"]        # a new DeprecationWarning fails the build, deliberately

filterwarnings = ["error"] is the one people hesitate over and then keep. It turns a library deprecation into a failing test on the day it appears, rather than a surprise when that library removes the feature two releases later.

Common failure modes

Four patterns account for most of the trouble people report with CLI test suites.

Tests that depend on each other. A test writes a config file into the repository, and a later test reads it. The fix is tmp_path everywhere and no writes outside it — plus running with pytest -p no:randomly off, so ordering problems surface rather than hide.

Assertions against whole help screens. Any change to a description breaks a dozen tests. Assert that a flag name appears, not that the screen matches; if you genuinely need the whole screen pinned, use a snapshot so the diff is reviewable.

Mocking too close to the test. Patching the function directly under test means the assertion proves nothing. Patch at a boundary — the transport, the clock, the filesystem — and let your own code run.

No test for the failure path. Most CLI bugs that reach users are in error handling, because it is the code least exercised during development. One test per documented exit code is a small investment that pays out repeatedly.

Where to go next

Frequently asked questions

Should I test through a subprocess to be realistic?

Once, in CI, against the installed wheel — and nowhere else. A subprocess costs interpreter start-up per test, hides tracebacks behind captured output, and tests the same code the runner does. The realism it adds is entirely about packaging, which is why one test is enough.

How do I test a command that writes files?

Give it a tmp_path and assert on what appeared. runner.isolated_filesystem() is an alternative for commands that write relative paths, since it changes the working directory for the duration of the block. Either way, never let a test write into the repository — a suite that leaves artifacts behind eventually depends on them.

My test passes locally and fails in CI. Where do I look first?

The environment and the working directory. A developer machine has exported variables, a populated cache and a config file in the home directory; CI has none of those. Clearing your tool's variables in a fixture and changing into tmp_path removes both classes of difference in two lines.

Do I need to test --help output?

Test that it exits 0 and mentions each command, which catches a broken registration. Testing the exact text is what snapshot tests are for, and only worth doing if the help output is documentation you have committed to keeping stable.

How many tests should a command have?

Usually one or two at the runner level — the happy path and the documented failure — plus whatever the underlying function needs at the unit level. If a command needs six runner tests, that is a strong signal it is holding logic that belongs in core/.

What about testing on Windows?

Add one Windows job to the CI matrix rather than special-casing paths in tests. The failures it surfaces are real — path separators in expected output, a console encoding that cannot render a character, a temporary file left open — and none of them are visible from Linux.

Can I run the suite against several Python versions locally?

uv run --python 3.11 pytest and uv run --python 3.13 pytest cover it without a separate tool, and tox or nox remain good options if you want one command to drive the whole matrix. Locally, running the ends of your supported range is usually enough — the middle versions rarely surface a distinct failure.