Python CLI Toolcraft

Architecture

Measuring CLI Test Coverage

Configure coverage.py for a Python CLI - branch coverage, subprocess measurement, sensible exclusions, and the code paths the numbers quietly miss.

Updated

Coverage is a tool for finding paths nobody tested, not a number to maximise. For a command-line tool it is unusually easy to reach a comfortable percentage while leaving the parts users touch first — the entry point, the error boundary, the non-terminal branch — entirely unexercised.

TL;DR

  • Measure src/, not the installed package, so line numbers point at your working tree.
  • Turn on branch coverage; line coverage hides half-tested conditionals.
  • Set fail_under to a floor you already clear, and raise it when it improves.
  • Exclude unreachable code with a reason, never with a blanket ignore.
  • Remember what coverage cannot see: the console script, lazy imports, and TTY-only branches.

Prerequisites

pytest, pytest-cov (or coverage directly), and a project laid out with the package under src/. Everything here is configured in pyproject.toml.

A configuration that tells the truth

[tool.coverage.run]
source = ["src"]
branch = true
parallel = true                     # combine data from subprocesses
omit = ["*/__main__.py"]

[tool.coverage.report]
show_missing = true
skip_covered = true
fail_under = 85
exclude_also = [
  "if TYPE_CHECKING:",
  "raise NotImplementedError",
  "if __name__ == .__main__.:",
  "@overload",
]
A coverage setup that tells the truth Coverage configuration for a command line project: measure the source package, run in subprocess mode for the smoke test, and exclude generated code. A coverage setup that tells the truth [tool.coverage] configured in pyproject.toml source = ["src"] measure the package, not the tests branch = true partially covered ifs show up parallel = true combine subprocess runs fail_under a floor CI enforces measuring src/ rather than the installed package is what keeps line numbers meaningful Branch coverage is the setting that turns a comfortable number into a useful one.

Three of those settings do most of the work.

source = ["src"] measures your package rather than whatever was imported, which means files with no tests at all appear at 0% instead of being silently absent. Measuring the installed package instead makes line numbers point into site-packages, which is unhelpful in a report.

branch = true is the difference between a comfortable number and a useful one. A function with if verbose: log(...) shows 100% line coverage from a single test that never sets verbose; with branch coverage it shows the untaken path.

exclude_also replaces the older exclude_lines and adds to the defaults rather than replacing them, which is what you want — the built-in pragma: no cover handling stays in place.

Run it as part of the normal suite:

pytest --cov --cov-report=term-missing --cov-report=html

What the number hides in a CLI

What CLI coverage numbers hide A table of command line code paths that coverage reports commonly miss, and the test that actually exercises each one. What CLI coverage numbers hide Blind spot Why it hides What catches it The entry point tests import the app, not the shim an installed smoke test Error boundary no test raises an unexpected error a test that raises deliberately Lazy command imports never imported during tests an import-all test The non-TTY branch the runner is never a terminal a test with isatty forced true A CLI can show 95% coverage and still fail on the first line a user runs.

Four paths are routinely invisible, and each needs a deliberate test.

The console script. Tests import app and invoke it in-process, so the generated shim, the [project.scripts] target and the packaging metadata are never touched. Coverage cannot see this at all — the only cover is the installed smoke test in CI.

The error boundary. Nothing in a normal suite raises an unexpected exception, so the except Exception clause that maps a bug to exit code 70 is never entered. Make a test raise on purpose:

def test_unexpected_error_exits_70(cli, monkeypatch):
    monkeypatch.setattr("mytool.commands.sync.core.sync_directory", _boom)

    result = cli.invoke(app, ["sync", "data"])

    assert result.exit_code == 70
    assert "internal error" in result.stderr

Lazily imported command modules. If commands resolve on demand, a suite that only tests three of them never imports the rest, and a broken import in a fourth stays hidden. An import-all test covers it in five lines:

def test_every_command_module_imports():
    package = importlib.import_module("mytool.commands")
    for module in pkgutil.iter_modules(package.__path__):
        importlib.import_module(f"mytool.commands.{module.name}")

The terminal-only branch. Under the runner, isatty() is False, so the colour, progress and prompt paths are never taken. Route the check through one helper and override it in a test, as the prompt guide describes.

Measuring across subprocesses

If any part of your suite spawns a process — a smoke test, a completion test, a subprocess.run of your own entry point — coverage needs to be told, or those lines appear untested.

[tool.coverage.run]
parallel = true
concurrency = ["multiprocessing"]

COVERAGE_PROCESS_START=pyproject.toml pytest --cov
coverage combine
coverage report

The COVERAGE_PROCESS_START variable makes child processes start measuring; coverage combine merges the resulting data files. It is worth setting up once if you have even a couple of subprocess tests, because otherwise their lines drag the number down and people start ignoring it.

Turning a gap into a test

Turning a coverage gap into a test A workflow for using a coverage report: find the uncovered branch, decide whether it matters, and either test it or mark it excluded with a reason. Turning a coverage gap into a test Uncovered branch from the report Can a user reach it? if not, exclude it Write the test for the path that matters Floor rises fail_under moves up triage yes commit Coverage is a tool for finding untested paths, not a number to maximise.

The workflow that keeps coverage useful is triage rather than accumulation. For each uncovered branch, ask whether a user can reach it.

If they can, write the test — and note that the interesting gaps are almost always error paths, which is exactly where CLI bugs live.

If they cannot, exclude it with a reason:

if sys.platform == "win32":  # pragma: no cover - exercised on the Windows CI job
    ...

A bare # pragma: no cover with no explanation is how genuinely untested code gets normalised. Requiring a reason in review keeps the exclusions honest, and the comment usually reveals whether the branch should have a test after all.

Setting a floor rather than a target

fail_under = 85

Pick a number you already clear, and raise it when the figure improves. That turns coverage into a ratchet: it cannot silently fall, and nobody is under pressure to write meaningless tests to reach an aspirational figure.

What to avoid is a target high enough that people write tests for the number. A test that invokes a function and asserts nothing raises coverage and proves nothing — worse, it makes the report lie. If you want a signal about test quality rather than reach, mutation testing (mutmut, cosmic-ray) answers that question honestly, at a much higher runtime cost.

For a pull request, a diff-coverage check is more useful than the absolute number: it asks whether the lines you changed are tested, which is the question a reviewer actually cares about.

UX considerations

Coverage output is read by people, so its ergonomics matter.

show_missing = true prints the uncovered line numbers next to each file, which is the difference between a report you act on and a number you glance at. skip_covered = true hides fully covered files so the list is short enough to read. And the HTML report is worth generating locally — clicking through to see which branch of a conditional was never taken is far faster than reading line numbers.

In CI, print the terminal report and upload the HTML as an artifact. Posting the percentage as a pull-request comment tends to produce arguments about the number rather than conversations about the missing tests.

Testing the behaviour

Coverage configuration is itself worth one sanity check, because a misconfigured source silently measures nothing:

def test_coverage_measures_the_package():
    config = tomllib.loads(Path("pyproject.toml").read_text())
    run = config["tool"]["coverage"]["run"]

    assert run["source"] == ["src"]
    assert run["branch"] is True

It looks pedantic and it catches the day someone adds an omit entry that swallows half the package while chasing a green build.

Conclusion

Configure coverage to measure your source with branches on, set a floor you already clear, and use the report to find untested paths rather than to produce a number. Then remember what it cannot see — the installed entry point, lazily imported commands, the terminal-only branches — and cover those deliberately, because they are the paths a user meets first.

Frequently asked questions

What coverage percentage should a CLI aim for?

There is no correct figure, but 80–90% is where most well-tested CLI projects land once the error paths are covered. Chasing the last few percent usually means testing generated code, defensive branches and platform-specific lines, which costs more than it returns.

Why does my coverage drop when I add lazy imports?

Because the modules are no longer imported during collection, so their lines are only measured if a test actually invokes that command. Add the import-all test described above: it restores the measurement and, more importantly, catches a broken import that lazy loading would otherwise hide until a user ran that command.

Should test files be included in the measurement?

No. Measuring tests inflates the number and tells you nothing — a test file is executed by definition. source = ["src"] excludes them automatically, which is another reason to prefer it over measuring by import.

How do I cover code that only runs on another platform?

Run the suite on that platform in CI and combine the data, or exclude it with a pragma naming the job that covers it. What to avoid is excluding it silently, which is how a Windows-only branch stays broken for a year.

Does coverage slow the test suite down?

Measurably but rarely painfully — expect roughly 10–30% on a Python suite. If it becomes an issue, run coverage in CI and without it locally; pytest with no --cov is the fast loop, and the CI run is where the floor is enforced anyway.

Is diff coverage better than a global threshold?

For reviewing a change, yes. A global floor stops the number falling; diff coverage answers the question a reviewer actually has, which is whether the lines in this pull request are tested. Running both is common: the floor protects the project, the diff check guides the review.