End-to-End Testing an Installed Python CLI

CliRunner tests are fast and thorough, and they should be the backbone of a CLI's test suite. But they run your command in-process, from the source tree, with stdout and stderr replaced by in-memory buffers and no real terminal, pipe or signal in sight. A class of bugs lives exactly in that gap: a console-script entry point that points at the wrong function, a command that behaves differently when its output is a pipe, an exit status that is correct in-process and wrong through the launcher, state that leaks between invocations in one process and never in real use. End-to-end tests close the gap by running the installed command as a separate process, the way users and scripts do. This guide builds a small pytest fixture that installs your CLI once per session into an isolated environment, a helper for running it, and a focused set of end-to-end tests. It belongs to the testing Python CLI applications topic.

Prerequisites

• A CLI packaged with a [project.scripts] entry point and a solid in-process suite, as in testing Click commands with CliRunner.
• uv available on the test machine (used here to build and install quickly).

Where end-to-end tests fit

Most tests should stay where they are: unit tests of core functions, and CliRunner tests of commands. End-to-end tests are the thin top layer — a few dozen at most — chosen to cover what only a real process can show:

The recipe: install once, run many

The expensive part is building and installing the package; do it once per test session and share the result:

# tests/e2e/conftest.py
from __future__ import annotations

import os
import subprocess
import sys
from collections.abc import Callable
from dataclasses import dataclass
from pathlib import Path

import pytest

PROJECT = Path(__file__).resolve().parents[2]
COMMAND = "mytool"


@dataclass(frozen=True)
class Run:
    code: int
    stdout: str
    stderr: str


@pytest.fixture(scope="session")
def installed_cli(tmp_path_factory: pytest.TempPathFactory) -> Path:
    """Build the wheel and install it into a fresh virtual environment; return the executable."""
    root = tmp_path_factory.mktemp("e2e")
    dist, venv = root / "dist", root / "venv"
    subprocess.run(["uv", "build", "--wheel", "--out-dir", str(dist)], cwd=PROJECT, check=True,
                   capture_output=True)
    wheel = next(dist.glob("*.whl"))
    subprocess.run(["uv", "venv", "--quiet", str(venv)], check=True)
    python = venv / ("Scripts/python.exe" if os.name == "nt" else "bin/python")
    subprocess.run(["uv", "pip", "install", "--quiet", "--python", str(python), str(wheel)],
                   check=True)
    exe = venv / ("Scripts" if os.name == "nt" else "bin") / (COMMAND + (".exe" if os.name == "nt" else ""))
    assert exe.exists(), f"console script {exe} was not installed"
    return exe


@pytest.fixture
def cli(installed_cli: Path, tmp_path: Path) -> Callable[..., Run]:
    """Run the installed command in an isolated directory with a controlled environment."""
    home = tmp_path / "home"
    home.mkdir()

    def run(*args: str, input: str | None = None, env: dict[str, str] | None = None,
            timeout: float = 30) -> Run:
        base = {"PATH": os.environ.get("PATH", ""), "HOME": str(home), "USERPROFILE": str(home),
                "XDG_CONFIG_HOME": str(home / ".config"), "NO_COLOR": "1",
                "SYSTEMROOT": os.environ.get("SYSTEMROOT", "")}
        proc = subprocess.run([str(installed_cli), *args], cwd=tmp_path, input=input,
                              capture_output=True, text=True, timeout=timeout,
                              env={**base, **(env or {})})
        return Run(proc.returncode, proc.stdout, proc.stderr)

    return run

Three isolation decisions matter:

• A fresh virtual environment with only the wheel installed. No development dependencies, no editable install, no source tree on sys.path. If a runtime dependency is missing from pyproject.toml, the command fails here — as it would for users.
• The working directory is a temporary directory, so Python cannot import the package from the checkout by accident.
• A controlled environment. A throwaway HOME (and USERPROFILE on Windows) keeps tests from reading or writing the developer's real config, keyring or cache; NO_COLOR makes output deterministic.

The recipe: tests that need a real process

# tests/e2e/test_installed.py
import json
import re
import signal
import subprocess
import sys
import time

import pytest

pytestmark = pytest.mark.e2e


def test_version_and_help(cli):
    r = cli("--version")
    assert r.code == 0 and re.fullmatch(r"mytool \d+\.\d+\.\d+.*\n", r.stdout)
    assert cli("--help").code == 0


def test_usage_error_exit_status(cli):
    r = cli("--no-such-flag")
    assert r.code == 2
    assert r.stdout == "" and "No such option" in r.stderr     # errors on stderr, not stdout


def test_json_output_is_clean_stdout(cli):
    r = cli("site", "list", "--json", "-v")
    assert r.code == 0
    json.loads(r.stdout)                                      # verbose chatter went to stderr


def test_reads_stdin_from_a_pipe(cli):
    r = cli("lint", "-", input="name: web\nreplicas: 2\n")
    assert r.code == 0


def test_config_is_written_under_home(cli, tmp_path):
    assert cli("config", "set", "region", "eu").code == 0
    assert (tmp_path / "home" / ".config" / "mytool" / "config.toml").exists()


@pytest.mark.skipif(sys.platform == "win32", reason="POSIX signals")
def test_sigint_exits_130(installed_cli, tmp_path):
    proc = subprocess.Popen([str(installed_cli), "watch", str(tmp_path)],
                            stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
    time.sleep(1.0)
    proc.send_signal(signal.SIGINT)
    _, err = proc.communicate(timeout=10)
    assert proc.returncode == 130
    assert "Traceback" not in err

Each test checks something CliRunner cannot: the installed launcher and version metadata, the real separation of stdout and stderr, stdin from an actual pipe, files written relative to a real home directory, and signal handling in a real process. Adjust the commands to your own tool; the categories are what matter.

Mark them (pytest.mark.e2e) and register the marker in pyproject.toml, so developers can run pytest -m "not e2e" for a fast loop and CI runs everything.

Keeping end-to-end tests deterministic

Process-level tests are more exposed to the machine they run on than in-process ones, and flakiness here quickly erodes trust in the whole suite. A few rules keep them stable:

• Control every input. The environment dictionary in the cli fixture starts almost empty on purpose: no inherited MYTOOL_* variables, no developer config, no proxy settings leaking in. Add variables explicitly per test.
• Never depend on the network. If a command talks to an API, give the CLI a base-URL setting and point it at a local fake server started by a fixture, so tests pass on an aeroplane and in locked-down CI.
• Avoid fixed sleeps where possible. The signal test above sleeps for a second to let the process start; for anything more complex, have the command print a "ready" line and wait for it on the pipe instead.
• Set timeouts on every subprocess call. A command that unexpectedly waits for input will otherwise hang the whole suite; timeout=30 turns it into a clear failure.
• Run on every platform you ship to. End-to-end tests are where Windows path, encoding and launcher differences surface, so include them in the Windows and macOS jobs of the matrix in testing a CLI across Python versions with GitHub Actions.

UX considerations

End-to-end tests protect the experience users actually have:

• Streams. A test that parses --json output with -v enabled proves that diagnostics never contaminate data — the property scripts depend on, explained in working with stdin, stdout and pipes.
• Exit statuses. Scripts branch on them; the real process's status is the only one that counts.
• No tracebacks. Asserting that stderr contains no Traceback for expected failures and interrupts guards the friendly-error behaviour described in friendly error messages and tracebacks.
• Startup time. An end-to-end test can time --help and fail if it regresses past a budget — something users feel on every invocation. Keep the budget generous to avoid flaky failures on slow CI machines.

Testing the behaviour

The end-to-end suite is itself the test, but two habits keep it trustworthy. First, run it against the artefact CI will publish — in CI, point the fixture at the wheel built by the build job rather than building again, exactly as in smoke-testing the built wheel in CI. Second, prove it can fail: temporarily break the entry point in pyproject.toml and confirm the session fixture fails with a clear message. An end-to-end suite that has never failed may not be testing the installed command at all.

Conclusion

Keep most CLI tests in-process and fast, and add a thin layer of end-to-end tests that run the installed command as a real process. Build and install once per session into an isolated environment with only runtime dependencies, run each test in a temporary directory with a controlled HOME and environment, and use those tests for what only a process can show: entry points, exit statuses, stream separation, pipes, files in real locations and signals. They take seconds, and they catch the bugs users would otherwise find first.

Frequently asked questions

Related

• Up: Testing Python CLI applications
• Testing Click commands with CliRunner
• Property-based testing CLI arguments with Hypothesis
• Snapshot testing CLI output
• Smoke-testing the built wheel in CI