Calling External Commands Safely with subprocess

Your CLI needs to run another program — rsync, terraform, ffmpeg, kubectl — and act on the result. The one-liner that works on your machine (subprocess.run(f"rsync -a {src} {dest}", shell=True)) ignores failures, breaks on spaces in paths, decodes output with whatever locale is active, can hang forever, and turns a malicious filename into a command. This guide replaces it with a small helper you can drop into any Typer or Click project: one function that runs a program with every behaviour chosen on purpose, and reports failures as your tool's errors rather than Python's. It is the practical companion to the subprocess topic overview.

Prerequisites

• Python 3.10 or newer (the code uses X | None union syntax).
• A CLI built with Typer or Click — the examples use Typer; the helper itself has no framework dependency.
• pytest for the tests at the end.
• Some external program to call. The examples use git and rsync, but anything on your PATH works.

Why the one-liner is dangerous

Before the recipe, it is worth being precise about what goes wrong with a command string, because each problem maps to one line of the fix.

With shell=True, Python runs /bin/sh -c "<your string>". The shell splits on whitespace, so a path with a space becomes two arguments. It expands * and ~, so a filename with a glob character matches other files. It interprets ;, |, &&, backticks and $(...), so a crafted value runs extra commands. Quoting correctly by hand is possible but fragile, and nobody reviewing the code can tell at a glance whether it was done right.

Without check=True, a failing child returns a non-zero returncode that nothing reads, and your tool carries on as though the step succeeded. Without an explicit encoding, text=True decodes with locale.getpreferredencoding() — UTF-8 on most Linux systems, often cp1252 on Windows — so the same output parses on one machine and raises UnicodeDecodeError on another. Without a timeout, a child waiting for a password prompt that will never be answered blocks your CLI until someone notices.

The recipe: one helper for every call

The helper below is the whole technique. It takes an argument list and returns a result object; every failure mode becomes a single exception type your command layer can turn into a message.

# src/mytool/proc.py
from __future__ import annotations

import os
import shutil
import subprocess
from dataclasses import dataclass
from pathlib import Path


class CommandError(Exception):
    """A child program could not be run, timed out, or failed."""

    def __init__(self, message: str, exit_code: int, detail: str = "") -> None:
        super().__init__(message)
        self.exit_code = exit_code
        self.detail = detail


@dataclass(frozen=True)
class Result:
    stdout: str
    stderr: str
    returncode: int


def run_command(
    argv: list[str],
    *,
    cwd: Path | None = None,
    extra_env: dict[str, str] | None = None,
    timeout: float = 120.0,
    check: bool = True,
) -> Result:
    """Run argv without a shell and return its decoded output."""
    if not argv:
        raise ValueError("argv must not be empty")

    program = shutil.which(argv[0])
    if program is None:
        raise CommandError(f"{argv[0]!r} was not found on PATH", exit_code=127)

    env = {**os.environ, **(extra_env or {})}
    try:
        proc = subprocess.run(
            [program, *argv[1:]],
            cwd=cwd,
            env=env,
            stdin=subprocess.DEVNULL,
            capture_output=True,
            text=True,
            encoding="utf-8",
            errors="replace",
            timeout=timeout,
            check=False,
        )
    except subprocess.TimeoutExpired as exc:
        raise CommandError(
            f"{argv[0]} did not finish within {timeout:g}s", exit_code=124
        ) from exc

    if check and proc.returncode != 0:
        raise CommandError(
            f"{argv[0]} exited with status {proc.returncode}",
            exit_code=1,
            detail=proc.stderr.strip(),
        )
    return Result(proc.stdout, proc.stderr, proc.returncode)

A few choices deserve explanation.

shutil.which() before running. Without it, a missing program raises FileNotFoundError from deep inside subprocess, with a message that names the internal call rather than the tool the user needs to install. Resolving first gives you a clean error and, on Windows, the full path to .cmd and .bat shims such as npm.cmd, which cannot be launched by bare name without a shell.

check=False internally, with our own check. subprocess.run(check=True) raises CalledProcessError, which we would only catch and re-raise. Checking the return code ourselves keeps one code path and lets callers opt out — useful for commands like git diff --quiet where exit code 1 is an answer, not a failure.

stdin=subprocess.DEVNULL. A child that decides to prompt ("Password:", "Overwrite? y/N") gets end-of-file immediately and fails fast instead of hanging. If your tool is reading its own input from a pipe, this also stops the child swallowing it.

errors="replace". Output you only display should never crash your tool over one bad byte. If you parse the output and need to know about corruption, switch that call to errors="strict".

Wiring it into a command

The command layer's only job is to call the helper and translate CommandError into user-facing output and an exit code. Keep that translation in one place — a small decorator or a top-level handler — rather than repeating try/except in every command.

# src/mytool/cli.py
from pathlib import Path

import typer

from mytool.proc import CommandError, run_command

app = typer.Typer(no_args_is_help=True)


@app.callback()
def main() -> None:
    """Deployment helpers for the docs site."""


@app.command()
def sync(
    src: Path = typer.Argument(..., exists=True, file_okay=False),
    dest: str = typer.Argument(..., help="Local path or host:path"),
    dry_run: bool = typer.Option(False, "--dry-run", "-n"),
) -> None:
    """Mirror SRC into DEST using rsync."""
    argv = ["rsync", "-a", "--delete"]
    if dry_run:
        argv.append("--dry-run")
    argv += ["--", f"{src}/", dest]
    try:
        run_command(argv, timeout=600)
    except CommandError as exc:
        typer.secho(f"error: {exc}", fg=typer.colors.RED, err=True)
        if exc.detail:
            typer.echo(exc.detail.splitlines()[-1], err=True)
        raise typer.Exit(exc.exit_code)
    typer.echo(f"synced {src} -> {dest}")


if __name__ == "__main__":
    app()

Notice -- before the positional paths. The argument list already prevents shell injection, but rsync still parses its own options — a directory named --rsh=... would otherwise be read as a flag. The shell injection guide covers that second layer in more depth.

UX considerations

The helper decides what happens; these decisions shape how it feels to the person running your tool.

• Fail before starting. If a command needs three programs, check all three with shutil.which() at the top of the command and report every missing one together. Discovering the second missing dependency after the first step has already modified files is the worst possible experience.
• Name the program and the fix. "rsync was not found on PATH" is good; adding "install it with brew install rsync or your package manager" is better. Users can act on the second one without searching.
• Show the tail, not the flood. When a child fails, its last few stderr lines usually contain the reason. Print those, and offer the full log behind --verbose.
• Keep your stdout clean. Child output you pass through goes to stderr unless it is genuinely your command's result. Scripts piping your tool depend on that separation; the reasoning is in working with stdin, stdout and pipes.
• Pick timeouts per call. Two minutes suits git fetch; a video transcode might need an hour. Expose a --timeout option for the long ones rather than one global value.

Testing the behaviour

Test the helper against real, always-available programs — the Python interpreter itself is ideal, because sys.executable exists on every machine that runs your tests. Test commands by replacing the helper, so they never depend on rsync being installed.

# tests/test_proc.py
import sys

import pytest

from mytool.proc import CommandError, run_command


def test_captures_stdout():
    result = run_command([sys.executable, "-c", "print('héllo')"])
    assert result.stdout == "héllo\n"


def test_nonzero_exit_raises_with_detail():
    code = "import sys; sys.stderr.write('boom\\n'); sys.exit(3)"
    with pytest.raises(CommandError) as info:
        run_command([sys.executable, "-c", code])
    assert info.value.exit_code == 1
    assert info.value.detail == "boom"


def test_check_false_returns_code():
    result = run_command([sys.executable, "-c", "raise SystemExit(1)"], check=False)
    assert result.returncode == 1


def test_missing_program_is_127():
    with pytest.raises(CommandError) as info:
        run_command(["definitely-not-a-real-program-xyz"])
    assert info.value.exit_code == 127


def test_timeout_is_124():
    with pytest.raises(CommandError) as info:
        run_command([sys.executable, "-c", "import time; time.sleep(5)"], timeout=0.2)
    assert info.value.exit_code == 124


def test_stdin_is_closed():
    result = run_command([sys.executable, "-c", "import sys; print(repr(sys.stdin.read()))"])
    assert result.stdout.strip() == "''"

For the command layer, patch run_command where the CLI module imports it and assert on the argument list — that is where the -- and --dry-run logic lives:

from typer.testing import CliRunner

from mytool import cli

runner = CliRunner()


def test_sync_builds_argv(monkeypatch, tmp_path):
    calls = []
    monkeypatch.setattr(cli, "run_command", lambda argv, **kw: calls.append(argv))
    result = runner.invoke(cli.app, ["sync", str(tmp_path), "backup:/srv", "-n"])
    assert result.exit_code == 0
    assert calls == [["rsync", "-a", "--delete", "--dry-run", "--", f"{tmp_path}/", "backup:/srv"]]

More patterns for this style of test are in testing Click commands with CliRunner.

Conclusion

A safe subprocess call is not clever code — it is a dozen deliberate keyword arguments and a clear error type. Put them in one helper, route every external program through it, and the recurring bugs disappear together: no injection because there is no shell, no silent failures because the exit code is checked, no hangs because there is a timeout and no stdin, and no encoding crashes because decoding is explicit. When you need output while the child is still running, graduate to streaming subprocess output in real time; when you need finer control over what a timeout kills, read handling subprocess timeouts and exit codes.

Frequently asked questions

Related

• Up: Running subprocesses from Python CLIs
• Handling subprocess timeouts and exit codes
• Avoiding shell injection in Python CLIs
• Wrapping git and other tools from a Python CLI
• Choosing exit codes for CLI tools