Separating Logs from Program Output in a Python CLI

mytool list --json | jq '.[0].name' fails with "parse error: Invalid numeric literal at line 1, column 8". The culprit is one line at the top of the output: Fetching servers.... Somebody added a friendly status message with print(), and every script that consumed the command's output broke. This is the most common way a CLI stops being composable, and it is entirely avoidable with one rule applied everywhere: stdout carries the program's result; stderr carries everything else — logs, progress, warnings, prompts, status messages. The rule is simple; applying it consistently means knowing where each library writes by default, routing Rich, logging and subprocess output deliberately, and testing that a piped run produces nothing but data. This guide does all of that. It belongs to the structured logging topic, alongside working with stdin, stdout and pipes.

Prerequisites

• A CLI with some machine-readable output, as in emitting JSON output for scripting.

Two streams, two audiences

Unix gives every process two output streams so that a pipeline can pass data along while diagnostics still reach the person watching. mytool list | grep web filters the list; a warning printed to stderr appears on the terminal, unfiltered, because stderr is not piped. Redirecting 2>/dev/null silences diagnostics without touching data; > out.json 2> run.log separates them into files. None of this works if the program mixes them.

Where things write by default

The surprises are in the defaults:

• print() and typer.echo()/click.echo() write to stdout unless told otherwise (file=sys.stderr, err=True).
• logging handlers created by basicConfig write to stderr — a good default.
• Rich Console() writes to stdout unless created with stderr=True. Status messages, tables of warnings and console.log() calls through a default console all land in the data stream.
• Rich progress bars and spinners write to their console's stream — stdout by default.
• Subprocesses inherit both streams; a tool you call can print to your stdout.

The recipe

Create two consoles once and use them deliberately:

# src/mytool/console.py
from rich.console import Console

out = Console()                    # results: tables, JSON, anything a script may consume
err = Console(stderr=True)         # everything else: status, progress, warnings, prompts

# src/mytool/cli.py
import json
import logging
import subprocess
import sys

import typer
from rich.progress import track

from mytool.console import err, out

app = typer.Typer()
log = logging.getLogger("mytool")
SERVERS = [{"name": "web-1", "status": "ok"}, {"name": "web-2", "status": "hot"}]


@app.callback()
def main(verbose: int = typer.Option(0, "-v", count=True)) -> None:
    """Fleet tool."""
    logging.basicConfig(level=logging.DEBUG if verbose else logging.WARNING,
                        stream=sys.stderr, format="%(levelname)s %(name)s: %(message)s")


@app.command("list")
def list_servers(as_json: bool = typer.Option(False, "--json")) -> None:
    """List servers."""
    err.print("[dim]fetching servers…[/dim]")
    log.debug("fetched %d servers", len(SERVERS))
    for _ in track(range(3), description="checking", console=err, transient=True):
        pass
    if as_json:
        sys.stdout.write(json.dumps(SERVERS) + "\n")
    else:
        for s in SERVERS:
            out.print(f"{s['name']:<8} {s['status']}")
    if any(s["status"] == "hot" for s in SERVERS):
        err.print("[yellow]warning:[/yellow] 1 server is running hot")


@app.command()
def version() -> None:
    """Show versions of mytool and the git it uses."""
    git = subprocess.run(["git", "--version"], capture_output=True, text=True)
    out.print(f"mytool 1.6.0 ({git.stdout.strip()})")

The status line, the progress bar, the debug log and the warning all go to stderr; only the list itself — table or JSON — goes to stdout. JSON is written with sys.stdout.write rather than through Rich, so no markup processing or line wrapping can alter it. The subprocess's output is captured and incorporated deliberately, rather than inherited and allowed to leak.

Edge cases that trip people up

• Prompts belong on stderr too. typer.prompt and Rich prompts write their question to the terminal; make sure custom prompts do as well, so mytool init > config.toml still shows the questions. See building interactive prompts and menus.
• "Nothing found" is a diagnostic. An empty result prints nothing on stdout (or [] in JSON mode) and a message on stderr.
• Summaries are diagnostics. "Synced 42 files in 3.1s" describes the run; unless the command's job is to produce that number, it goes to stderr.
• Errors are diagnostics, even in JSON mode. A machine-readable error object still goes to stderr with a non-zero exit code, as in reporting machine-readable errors in JSON mode.
• Windows encoding applies to both streams. Redirecting stderr to a file on Windows can hit encoding problems just like stdout; the fixes in fixing Unicode and encoding errors on Windows apply.

Subprocess output: pass through, capture or relabel

Child processes need the same decision for their output. There are three good options, depending on what the child prints:

import subprocess
import sys

# 1. The child's output IS your result: let it inherit stdout.
subprocess.run(["git", "log", "--oneline", "-5"], check=True)

# 2. You need the output as data: capture it and decide what to print.
head = subprocess.run(["git", "rev-parse", "HEAD"], capture_output=True, text=True,
                      check=True).stdout.strip()

# 3. The child's output is progress chatter: send its stdout to YOUR stderr.
subprocess.run(["npm", "run", "build"], stdout=sys.stderr, check=True)

The third form is the one people forget. A build tool, package manager or test runner invoked as a step of your command writes progress to its stdout, which by inheritance is your stdout — and lands in the middle of your JSON. Passing stdout=sys.stderr relabels it as diagnostics: the user still sees the build output on the terminal, and anything piped from your command stays clean. When the child's chatter is very long, capture it instead, and show it only on failure or at -v, as in streaming subprocess output in real time.

On Windows and in CI, sys.stderr can be a pipe without a real file descriptor in unusual hosts; if subprocess complains, fall back to stdout=subprocess.PIPE and forward the captured text to stderr yourself.

Catch stray prints with a lint rule

The usual regression is a new print() added for debugging or as a friendly message. Ruff's T20 rules (flake8-print) flag print calls; enable them for the package and allow the few legitimate uses explicitly:

[tool.ruff.lint]
extend-select = ["T20"]

[tool.ruff.lint.per-file-ignores]
"src/mytool/console.py" = ["T20"]
"tests/**" = ["T20"]

Combined with two named consoles, that makes the right choice the easy one: write results with out, everything else with err, and the linter flags anything that bypasses both.

Testing the behaviour

Click 8.2 and later (and Typer) capture stdout and stderr separately in CliRunner results. Assert that stdout contains exactly the data, even at the highest verbosity:

# tests/test_streams.py
import json

from typer.testing import CliRunner

from mytool.cli import app

runner = CliRunner()


def test_json_stdout_is_clean_even_when_verbose():
    result = runner.invoke(app, ["-v", "list", "--json"])
    assert result.exit_code == 0
    assert json.loads(result.stdout) == [{"name": "web-1", "status": "ok"},
                                         {"name": "web-2", "status": "hot"}]
    assert "fetching servers" in result.stderr
    assert "warning" in result.stderr


def test_table_output_has_no_diagnostics():
    result = runner.invoke(app, ["list"])
    assert result.stdout.splitlines() == ["web-1    ok", "web-2    hot"]

Running the verbose JSON case is the key: it is the combination most likely to reveal a debug message or progress bar on the wrong stream. An end-to-end variant pipes the installed command into python -c "import json,sys; json.load(sys.stdin)" in CI, which also catches output written by subprocesses and libraries outside the runner.

Conclusion

Composability rests on one rule: stdout for the result, stderr for everything else. Know the defaults — print, echo and Rich consoles go to stdout; logging goes to stderr — create two named consoles, route progress, prompts, warnings, summaries and errors to stderr, capture subprocess output instead of inheriting it, flag stray print() calls with Ruff, and test that verbose JSON runs still produce parseable stdout.

Frequently asked questions

Related

• Up: Structured logging for CLI apps
• Logging with structlog in a CLI
• Adding verbose and quiet logging flags
• Detecting a TTY and adapting output
• Handling broken pipe and SIGPIPE