Logging with structlog in a Python CLI

The standard logging module can produce structured logs, as shown in structured JSON logging in Python CLIs, but it takes a custom formatter and some discipline to keep messages machine-friendly. structlog is built around the idea instead: a log call is an event with key-value fields (log.info("sync.start", files=3)), passed through a pipeline of processors that add timestamps, levels and context, and finally rendered — as colourful aligned text for a person at a terminal, or as one JSON object per line for a log collector. For a CLI that runs both interactively and in CI, that dual rendering is exactly what is needed. This guide configures structlog for a Typer CLI, maps -v flags to levels, binds per-run context, routes library logs (httpx, urllib3) through the same pipeline, and tests log output. It belongs to the structured logging topic.

Prerequisites

• uv add structlog (examples checked with structlog 26).
• The verbosity conventions from adding verbose and quiet logging flags.

The pipeline

A structlog logger collects the event name and keyword arguments into a dictionary, the event dict. Processors then run in order, each receiving and returning that dictionary: one merges context variables bound earlier, one adds the level, one adds a timestamp, one formats an exception, and the last — the renderer — turns the dictionary into a string. Swapping the renderer is the only difference between console and JSON output; everything else, including the fields, stays identical.

The recipe

# src/mytool/logs.py
from __future__ import annotations

import logging
import sys

import structlog

SHARED = [
    structlog.contextvars.merge_contextvars,
    structlog.stdlib.add_log_level,
    structlog.stdlib.add_logger_name,
    structlog.processors.TimeStamper(fmt="iso", utc=True),
]


def configure_logging(verbosity: int = 0, *, json_logs: bool | None = None) -> None:
    """One pipeline for structlog and stdlib loggers, rendered on stderr."""
    level = {0: logging.WARNING, 1: logging.INFO}.get(verbosity, logging.DEBUG)
    if json_logs is None:
        json_logs = not sys.stderr.isatty()
    renderer = (structlog.processors.JSONRenderer() if json_logs
                else structlog.dev.ConsoleRenderer(colors=sys.stderr.isatty()))
    structlog.configure(
        processors=[*SHARED, structlog.stdlib.ProcessorFormatter.wrap_for_formatter],
        logger_factory=structlog.stdlib.LoggerFactory(),
        wrapper_class=structlog.stdlib.BoundLogger,
        cache_logger_on_first_use=False,
    )
    handler = logging.StreamHandler(sys.stderr)
    handler.setFormatter(structlog.stdlib.ProcessorFormatter(
        foreign_pre_chain=SHARED,
        processors=[structlog.stdlib.ProcessorFormatter.remove_processors_meta,
                    structlog.processors.format_exc_info, renderer],
    ))
    root = logging.getLogger()
    root.handlers[:] = [handler]
    root.setLevel(level)

This configuration routes structlog through the standard library's logging machinery rather than printing directly. That costs a few more lines, and buys the most important property for a CLI: logs from libraries go through the same pipeline. httpx, urllib3, botocore and friends log with the standard logging module; with foreign_pre_chain, their records get the same timestamp, level and bound context fields, and are rendered by the same renderer. Without it, a JSON log stream from your CI job is interrupted by plain-text lines from libraries that a log collector cannot parse.

Two more decisions are encoded here. Logs go to stderr, always, so stdout stays reserved for the command's output — see separating logs from program output. And the renderer is chosen by whether stderr is a terminal: people get the readable console renderer, pipes and CI get JSON, and a --log-format option can override either way.

Wiring it into the CLI

# src/mytool/cli.py
import uuid

import structlog
import typer

from mytool.logs import configure_logging

app = typer.Typer()
log = structlog.get_logger("mytool")


@app.callback()
def main(
    verbose: int = typer.Option(0, "--verbose", "-v", count=True, help="Repeat for more detail."),
    log_format: str = typer.Option("auto", help="auto, console or json."),
) -> None:
    """Sync tool."""
    configure_logging(verbose, json_logs=None if log_format == "auto" else log_format == "json")
    structlog.contextvars.bind_contextvars(run_id=uuid.uuid4().hex[:8])


@app.command()
def sync(files: int = 3) -> None:
    """Sync files."""
    structlog.contextvars.bind_contextvars(command="sync")
    log.info("sync.start", files=files)
    for n in range(files):
        log.debug("sync.file", index=n)
    log.info("sync.done", files=files)
    typer.echo(f"synced {files} files")

bind_contextvars attaches fields to every subsequent log line in the current context — here a short run ID and the command name — without passing a logger around. That run ID is what lets you find every line of one invocation in a shared log, the idea developed in adding trace IDs and context to CLI logs.

Event names and fields

structlog rewards a naming discipline: an event name that is a stable identifier (sync.start, upload.retry, config.loaded) and fields for everything that varies (files=3, attempt=2, path=...). Messages like f"Uploading {path}" work but defeat filtering; log.info("upload.start", path=str(path)) can be queried, counted and alerted on.

Writing your own processors

A processor is just a function that takes (logger, method_name, event_dict) and returns the event dictionary, which makes cross-cutting rules easy to enforce in one place. The most useful one for a CLI is redaction, inserted into SHARED before the renderer so it applies to library logs too:

SENSITIVE = {"token", "password", "secret", "authorization", "api_key"}


def redact(logger, method_name, event_dict):
    for key in list(event_dict):
        if key.lower() in SENSITIVE:
            event_dict[key] = "***"
    return event_dict

{"user": "ann", "token": "***", "Authorization": "***", "event": "login"}

Key-based redaction catches the common mistake of logging a credential as a field; it does not catch a secret embedded in a free-text message, which is one more reason to keep variable data in fields rather than in the event string. Other small processors that earn their place: one that adds the tool version to every line, one that drops very chatty debug events from a noisy library unless -vvv is given, and one that shortens absolute paths under the user's home directory to ~/… for readability and privacy.

UX considerations

• Quiet by default. WARNING at verbosity 0 means a normal run prints only the command's output; -v adds progress-level events and -vv debug detail.
• Readable on a terminal. The console renderer aligns and colours levels and fields; keep event names short so lines stay scannable.
• Exceptions in full only when useful. log.exception(...) attaches the traceback; at verbosity 0 the CLI should still show a one-line error message to the user, as in friendly error messages and tracebacks.
• Never log secrets. Add a redaction processor before the renderer if fields might carry tokens; the patterns are in redacting secrets from CLI output and logs.

Testing the behaviour

structlog ships capture_logs, a context manager that records event dictionaries instead of rendering them — ideal for asserting that the right events with the right fields were emitted:

# tests/test_logs.py
import json

import structlog
from structlog.testing import capture_logs
from typer.testing import CliRunner

from mytool.cli import app


def test_sync_emits_start_and_done_events(monkeypatch):
    # The CLI callback calls structlog.configure(), which would replace capture_logs' setup.
    monkeypatch.setattr("mytool.cli.configure_logging", lambda *a, **k: None)
    with capture_logs() as logs:
        result = CliRunner().invoke(app, ["-v", "sync", "--files", "2"])
    assert result.exit_code == 0
    events = [entry["event"] for entry in logs]
    assert events[0] == "sync.start" and events[-1] == "sync.done"
    assert logs[0]["files"] == 2


def test_json_logs_include_bound_context(capsys):
    from mytool.logs import configure_logging

    configure_logging(1, json_logs=True)
    structlog.contextvars.bind_contextvars(run_id="r-1")
    structlog.get_logger("mytool").info("check", ok=True)
    line = json.loads(capsys.readouterr().err.strip().splitlines()[-1])
    assert line["event"] == "check" and line["run_id"] == "r-1" and line["level"] == "info"
    structlog.contextvars.clear_contextvars()

One detail matters in the first test: capture_logs works by temporarily reconfiguring structlog, and the CLI's callback calls structlog.configure() on every run, which would silently replace the capturing setup and leave logs empty. Patching configure_logging out for that test keeps the capture in place. capture_logs then tests what was logged independently of the configuration; the second test checks the rendering configuration itself, including that bound context reaches the JSON output. Clearing context variables at the end keeps one test's fields out of the next.

Conclusion

structlog fits CLIs well because one event can be rendered two ways: readable console logs for people, JSON lines for pipes and CI. Configure it through the standard library so library logs share the same pipeline and fields, write logs to stderr, map -v flags to levels, bind run-wide context with bind_contextvars, name events as stable identifiers with variable data in fields, and test emitted events with capture_logs.

Frequently asked questions

Related

• Up: Structured logging for CLI apps
• Structured JSON logging in Python CLIs
• Adding verbose and quiet logging flags
• Separating logs from program output
• Adding trace IDs and context to CLI logs