Writing Rotating Log Files from a Python CLI

A good CLI is quiet on the terminal: a line or two on success, a clear message on failure. But when something goes wrong — especially intermittently, or on a user's machine you cannot see — you want far more detail than the terminal showed: every request, every decision, timings, the full traceback. Asking the user to "run it again with -vvv" works only if the problem reproduces. The answer many mature tools use is a persistent debug log: every run writes detailed logs to a file in the platform's log directory, rotated so it never grows without bound, while the console handler shows only what the user asked to see. When a command fails, the error message points at the file. This guide sets that up with the standard library's RotatingFileHandler, integrates it with verbosity flags, handles unwritable locations gracefully, and tests rotation. It belongs to the structured logging for CLI apps topic.

Prerequisites

• Python 3.10+ and platformdirs for the log location (see storing app data with platformdirs).
• Logging configured in one place at startup, as described in the logging topic overview.

Two destinations, two levels

The trick is to set levels per handler, not only on the logger. The root logger lets everything through (DEBUG); the console handler filters to what the user asked for with -v flags; the file handler keeps everything. Library code keeps logging normally — it never knows two destinations exist.

The recipe

# src/mytool/logsetup.py
from __future__ import annotations

import logging
import logging.handlers
import sys
from pathlib import Path

from platformdirs import user_log_path

LOG_FORMAT = "%(asctime)s %(levelname)-7s %(name)s [%(process)d] %(message)s"


def default_log_file() -> Path:
    return user_log_path("mytool", appauthor=False) / "mytool.log"


def configure_logging(verbosity: int = 0, log_file: Path | None = None,
                      max_bytes: int = 1_000_000, backups: int = 3) -> Path | None:
    """Console at the user's level on stderr; everything at DEBUG in a rotating file.

    Returns the log file path, or None if file logging could not be set up.
    """
    root = logging.getLogger()
    root.setLevel(logging.DEBUG)
    for h in list(root.handlers):
        root.removeHandler(h)
        h.close()

    console = logging.StreamHandler(sys.stderr)
    console.setLevel({0: logging.WARNING, 1: logging.INFO}.get(verbosity, logging.DEBUG))
    console.setFormatter(logging.Formatter("%(levelname)s: %(message)s"))
    root.addHandler(console)

    path = log_file or default_log_file()
    try:
        path.parent.mkdir(parents=True, exist_ok=True)
        file_handler = logging.handlers.RotatingFileHandler(
            path, maxBytes=max_bytes, backupCount=backups, encoding="utf-8", delay=True)
    except OSError as exc:                                  # read-only home, full disk...
        logging.getLogger(__name__).warning("file logging disabled: %s", exc)
        return None
    file_handler.setLevel(logging.DEBUG)
    file_handler.setFormatter(logging.Formatter(LOG_FORMAT))
    root.addHandler(file_handler)
    for noisy in ("httpx", "httpcore", "urllib3"):
        logging.getLogger(noisy).setLevel(logging.INFO)
    return path

# src/mytool/cli.py
from __future__ import annotations

import logging
from pathlib import Path
from typing import Annotated

import typer

from mytool.logsetup import configure_logging

app = typer.Typer()
log = logging.getLogger("mytool.sync")


@app.callback()
def main(
    ctx: typer.Context,
    verbose: Annotated[int, typer.Option("--verbose", "-v", count=True)] = 0,
    log_file: Annotated[Path | None, typer.Option(envvar="MYTOOL_LOG_FILE", help="Where to write the detailed log.")] = None,
) -> None:
    """Sync tool with a persistent debug log."""
    ctx.obj = configure_logging(verbose, log_file)


@app.command()
def sync(ctx: typer.Context, fail: bool = False) -> None:
    """Sync items (use --fail to simulate an error)."""
    log.debug("starting sync with 3 items")
    for i in range(3):
        log.debug("item %d processed", i)
    if fail:
        log.error("sync failed: remote closed the connection")
        if ctx.obj:
            typer.echo(f"details: {ctx.obj}", err=True)
        raise typer.Exit(1)
    log.info("synced 3 items")


if __name__ == "__main__":
    app()

The details

RotatingFileHandler(maxBytes=..., backupCount=...) renames mytool.log to mytool.log.1 (and shifts older files up) when the file would exceed the size limit, keeping at most backupCount old files. One megabyte and three backups caps disk use at about four megabytes — plenty for recent history, never a disk-filling surprise.

delay=True opens the file only when the first record is written, so commands that log nothing (such as --help) never create or touch the file.

encoding="utf-8" makes the log readable on every platform and immune to the Windows code-page problems described in fixing Unicode and encoding errors on Windows.

The file format carries timestamps and the process ID. The console format is terse for humans; the file format includes when and which run, so interleaved runs can be told apart. For machine-parsed logs, use the JSON formatter from structured JSON logging in Python CLIs on the file handler only.

Failure to set up file logging is not fatal. A read-only home directory, a full disk or a locked-down container must not stop the tool; it logs one warning and continues with console logging only.

Noisy libraries are turned down. At DEBUG, HTTP libraries log every connection detail; capping them at INFO keeps the file useful. (Turn them up with a dedicated --debug-http flag when needed — with secret redaction in place.)

Configuration is idempotent. Removing existing handlers first means calling configure_logging twice — in tests, or from a long-lived process — never duplicates output.

Choosing a rotation strategy

Size-based rotation suits most CLIs: it bounds disk use regardless of how often the tool runs. Time-based rotation (TimedRotatingFileHandler) suits long-running services, where "yesterday's log" is a meaningful unit. One file per run — named with a timestamp and pruned to the last N — suits batch jobs and CI, where each run's log may be uploaded as an artefact.

One caveat applies to all of them: rotation is not coordinated across processes. Two invocations running at once can both decide to rotate, occasionally losing a few lines of the older file. For interactive CLIs that is acceptable; for a service that runs many concurrent workers, log to stderr and let the supervisor (journald, Docker) collect and rotate, as discussed in running a CLI on a schedule with cron and systemd.

UX considerations

• Point to the log on failure. "details: ~/.local/state/mytool/log/mytool.log" turns a one-line error into a complete story the user can attach to a bug report.
• Make the location discoverable. Include the log path in mytool paths or mytool doctor output, and allow --log-file or MYTOOL_LOG_FILE to override it.
• Never write secrets to the file. A persistent log lives longer and travels further than terminal output; attach the redaction filter to the file handler.
• Offer an opt-out. Some environments forbid writing logs to disk; MYTOOL_LOG_FILE= set to an empty value, or --no-log-file, should disable it.

Testing the behaviour

Point the file handler at tmp_path, run commands through CliRunner, and call logging.shutdown() before reading the file so buffered data is flushed and the handle released:

# tests/test_logging.py
import logging

from typer.testing import CliRunner

from mytool.cli import app
from mytool.logsetup import configure_logging

runner = CliRunner()


def test_file_gets_debug_console_stays_quiet(tmp_path):
    log_file = tmp_path / "logs" / "mytool.log"
    result = runner.invoke(app, ["sync"], env={"MYTOOL_LOG_FILE": str(log_file)})
    assert result.exit_code == 0
    assert result.output == ""                             # nothing at WARNING or above
    logging.shutdown()
    text = log_file.read_text(encoding="utf-8")
    assert "item 2 processed" in text and "DEBUG" in text


def test_failure_points_at_the_log(tmp_path):
    log_file = tmp_path / "mytool.log"
    result = runner.invoke(app, ["sync", "--fail"], env={"MYTOOL_LOG_FILE": str(log_file)})
    assert result.exit_code == 1
    assert "sync failed" in result.output and str(log_file) in result.output


def test_rotation_keeps_a_bounded_number_of_files(tmp_path):
    path = tmp_path / "mytool.log"
    configure_logging(0, path, max_bytes=2_000, backups=2)
    logger = logging.getLogger("rot")
    for i in range(500):
        logger.debug("line %04d %s", i, "x" * 40)
    logging.shutdown()
    assert sorted(p.name for p in tmp_path.iterdir()) == ["mytool.log", "mytool.log.1", "mytool.log.2"]


def test_unwritable_location_disables_file_logging(tmp_path):
    blocker = tmp_path / "not-a-dir"
    blocker.write_text("")
    assert configure_logging(0, blocker / "mytool.log") is None

The rotation test writes enough records to force several rotations and asserts that exactly the configured number of backups exists — the property that keeps disks from filling. The last test proves the tool keeps working when the log location is unusable.

Conclusion

A persistent, rotating debug log gives you the detail of -vvv for every run without making the terminal noisy. Log everything at DEBUG to a RotatingFileHandler in the platform log directory, filter the console handler by verbosity, write UTF-8 with timestamps and process IDs, create the file lazily, keep going when the location is unwritable, and point users at the file when something fails. Bounded, discoverable and redacted, it turns hard-to-reproduce bug reports into diagnosable ones.

Frequently asked questions

Related

• Up: Structured logging for CLI apps
• Adding verbose and quiet logging flags
• Structured JSON logging in Python CLIs
• Adding trace IDs and context to CLI logs
• Storing app data with platformdirs