Redacting Secrets from CLI Output and Logs

A user hits a bug, reruns your CLI with --debug as the issue template asks, and pastes the output into a public ticket. Line 14 is DEBUG httpx: Authorization: Bearer eyJhbGciOi... — a live production token, now indexed by search engines. Nobody did anything wrong: the debug output was designed to be shared, and a library below your code printed a header you never touched. Secrets leak through output far more often than through storage, and the fix cannot rely on every call site being careful. This guide adds a redaction layer to your CLI's logging that masks registered secret values and common token patterns before anything is written, extends it to tracebacks, and tests that it works. It is part of the secrets and credentials topic.

Prerequisites

• Python 3.10+ and the standard logging module, configured once at startup as described in structured logging for CLI apps.
• Secrets read through one place — the resolver from the topic overview or reading secrets from env and files — so they can be registered for masking.

Where to redact

Redaction belongs at the last possible moment: after a log record has been created by any module, including third-party libraries, and before any handler writes it anywhere. In Python's logging model, that is a logging.Filter attached to each handler. A filter on a logger only sees records created by that logger; a filter on the handlers sees everything that reaches the output.

Two complementary techniques make up the filter:

• Known values. Every secret your tool reads is registered with the redactor at the moment it is read. Any occurrence of that exact string in any output becomes ****. This catches your token no matter which library prints it or in which format.
• Patterns. Some secrets you never see as values: tokens embedded in URLs from config, credentials in a library's own error message, headers built by an SDK. Regular expressions for Authorization headers, user:password@ in URLs and well-known token prefixes (GitHub's ghp_, Slack's xoxb-, AWS access key IDs starting AKIA) catch the common cases.

The recipe

# src/mytool/redact.py
from __future__ import annotations

import logging
import re
import threading
from typing import TextIO

MASK = "****"

PATTERNS: list[tuple[re.Pattern[str], str]] = [
    (re.compile(r"(?i)(authorization:\s*(?:bearer|basic|token)\s+)\S+"), r"\1" + MASK),
    (re.compile(r"(?i)\b(bearer\s+)[A-Za-z0-9._~+/-]{8,}=*"), r"\1" + MASK),
    (re.compile(r"(://)[^/\s:@]+:[^/\s@]+@"), r"\1" + MASK + "@"),
    (re.compile(r"\b(ghp|gho|ghs|ghu|github_pat)_[A-Za-z0-9_]{10,}"), r"\1_" + MASK),
    (re.compile(r"\b(xox[abposr])-[A-Za-z0-9-]{10,}"), r"\1-" + MASK),
    (re.compile(r"\bAKIA[0-9A-Z]{16}\b"), "AKIA" + MASK),
    (re.compile(r"(?i)((?:password|passwd|secret|token|api_key)\s*[=:]\s*)[^\s,;&]+"), r"\1" + MASK),
]


class Redactor:
    """Masks registered secret values and known token shapes in text."""

    def __init__(self) -> None:
        self._values: set[str] = set()
        self._lock = threading.Lock()

    def register(self, value: str | None) -> None:
        if value and len(value) >= 6:          # short values would mask ordinary words
            with self._lock:
                self._values.add(value)

    def __call__(self, text: str) -> str:
        with self._lock:
            values = sorted(self._values, key=len, reverse=True)
        for v in values:
            text = text.replace(v, MASK)
        for pattern, repl in PATTERNS:
            text = pattern.sub(repl, text)
        return text


redactor = Redactor()


class RedactingFilter(logging.Filter):
    def filter(self, record: logging.LogRecord) -> bool:
        record.msg = redactor(record.getMessage())
        record.args = None
        if record.exc_info and not record.exc_text:
            record.exc_text = logging.Formatter().formatException(record.exc_info)
        if record.exc_text:
            record.exc_text = redactor(record.exc_text)
        return True


def install(level: int = logging.WARNING, stream: TextIO | None = None) -> None:
    """Configure root logging with redaction on every handler."""
    handler = logging.StreamHandler(stream)       # None means sys.stderr
    handler.addFilter(RedactingFilter())
    handler.setFormatter(logging.Formatter("%(levelname)s %(name)s: %(message)s"))
    root = logging.getLogger()
    root.handlers[:] = [handler]
    root.setLevel(level)

A few implementation details carry weight:

• The message is formatted before masking (record.getMessage() merges msg % args), and args is cleared afterwards. Otherwise a secret passed as an argument — log.debug("token=%s", token) — would be substituted after the filter ran.
• Tracebacks are formatted and masked too. A logged exception's text is rendered into exc_text and redacted, because exception messages routinely include URLs, connection strings and request details.
• Longest values first. If one secret contains another, replacing the longer first avoids leaving a partial secret behind.
• A minimum length. Registering a three-character value would mask ordinary words all over the output. Six characters is a practical floor for real credentials.

Registering secrets happens where they are read. With the resolver from the topic page, one line is enough:

from mytool.redact import redactor


def load_token(...):
    token, source = resolve_token(...)
    redactor.register(token.reveal())
    return token, source

Debug flags and HTTP tracing

The riskiest moment is --debug, because it usually turns on library logging you do not otherwise see — httpx and httpcore log request lines and, at trace level, headers. Wire the flag so redaction is already installed before those loggers are enabled:

import logging

import typer

from mytool.redact import install


@app.callback()
def main(debug: bool = typer.Option(False, "--debug", help="Verbose diagnostic output.")) -> None:
    install(logging.DEBUG if debug else logging.WARNING)
    if debug:
        logging.getLogger("httpx").setLevel(logging.DEBUG)
        logging.getLogger("httpcore").setLevel(logging.DEBUG)

Output that bypasses logging

Not everything goes through logging. Three other paths need attention:

• Uncaught tracebacks. Python's default hook prints straight to stderr. Install a sys.excepthook that formats the traceback, passes it through redactor, and prints it — or better, convert unexpected errors into a one-line message plus a redacted traceback only in --debug mode, as in friendly error messages and tracebacks.
• Rich tracebacks with locals. rich.traceback.install(show_locals=True) prints every local variable in every frame — including tokens. Leave show_locals off in anything users run, or wrap secrets in a type whose repr is masked so the locals display is harmless.
• Your own print and typer.echo. Keep secrets wrapped in a masking type (see the Secret class in the topic overview) and these become safe by default.

UX considerations

• Mask visibly. **** makes it obvious to the reader that something was removed on purpose, which is better than silently dropping text and leaving a confusing line.
• Keep enough context to debug. Masking only the secret — Bearer ****, https://****@registry — leaves the header name and host visible, which is usually what a maintainer needs.
• Tell users debug output is safe to share. If you redact, say so in the help text for --debug. It makes people more willing to provide diagnostics.
• Do not rely on redaction alone. It is a safety net for mistakes, not a licence to log secrets. Code review should still reject log.debug(f"token={token}").
• Redact files too. If your tool writes a log file (see writing rotating log files from a CLI), attach the same filter to that handler.

Testing the behaviour

Leak tests are cheap and high-value. Log through the real configuration and assert the secret never appears in captured output:

# tests/test_redact.py
import io
import logging

import pytest

from mytool.redact import MASK, Redactor, install, redactor


@pytest.fixture
def captured():
    stream = io.StringIO()
    install(logging.DEBUG, stream=stream)   # the real configuration, into a buffer
    return stream.getvalue


def test_registered_value_is_masked_everywhere(captured):
    redactor.register("sk_live_51HxSecretValue")
    log = logging.getLogger("some.library")
    log.debug("connecting with key=%s to %s", "sk_live_51HxSecretValue", "api")
    out = captured()
    assert "sk_live_51HxSecretValue" not in out and MASK in out


@pytest.mark.parametrize("line, leaked", [
    ("Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJhIn0.sig", "eyJhbGci"),
    ("GET https://deploy:hunter2@registry.example.com/v2/", "hunter2"),
    ("using ghp_16C7e42F292c6912E7710c838347Ae178B4a", "16C7e42F"),
    ("password=correct-horse-battery", "correct-horse"),
])
def test_patterns(captured, line, leaked):
    logging.getLogger("x").warning(line)
    assert leaked not in captured()


def test_tracebacks_are_redacted(captured):
    redactor.register("tok_abcdefghijkl")
    try:
        raise RuntimeError("request failed for token tok_abcdefghijkl")
    except RuntimeError:
        logging.getLogger("x").exception("boom")
    assert "tok_abcdefghijkl" not in captured()


def test_short_values_are_not_registered():
    r = Redactor()
    r.register("abc")
    assert r("abc def") == "abc def"

Keep a fixture of realistic secret shapes in the test suite and add to it whenever a new kind of credential enters the tool. The parametrised pattern test is the place for that.

Conclusion

Redaction turns "please be careful what you log" into a property of the system. Register every secret with a redactor when you read it, attach a filter to every logging handler that masks registered values and common token patterns — including in formatted tracebacks — and install it before --debug enables noisy library loggers. Close the other paths with a masking Secret type and a redacting exception hook, and keep a parametrised leak test so it stays fixed.

Frequently asked questions

Related

• Up: Secrets and credentials in Python CLIs
• Reading secrets from env and files
• Structured JSON logging in Python CLIs
• Adding trace IDs and context to CLI logs
• Friendly error messages and tracebacks