Python CLI Toolcraft

Architecture

Snapshot Testing CLI Output

Pin help screens, tables, and generated files with snapshot tests - normalise unstable values, review diffs, and re-record intentional changes safely.

Updated

Some CLI output is documentation: the help screen, a generated config file, a report people read every morning. Asserting on fragments of it misses drift, and asserting on the whole thing by hand is unmaintainable. Snapshot testing records the output once, commits it, and turns any later change into a reviewable diff.

TL;DR

  • Snapshot output that is read by people and must not drift silently; assert directly on everything else.
  • Normalise timestamps, temporary paths, durations and ordering before comparing.
  • Force a fixed terminal width and disable colour so the snapshot is machine-independent.
  • Re-record deliberately (--snapshot-update), review the diff, and commit it with the change that caused it.
  • Keep each snapshot small enough that it fails for exactly one reason.

Prerequisites

A CLI with output worth pinning, pytest, and either syrupy (pip install syrupy) or twenty lines of your own. This guide shows both — the hand-rolled version first, because it makes the mechanism obvious.

When a snapshot is the right tool

Is this output worth snapshotting? A decision diagram for snapshot testing: snapshot output that is documentation and must not drift, assert on specific values for everything else. Is this output worth snapshotting? Would a change to this output surprise a user? Yes — help screens, reports, generated files Snapshot it drift becomes a reviewable diff No — an internal message or a computed value Assert directly a targeted assertion reads better Snapshots earn their keep on output people read; elsewhere they lock in detail nobody cares about.

The test is whether a change to the output would surprise someone. Help screens qualify: people copy commands out of them, and a silently reordered option list makes documentation wrong. A generated pyproject.toml from your init command qualifies. An internal debug line does not, and neither does a computed number — assert result.uploaded == 3 is clearer than any snapshot.

Where snapshots go wrong is scope. One snapshot covering six behaviours fails whenever any of them changes, and the diff no longer tells you which. Prefer several small snapshots over one large one.

A minimal snapshot fixture

# tests/conftest.py
import os
import pytest

@pytest.fixture
def snapshot(request):
    """Compare text against tests/__snapshots__/<test name>.txt."""
    directory = request.path.parent / "__snapshots__"
    directory.mkdir(exist_ok=True)
    path = directory / f"{request.node.name}.txt"

    def compare(actual: str) -> None:
        if os.environ.get("UPDATE_SNAPSHOTS") or not path.exists():
            path.write_text(actual, encoding="utf-8")
            if not os.environ.get("UPDATE_SNAPSHOTS"):
                pytest.skip(f"recorded new snapshot: {path}")
            return
        expected = path.read_text(encoding="utf-8")
        assert actual == expected, f"snapshot mismatch; UPDATE_SNAPSHOTS=1 to re-record ({path})"

    return compare

def test_help_screen(cli, snapshot):
    result = cli.invoke(app, ["--help"], env={"COLUMNS": "80", "NO_COLOR": "1"})

    assert result.exit_code == 0
    snapshot(result.stdout)
How a snapshot test evolves The workflow of snapshot testing: the first run records the output, later runs compare against it, and an intentional change is reviewed and re-recorded. How a snapshot test evolves You Test run Snapshot file Review first run: nothing recorded yet write the captured output later runs: diff against the file intentional change: re-record and commit The snapshot lives in the repository, so a change to help output shows up as a reviewable diff.

Two behaviours make this safe to live with. A missing snapshot is recorded and the test is skipped, rather than silently passing — so a new snapshot is never mistaken for a passing assertion. And re-recording requires an explicit environment variable, so a stale snapshot cannot be updated by accident during an ordinary run.

With syrupy the same test is shorter and the mechanics are identical:

def test_help_screen(cli, snapshot):
    result = cli.invoke(app, ["--help"], env={"COLUMNS": "80", "NO_COLOR": "1"})
    assert result.stdout == snapshot

pytest --snapshot-update re-records, and unused snapshots are reported so deleted tests do not leave orphans behind.

Normalising unstable output

A snapshot is only useful if the same input produces the same bytes on every machine. Four things routinely break that.

Making output stable enough to snapshot Values that must be normalised before comparing command output against a stored snapshot, and the practices that make snapshots brittle. Making output stable enough to snapshot Normalise first ✓ Replace timestamps and durations with a fixed token ✓ Replace temporary paths with a placeholder ✓ Sort anything whose order is not guaranteed ✓ Force a fixed terminal width and disable colour Guarantees a flaky test ✗ Snapshotting output that contains a wall-clock time ✗ Letting the terminal width vary between machines ✗ Recording a snapshot from a failing run ✗ One giant snapshot covering six behaviours at once A snapshot is only useful while it fails for exactly one reason. 
import re

SUBSTITUTIONS = [
    (re.compile(r"\d{4}-\d{2}-\d{2}T[\d:.]+Z?"), "<TIMESTAMP>"),
    (re.compile(r"in \d+\.\d+s"), "in <DURATION>"),
    (re.compile(r"/tmp/[^\s\"']+"), "<TMPDIR>"),
    (re.compile(r"mytool \d+\.\d+\.\d+"), "mytool <VERSION>"),
]

def normalise(text: str) -> str:
    for pattern, replacement in SUBSTITUTIONS:
        text = pattern.sub(replacement, text)
    return text.replace("\r\n", "\n").rstrip() + "\n"

The version substitution is the one people forget: without it, every release breaks every snapshot that includes the help footer.

Terminal width is the other frequent culprit. Click and Rich both size output to the terminal, and the runner inherits whatever COLUMNS happens to be. Pin it:

result = cli.invoke(app, ["status"], env={"COLUMNS": "100", "NO_COLOR": "1", "TERM": "dumb"})

Ordering matters too. If your command prints a set, or iterates over a dictionary populated from the filesystem, sort before printing — not in the test, but in the command. Output that is stable for tests is output that is stable for users piping it into diff.

Snapshotting files, not just stdout

The same technique covers generated files, which are often the more valuable target:

def test_init_generates_a_project(cli, snapshot, tmp_path):
    result = cli.invoke(app, ["init", "--name", "demo", "--dir", str(tmp_path)])
    assert result.exit_code == 0

    generated = sorted(p.relative_to(tmp_path) for p in tmp_path.rglob("*") if p.is_file())
    snapshot("\n".join(str(p) for p in generated))            # the file list
    snapshot_file(tmp_path / "pyproject.toml")                # and one file's contents

Snapshotting the file list separately from the contents is worth doing: adding a file and changing a file are different reviews, and separate snapshots keep the diffs meaningful.

UX considerations

Snapshots have a social cost as well as a technical one. Two habits keep them from becoming noise nobody reads.

Re-record in the commit that caused the change. A pull request that renames a flag should contain both the code change and the updated snapshot, so the reviewer sees the user-visible effect next to the cause. Updating snapshots in a separate "fix tests" commit throws that away.

Never re-record to make a failure go away. The failure is the feature. If a snapshot changed and nobody intended it, the code changed in a way nobody intended — which is exactly what the snapshot was for.

It is worth putting both of those in a comment at the top of the snapshot directory's README, or in the failure message itself, because the temptation to run --snapshot-update on a red build is strong.

Testing the behaviour

Snapshots complement targeted assertions rather than replacing them. A good pairing:

def test_status_table(cli, snapshot):
    result = cli.invoke(app, ["status"], env={"COLUMNS": "100", "NO_COLOR": "1"})

    assert result.exit_code == 0                  # behaviour: it worked
    assert "prod" in result.stdout                # behaviour: the important row is present
    snapshot(normalise(result.stdout))            # presentation: nothing else drifted

The first two assertions fail with a clear message when the feature breaks. The snapshot fails when the presentation changes, which is a different signal and often a different fix.

Conclusion

Snapshot tests are the cheapest way to keep output that people depend on from drifting. Normalise the unstable parts, keep each snapshot narrow, record intentionally, and treat an unexpected diff as the finding it is. For output nobody reads, an ordinary assertion is still the better tool.

Frequently asked questions

Should snapshots be committed to the repository?

Yes — that is the whole point. A snapshot in version control means a change to output appears in the diff of the commit that caused it, and reviewers can see the user-visible effect without running anything.

How do I stop snapshots from becoming a wall of unreviewed diffs?

Keep them small and few. One snapshot per meaningful output, not one per test; normalise aggressively so unrelated changes do not touch them; and delete snapshots for tests you removed. syrupy reports orphaned snapshots, which makes the last part automatic.

What about colour codes in the output?

Disable them in tests with NO_COLOR=1 and a fixed TERM. Snapshotting escape sequences produces files nobody can read in a diff, and the colour is rarely the property you meant to pin. If colour itself matters, assert on it separately with a targeted test.

Can I snapshot JSON output?

Yes, and it is a good use of the technique — but parse and re-serialise with sorted keys before comparing, so key ordering cannot cause spurious diffs. Better still, assert on the parsed structure for the fields you care about and snapshot the whole document for drift.

Does this work for Windows?

With one normalisation: replace \r\n with \n before comparing, and use forward slashes in any path you include. Without that, a snapshot recorded on Linux fails on Windows for reasons that have nothing to do with your code.

What is the right size for a snapshot?

Small enough that a failure names one thing. A single command's output at a fixed width is a good unit; the combined output of six commands is not. If a snapshot regularly changes for reasons the test is not about, split it or normalise harder.

Should snapshot tests run in CI, or only locally?

In CI, and without the update flag available — a snapshot that can be re-recorded by the build is not a check. Set the update mechanism behind an environment variable that CI never sets, so the only way to change a snapshot is a deliberate local run and a commit.

How do snapshots interact with parametrised tests?

Each parameter set needs its own snapshot file, keyed on the test id. Both the fixture shown above and syrupy handle this automatically because the node name includes the parameters — which is also why parameter values should be short and readable rather than whole objects.

Can a snapshot replace documentation of the output format?

Not on its own, but it pairs well with it. A snapshot proves the format has not changed; the documentation says what the fields mean and which of them are guaranteed. Together they turn "we think this is stable" into something a reviewer can verify in a diff.