Caching Expensive Work Between Python CLI Runs

A CLI process starts from nothing every time. Whatever it computed on the last run — a parsed project index, a resolved dependency graph, a list of repositories fetched from an API, compiled templates — is gone, and the next invocation does it all again. For a command people run dozens of times an hour, or that shell completion calls on every Tab press, that repeated work is the difference between a tool that feels instant and one that feels sluggish. Caching results on disk between runs fixes it, but a cache that returns a wrong answer is worse than no cache at all. This guide builds a small, safe on-disk cache for a Python CLI: keys that capture everything the result depends on, time-to-live for remote data, atomic writes, an escape hatch for users, and tests that prove invalidation works. It belongs to the CLI startup performance and lazy loading topic.

Prerequisites

• Python 3.10+, and platformdirs for the cache location (see storing app data with platformdirs).
• A measurement showing the work is actually slow. Profiling Python CLI startup time shows how to find where time goes; if imports dominate, lazy loading helps more than caching.

Is it worth caching?

Caching pays when the work is slow (hundreds of milliseconds or more), repeated with the same inputs (the same project, the same API query), and safe to recompute if the cache turns out to be wrong. It is a poor fit for work that is fast already, whose inputs change on every run, or whose correctness is critical and hard to validate. For a CLI, the classic candidates are: indexing or parsing a project tree, fetching slowly changing remote data (a list of teams, regions, available versions), expensive computations whose inputs are files, and anything shell completion needs.

The shape of a trustworthy cache

The one rule that makes a cache trustworthy: everything that can change the answer must be part of the key. For work derived from files, that means the files' contents (or, more cheaply, their paths, sizes and modification times). For every cache, it also means the version of your tool — otherwise an upgrade that changes the computation returns results computed by the old code. For remote data, where you cannot see the inputs, a time-to-live bounds staleness instead.

The recipe

# src/mytool/cache.py
from __future__ import annotations

import hashlib
import json
import os
import tempfile
import time
from collections.abc import Callable
from importlib.metadata import PackageNotFoundError, version
from pathlib import Path
from typing import Any

from platformdirs import user_cache_path

try:
    TOOL_VERSION = version("mytool")
except PackageNotFoundError:
    TOOL_VERSION = "dev"

FORMAT = 1          # bump when the cache entry layout changes


class DiskCache:
    def __init__(self, directory: Path | None = None, enabled: bool = True,
                 clock: Callable[[], float] = time.time) -> None:
        self.dir = directory or user_cache_path("mytool", appauthor=False) / "results"
        self.enabled = enabled and os.environ.get("MYTOOL_NO_CACHE") != "1"
        self.clock = clock

    @staticmethod
    def key(namespace: str, **parts: Any) -> str:
        payload = json.dumps({"ns": namespace, "v": TOOL_VERSION, "fmt": FORMAT, **parts},
                             sort_keys=True, default=str)
        return hashlib.sha256(payload.encode()).hexdigest()

    def get(self, key: str, max_age: float | None = None) -> Any | None:
        if not self.enabled:
            return None
        path = self.dir / f"{key}.json"
        try:
            entry = json.loads(path.read_text(encoding="utf-8"))
        except (FileNotFoundError, json.JSONDecodeError, UnicodeDecodeError):
            return None                                   # missing or corrupt: a miss, never an error
        if max_age is not None and self.clock() - entry["created"] > max_age:
            return None
        return entry["value"]

    def set(self, key: str, value: Any) -> None:
        if not self.enabled:
            return
        self.dir.mkdir(parents=True, exist_ok=True)
        data = json.dumps({"created": self.clock(), "value": value})
        fd, tmp = tempfile.mkstemp(dir=self.dir, suffix=".tmp")
        with os.fdopen(fd, "w", encoding="utf-8") as fh:
            fh.write(data)
        os.replace(tmp, self.dir / f"{key}.json")          # readers never see half an entry

    def clear(self) -> int:
        removed = 0
        for p in self.dir.glob("*.json"):
            p.unlink(missing_ok=True)
            removed += 1
        return removed


def files_fingerprint(paths: list[Path]) -> list[tuple[str, int, int]]:
    """Cheap change detection: path, size and mtime in nanoseconds for each file."""
    fp = []
    for p in sorted(paths):
        st = p.stat()
        fp.append((str(p), st.st_size, st.st_mtime_ns))
    return fp

Using it in a command that indexes a documentation tree:

# src/mytool/cli.py
from pathlib import Path
from typing import Annotated

import typer

from mytool.cache import DiskCache, files_fingerprint

app = typer.Typer()


def build_index(files: list[Path]) -> dict[str, list[str]]:
    """The slow part: parse every file and collect headings."""
    index: dict[str, list[str]] = {}
    for f in files:
        index[str(f)] = [line.lstrip("# ").strip() for line in f.read_text(encoding="utf-8").splitlines()
                         if line.startswith("#")]
    return index


@app.callback()
def main() -> None:
    """Docs tools."""


@app.command()
def headings(
    root: Annotated[Path, typer.Argument(exists=True, file_okay=False)] = Path("docs"),
    no_cache: Annotated[bool, typer.Option("--no-cache", help="Ignore and do not update the cache.")] = False,
) -> None:
    """Print every heading in the docs tree."""
    cache = DiskCache(enabled=not no_cache)
    files = sorted(root.rglob("*.md"))
    key = DiskCache.key("headings", root=str(root.resolve()), files=files_fingerprint(files))
    index = cache.get(key)
    if index is None:
        index = build_index(files)
        cache.set(key, index)
    for path, hs in index.items():
        for h in hs:
            typer.echo(f"{path}: {h}")


@app.command("clear-cache")
def clear_cache() -> None:
    """Delete cached results."""
    typer.echo(f"removed {DiskCache().clear()} cached entries", err=True)


if __name__ == "__main__":
    app()

The decisions that make it safe

The key includes the tool version and a format number. Upgrading mytool changes the key, so results computed by old code are never returned by new code. FORMAT covers changes to the entry layout itself.

File fingerprints use size and nanosecond mtime. Hashing every file's contents would be exact but may cost as much as the work being cached; size plus modification time detects virtually all real edits cheaply. For inputs where that is not good enough — files that are rewritten with identical timestamps by some tools — hash the contents instead.

Corrupt or missing entries are misses. A truncated file, a JSON error or a Unicode error means "recompute", never a crash. The cache must never be the reason a command fails.

Writes are atomic. The entry is written to a temporary file and renamed into place, the pattern from writing files atomically in Python CLIs, so two concurrent runs cannot read half an entry.

JSON, not pickle. JSON cannot execute code when loaded and survives Python upgrades. Pickle is faster for complex objects but turns a writable cache directory into a code-execution risk and breaks across versions.

Remote data gets a TTL. For results you cannot fingerprint — an API listing — pass max_age to get: cache.get(key, max_age=600) accepts entries up to ten minutes old. Choose the TTL by how stale an answer users can tolerate.

UX considerations

• Always offer --no-cache (and an environment variable for scripts). When a user suspects a stale result, bypassing the cache must be one flag away.
• Offer a way to clear it, and mention in help that the cache is safe to delete. Keep it in the platform cache directory so system tools and users recognise it as disposable.
• Keep it invisible when it works. Do not announce cache hits in normal output; print them only with -v, where they help explain timing.
• Refresh remote data explicitly. A --refresh flag that ignores the TTL and rewrites the entry is friendlier than making users clear the whole cache.
• Mind shell completion. Completion callbacks benefit most from caching, because they run on every Tab press, but they must stay fast even on a miss — fall back to no suggestions rather than blocking on a slow computation.

Testing the behaviour

The properties to test are hits, invalidation by input change, invalidation by version, TTL expiry and resilience to corruption. An injectable clock and a temporary cache directory make each one deterministic:

# tests/test_cache.py
from pathlib import Path

from mytool import cache as cache_mod
from mytool.cache import DiskCache, files_fingerprint


def test_hit_and_miss(tmp_path):
    c = DiskCache(tmp_path)
    k = DiskCache.key("t", x=1)
    assert c.get(k) is None
    c.set(k, {"answer": 42})
    assert c.get(k) == {"answer": 42}


def test_file_change_changes_the_key(tmp_path):
    f = tmp_path / "a.md"
    f.write_text("# One\n")
    k1 = DiskCache.key("idx", files=files_fingerprint([f]))
    f.write_text("# One\n# Two\n")
    k2 = DiskCache.key("idx", files=files_fingerprint([f]))
    assert k1 != k2


def test_tool_version_is_part_of_the_key(monkeypatch):
    k1 = DiskCache.key("t", x=1)
    monkeypatch.setattr(cache_mod, "TOOL_VERSION", "99.0")
    assert DiskCache.key("t", x=1) != k1


def test_ttl_expiry(tmp_path):
    now = [1000.0]
    c = DiskCache(tmp_path, clock=lambda: now[0])
    c.set("k", "v")
    now[0] += 599
    assert c.get("k", max_age=600) == "v"
    now[0] += 2
    assert c.get("k", max_age=600) is None


def test_corrupt_entry_is_a_miss(tmp_path):
    c = DiskCache(tmp_path)
    (tmp_path / "k.json").write_text("{not json")
    assert c.get("k") is None


def test_disabled_by_env(tmp_path, monkeypatch):
    monkeypatch.setenv("MYTOOL_NO_CACHE", "1")
    c = DiskCache(tmp_path)
    c.set("k", "v")
    assert c.get("k") is None and not list(Path(tmp_path).glob("*.json"))

A command-level test can additionally patch build_index with a counter and assert it runs once across two invocations with unchanged files, and twice after a file changes — the behaviour users actually experience.

Conclusion

An on-disk cache can make a repeated CLI command feel instant, provided it can be trusted. Put everything that affects the result into the key — input fingerprints, the tool version, a format number — bound remote data with a TTL, treat corruption as a miss, write atomically, store JSON rather than pickle, keep entries in the platform cache directory, and give users --no-cache and a way to clear it. Test hits, every form of invalidation and corruption with an injected clock, and the cache becomes a pure speed-up with no new ways to be wrong.

Frequently asked questions

Related

• Up: CLI startup performance and lazy loading
• Profiling Python CLI startup time
• Lazy-loading subcommands for faster startup
• Storing app data with platformdirs
• Dynamic completion values from APIs and files