Safe Temporary Files and Directories in CLIs

Many CLI commands need somewhere to work: extracting an archive before validating it, rendering intermediate files for a document build, staging a download before moving it into place, or handing a generated config file to a child program. The quick approach — open("/tmp/mytool.out", "w") and a os.remove at the end — leaks files whenever the command fails, collides when two runs overlap, breaks on Windows, and on a shared machine lets another user redirect your writes with a symlink. This guide covers the tempfile module as a CLI should use it: which function to reach for, how to guarantee cleanup even on Ctrl+C, the Windows quirk with named temporary files, and a --keep-temp flag for debugging. It is part of the filesystem topic.

Prerequisites

• Python 3.12+ for NamedTemporaryFile(delete_on_close=False); earlier versions are covered with an alternative.
• A Typer or Click CLI.
• Some command that needs scratch space. The running example renders a set of Markdown chapters with an external tool.

Choosing the right tool

The tempfile module has two tiers. The high-level context managers create and clean up in one construct; the low-level functions create securely and leave cleanup to you.

For most CLI work, TemporaryDirectory is the right default. A directory lets you create as many files as you need with meaningful names, pass the directory to a child process, and remove everything with one cleanup — which the context manager guarantees, whether the block exits normally or with an exception.

mkstemp() is the right choice when the temporary file will be renamed into place as the final step of an atomic write, because you must control exactly when it is removed (never, on success). That pattern has its own guide: writing files atomically in Python CLIs.

Why not just pick a name?

A fixed or guessable path in a shared directory is a race. Between checking whether /tmp/mytool.out exists and opening it, anyone else on the machine can create it — as a symlink to a file of yours. Your tool then writes through the link and overwrites that file with your privileges.

Every tempfile function avoids this by generating a random name and creating the file with O_CREAT | O_EXCL in a single system call, which fails rather than following an existing path. Files are created readable only by you (0600), and TemporaryDirectory/mkdtemp create directories as 0700. You get those guarantees simply by never constructing temporary paths yourself.

The recipe

Here is a render command that builds every chapter in a private scratch directory and only copies results to the destination once everything has succeeded. It supports --keep-temp for debugging and cleans up on Ctrl+C.

# src/mytool/render.py
from __future__ import annotations

import shutil
import subprocess
import tempfile
from collections.abc import Iterator
from contextlib import contextmanager
from pathlib import Path

import typer


@contextmanager
def workdir(prefix: str, keep: bool = False) -> Iterator[Path]:
    """A private scratch directory, removed on exit unless keep=True."""
    path = Path(tempfile.mkdtemp(prefix=prefix))
    try:
        yield path
    except BaseException:
        if keep:
            typer.echo(f"work directory kept at {path}", err=True)
        raise
    finally:
        if not keep:
            shutil.rmtree(path, ignore_errors=True)


def render_book(chapters: list[Path], dest: Path, keep_temp: bool = False) -> int:
    with workdir(prefix="mytool-render-", keep=keep_temp) as work:
        built = work / "out"
        built.mkdir()
        for chapter in chapters:
            target = built / chapter.with_suffix(".html").name
            subprocess.run(
                ["pandoc", "--standalone", "-o", str(target), "--", str(chapter)],
                check=True, cwd=work, stdin=subprocess.DEVNULL,
            )
        # Everything built: publish in one step.
        staging = dest.with_name(f".{dest.name}.staging")
        shutil.rmtree(staging, ignore_errors=True)
        shutil.copytree(built, staging)
        shutil.rmtree(dest, ignore_errors=True)
        staging.rename(dest)
        if keep_temp:
            typer.echo(f"work directory kept at {work}", err=True)
        return len(chapters)

# src/mytool/cli.py
from pathlib import Path

import typer

from mytool.render import render_book

app = typer.Typer()


@app.callback()
def main() -> None:
    """Book tools."""


@app.command()
def render(
    source: Path = typer.Argument(..., exists=True, file_okay=False),
    dest: Path = typer.Option(Path("site"), "--out", "-o"),
    keep_temp: bool = typer.Option(False, "--keep-temp", help="Leave the work directory for inspection."),
) -> None:
    """Render every chapter in SOURCE to HTML."""
    chapters = sorted(source.glob("*.md"))
    if not chapters:
        typer.echo(f"no .md files in {source}", err=True)
        raise typer.Exit(1)
    n = render_book(chapters, dest.resolve(), keep_temp=keep_temp)
    typer.echo(f"rendered {n} chapters into {dest}", err=True)


if __name__ == "__main__":
    app()

Why a hand-written workdir() rather than TemporaryDirectory directly? Only because of --keep-temp. TemporaryDirectory(delete=False) exists from Python 3.12, but the custom manager also prints the location when a failure occurs, which is exactly when the user wants to look inside. Without that flag, with tempfile.TemporaryDirectory(prefix="mytool-render-") as tmp: is all you need.

Ctrl+C and signals

KeyboardInterrupt is an exception, so finally blocks run and the directory is removed when a user presses Ctrl+C. SIGTERM is different: by default Python dies immediately without running finally. If your CLI runs under a supervisor, CI system or timeout, install a handler that converts SIGTERM into an exception so the same cleanup runs — the pattern in handling SIGTERM and graceful shutdown. SIGKILL cannot be handled at all; the operating system's periodic cleaning of the temp directory is the backstop.

Handing a temporary file to another program

Sometimes a child program needs a file path: a generated config for ssh -F, a list of files for tar -T. NamedTemporaryFile gives you a real path, but on Windows a file opened with delete-on-close cannot be opened a second time by another process. Python 3.12 added a parameter that solves it: the file is deleted when the context manager exits rather than when it is closed.

import subprocess
import tempfile


def run_with_config(config_text: str) -> None:
    with tempfile.NamedTemporaryFile("w", suffix=".conf", encoding="utf-8",
                                     delete_on_close=False) as fh:
        fh.write(config_text)
        fh.close()                       # flush and release it for the child
        subprocess.run(["ssh", "-F", fh.name, "build-host", "true"], check=True)
    # deleted here, on every platform

On Python 3.11 and earlier, create the file inside a TemporaryDirectory instead — writing Path(tmp) / "ssh.conf" — which works everywhere and is cleaned up with the directory.

UX considerations

• Clean up by default, keep on request. A --keep-temp (or --debug) flag that preserves the work directory turns an opaque failure into something the user can inspect — and prints the path so they do not have to hunt for it.
• Use a recognisable prefix. mytool-render-8f2k1c in /tmp tells an administrator which program left it. Anonymous tmpab12cd directories get deleted with suspicion or not at all.
• Respect TMPDIR. tempfile already reads TMPDIR (and TEMP/TMP on Windows). Users with a small /tmp or a RAM disk rely on that; never hard-code /tmp.
• Publish results in one step. Building in scratch space and renaming into place means users never see a half-built output directory. For single files, that is the atomic write; for directories, it is the staging-and-rename shown above.
• Watch the size. If a command can fill gigabytes of scratch space, check free space up front with shutil.disk_usage(tempfile.gettempdir()) and fail with a clear message rather than a No space left on device traceback halfway through.

Testing the behaviour

The properties to test are: nothing is left behind on success, nothing is left behind on failure, and --keep-temp does keep it. Redirect the temp directory to tmp_path so the test can see exactly what was created:

# tests/test_workdir.py
import tempfile

import pytest

from mytool.render import workdir


@pytest.fixture
def scratch(tmp_path, monkeypatch):
    monkeypatch.setattr(tempfile, "tempdir", str(tmp_path))
    return tmp_path


def test_removed_on_success(scratch):
    with workdir("t-") as work:
        (work / "a.txt").write_text("x")
    assert list(scratch.iterdir()) == []


def test_removed_on_error(scratch):
    with pytest.raises(RuntimeError):
        with workdir("t-"):
            raise RuntimeError("boom")
    assert list(scratch.iterdir()) == []


def test_removed_on_ctrl_c(scratch):
    with pytest.raises(KeyboardInterrupt):
        with workdir("t-"):
            raise KeyboardInterrupt
    assert list(scratch.iterdir()) == []


def test_kept_on_request(scratch, capsys):
    with pytest.raises(RuntimeError):
        with workdir("t-", keep=True) as work:
            raise RuntimeError("boom")
    assert work.exists()
    assert str(work) in capsys.readouterr().err

Setting tempfile.tempdir is the documented override for the default location and affects every tempfile function, which makes it more robust than patching individual calls. The broader approach to isolating tests from the real filesystem is in mocking filesystem and network in CLI tests.

Conclusion

Temporary files are easy to get almost right. Let tempfile choose the names so there is nothing to race, wrap scratch space in a context manager so cleanup survives exceptions and Ctrl+C, convert SIGTERM into an exception when running under supervisors, use delete_on_close=False or a temporary directory when a child needs the path, and give users --keep-temp for the day something goes wrong. Build in scratch space and publish in one step, and a failed command will never leave a half-finished result behind.

Frequently asked questions

Related

• Up: Filesystem paths and atomic writes
• Writing files atomically in Python CLIs
• File locking for concurrent CLI runs
• Handling KeyboardInterrupt cleanly
• Calling external commands safely with subprocess