Filesystem Paths and Atomic Writes for CLIs

A command-line tool is, more often than not, a program that reads some files and writes some others. That makes the filesystem the place where a CLI's bugs do the most lasting damage. A crash in the middle of a write leaves a config file empty. A path built with string concatenation works on the author's Mac and breaks on a colleague's Windows laptop. Two cron jobs running the same command at once silently lose each other's updates. A tool that drops .mytool-cache into whatever directory it was run from leaves litter across every project on the machine. None of these show up in a quick manual test, and all of them show up eventually in real use.

This topic covers the file-handling habits that separate a dependable CLI from a script: writing atomically, using pathlib as the one path API, keeping config, cache and state in the right per-user directories, creating temporary files safely, and locking when concurrent runs are possible. It sits in the CLI Runtime & Systems Integration section alongside running subprocesses and secrets and credentials, which both lean on the patterns here.

TL;DR

• Never overwrite a file in place. Write to a temporary file in the same directory, fsync it, then os.replace() it over the target. Readers see the old file or the new one, never a torn one.
• Use pathlib.Path everywhere inside your code, and read and write text with an explicit encoding="utf-8".
• Put your own files in per-user directories from platformdirs: config the user edits, state the tool owns, cache the tool can throw away. Never write into the current directory unless the user asked for output there.
• Create temporary files with tempfile, inside with blocks, so they are unpredictable and always cleaned up.
• Lock when two runs could collide. An OS-level lock (via filelock or fcntl.flock) releases automatically if the process dies.

Five ways file handling goes wrong

It is worth naming the failure modes, because each one maps to a technique in this topic and each one is invisible until it happens to a user.

Torn writes. open(path, "w") truncates the file the moment it opens. If the process then crashes, is killed by a timeout, loses power, or hits a full disk, the file is left empty or half-written. For a user's config or a tool's state file, that often means the next run fails to start at all. The cure is the atomic write pattern.

Platform assumptions. "~/.mytool/" + name, path.split("/")[-1] and os.path.join(root, "a/b") all encode a Unix view of paths. On Windows, ~ is not expanded by open(), separators differ, and drive letters change the rules for absolute paths. Cross-platform paths with pathlib replaces the string manipulation with an API that gets it right everywhere.

Concurrent runs. Cron fires while a previous run is still going; a developer opens two terminals; CI runs a matrix of jobs on one runner. Two processes each read a state file, change it, and write it back — and one update disappears without an error. Atomic writes do not fix this; file locking for concurrent CLI runs does.

Leaked temporary files. A tool that extracts archives or renders documents into /tmp and forgets to clean up after a failure fills disks slowly. One that uses predictable names like /tmp/mytool.out is also exposed to symlink attacks on shared machines. Safe temporary files and directories covers both.

Files in the wrong place. Caches in the working directory, config in the home directory root, logs next to the executable. Each is a small annoyance; together they make a tool feel careless. Storing app data with platformdirs puts each kind of file where the operating system expects it.

Writing files you cannot corrupt

The atomic write is the most valuable single technique in this topic, and it is short enough to show here in full. The idea: create a temporary file in the same directory as the target, write everything to it, force it to disk, then rename it over the target. A rename within one filesystem is atomic — at any instant the path refers to either the complete old file or the complete new one.

import os
import tempfile
from pathlib import Path


def write_atomic(path: Path, data: str, encoding: str = "utf-8") -> None:
    path = Path(path)
    fd, tmp = tempfile.mkstemp(dir=path.parent, prefix=f".{path.name}.", suffix=".tmp")
    try:
        with os.fdopen(fd, "w", encoding=encoding, newline="") as fh:
            fh.write(data)
            fh.flush()
            os.fsync(fh.fileno())
        if path.exists():
            os.chmod(tmp, path.stat().st_mode & 0o7777)
        os.replace(tmp, path)
    except BaseException:
        Path(tmp).unlink(missing_ok=True)
        raise

Three details make it correct rather than approximately correct. The temp file lives in path.parent, because a rename across filesystems (from /tmp to your home directory, say) is really a copy and is not atomic. os.fsync() pushes the data to disk before the rename, so a power loss cannot leave a renamed-but-empty file. And os.replace() — not os.rename() — overwrites the target on Windows as well as POSIX. The full guide adds permission handling, binary data, JSON helpers and tests.

One path type, used consistently

pathlib.Path should be the only representation of a path inside your program. Parse arguments into Path objects at the edge — Typer does this when you annotate a parameter as Path, and Click does it with click.Path(path_type=Path) — and pass them around as Path until something outside Python needs a string.

from pathlib import Path

import typer

app = typer.Typer()


@app.command()
def index(
    root: Path = typer.Argument(Path("."), exists=True, file_okay=False, resolve_path=True),
    out: Path = typer.Option(Path("index.json"), "--out", "-o", dir_okay=False),
) -> None:
    """Index every Markdown file under ROOT."""
    docs = sorted(p.relative_to(root) for p in root.rglob("*.md") if p.is_file())
    out.parent.mkdir(parents=True, exist_ok=True)
    out.write_text("\n".join(d.as_posix() for d in docs) + "\n", encoding="utf-8")
    typer.echo(f"indexed {len(docs)} files -> {out}", err=True)


if __name__ == "__main__":
    app()

Two habits in that snippet are worth adopting everywhere. as_posix() gives a stable, forward-slash representation for output that other programs or other platforms will read, while str(path) gives the native form for display to a local user. And encoding="utf-8" is explicit on every text read and write: until UTF-8 mode becomes the default (PEP 686, planned for Python 3.15), Path.write_text() without it uses the locale encoding, which is still often cp1252 on Windows. Validating file and directory paths in CLIs covers the argument side — existence, permissions and friendly errors — in more depth.

Where your tool's own files belong

A CLI deals with two very different kinds of file. The user's files — the inputs they point it at and the outputs they ask for — live wherever the user says, and your tool should only touch them when asked. Your tool's own files — settings, caches, history, downloaded data, logs — belong in directories the operating system designates for each application, not in the user's projects and not scattered across the home directory.

The platformdirs package knows those directories for Linux (following the XDG base-directory specification), macOS and Windows:

from platformdirs import PlatformDirs

dirs = PlatformDirs("mytool", appauthor=False)
config_file = dirs.user_config_path / "config.toml"
cache_dir = dirs.user_cache_path
state_file = dirs.user_state_path / "last-run.json"

The distinction between the directories is not pedantry. Users back up their config directory and expect to be able to delete caches freely; system cleaners and container images treat them differently. A good test is to ask "if this file vanished, what would be lost?" — nothing means cache, the user's choices means config, and the tool's own memory means state. The platformdirs guide covers environment overrides, --config flags and a paths command, and handling configuration files and environment variables covers what goes inside the config file.

Temporary files, created and removed safely

When a command needs scratch space — extracting an archive, rendering intermediate files, staging a download — use the tempfile module rather than inventing names. TemporaryDirectory() as a context manager gives you a private, randomly named directory that is deleted when the block ends, even if an exception is raised:

import shutil
import subprocess
import tempfile
from pathlib import Path


def build_site(source: Path, dest: Path) -> None:
    with tempfile.TemporaryDirectory(prefix="mytool-build-") as tmp:
        work = Path(tmp)
        subprocess.run(["mkdocs", "build", "-f", str(source / "mkdocs.yml"), "-d", str(work / "site")],
                       check=True)
        if dest.exists():
            shutil.rmtree(dest)
        shutil.copytree(work / "site", dest)

Building into a temporary directory and only copying the result into place once the build succeeds is the directory-level cousin of the atomic write: a failed build never leaves the destination half-updated. Random names created with O_EXCL also close the race in which another user on a shared machine pre-creates your predictable path as a symlink. The temporary files guide covers NamedTemporaryFile on Windows, keeping temp directories for debugging, and cleanup on Ctrl+C.

Two runs, one file

Atomic writes guarantee that every write is complete. They do not guarantee that a write is based on the latest data. If two invocations of your tool read the same state file, both modify it, and both write it back atomically, the file is perfectly well-formed and one of the updates is gone. When your CLI keeps state that concurrent runs update — a download cache index, a queue, a counter, a "last synced" marker — you need a lock around the read-modify-write cycle.

from filelock import FileLock, Timeout

from mytool.paths import dirs

lock = FileLock(dirs.user_state_path / "sync.lock", timeout=30)
try:
    with lock:
        run_sync()  # read state, work, write state atomically
except Timeout:
    raise SystemExit("another sync is running; try again shortly")

filelock uses the operating system's own locking (fcntl.flock on POSIX, msvcrt.locking on Windows), so a lock held by a process that crashes or is killed is released automatically. That is the property hand-rolled "create a .lock file and delete it at the end" schemes lack: one kill -9 and every later run waits for a lock nobody holds. The locking guide covers timeouts, --no-wait, and the difference between locking a resource and preventing a second instance of the whole tool.

Text, bytes and line endings

Three smaller decisions come up every time a CLI writes a file, and making them once, in one helper, saves a steady trickle of bug reports.

Text or bytes. If you are copying, hashing or transforming data you did not create — an uploaded file, an archive member, a download — keep it as bytes (read_bytes(), write_bytes(), open(..., "rb")). Decoding and re-encoding data you do not own risks changing it. Decode only what you actually parse.

Encoding. For text your tool creates, choose UTF-8 explicitly. For text you read from users, UTF-8 is still the right default, but decide what happens on a bad byte: errors="strict" is correct for configuration (fail loudly and name the file), while errors="replace" suits log files and other content you only display. A byte-order mark from a Windows editor is a common surprise in config files; encoding="utf-8-sig" reads files with or without one.

Line endings. In text mode Python translates \n to the platform's line separator on write, so the same code produces \r\n files on Windows. That is usually what a local user wants for files they open in an editor, and usually wrong for files that are committed to git, compared in tests or consumed on other systems. Pass newline="" (or newline="\n") when you need byte-identical output everywhere — the atomic write helper above does exactly that.

Tools that process large inputs should also avoid reading whole files into memory: iterate line by line, or in fixed-size chunks for binary data. Processing large files and NDJSON streams covers streaming input without surprises.

How the pieces combine

In practice these techniques compose into a single small module that the rest of your CLI imports — a paths.py that defines the directories and a files.py that provides write_atomic, read_json and a locked() context manager. Commands then never call open() with "w" directly:

from contextlib import contextmanager
import json
from pathlib import Path
from typing import Any, Iterator

from filelock import FileLock
from platformdirs import PlatformDirs

dirs = PlatformDirs("mytool", appauthor=False, ensure_exists=True)


@contextmanager
def locked_state(name: str) -> Iterator[dict[str, Any]]:
    """Load a JSON state file under a lock and save it atomically on success."""
    path = dirs.user_state_path / f"{name}.json"
    with FileLock(str(path) + ".lock", timeout=30):
        data = json.loads(path.read_text(encoding="utf-8")) if path.exists() else {}
        yield data
        write_atomic(path, json.dumps(data, indent=2, sort_keys=True) + "\n")

A command that uses with locked_state("history") as state: state["runs"] = state.get("runs", 0) + 1 gets correct behaviour under crashes, concurrency and every operating system, without any of those concerns appearing in its own code. If the command raises inside the block, nothing is written and the previous state is kept — a transaction in miniature.

Testing that module is straightforward because everything is a path. Point PlatformDirs at tmp_path in tests (by patching the module-level dirs or setting XDG_* variables on Linux) and assert on file contents; the techniques are in mocking filesystem and network in CLI tests.

Key takeaways

• Treat every write to a file that matters as a transaction: temp file in the same directory, fsync, os.replace.
• Parse paths into pathlib.Path at the edge and keep them as Path objects; always pass encoding="utf-8".
• Keep the user's files and your tool's files separate, and put the latter in platformdirs config, state and cache directories.
• Use tempfile with context managers for scratch space; never invent predictable temp names.
• Add an OS-level lock around read-modify-write cycles that concurrent runs can reach.
• Wrap all of it in one small module so commands never handle raw file writes.

Frequently asked questions

Related

• Up: CLI Runtime & Systems Integration
• Down: Writing files atomically in Python CLIs
• Down: Cross-platform paths with pathlib
• Down: Storing app data with platformdirs
• Down: Safe temporary files and directories
• Down: File locking for concurrent CLI runs
• Sideways: Running subprocesses from Python CLIs
• Sideways: Handling configuration files and environment variables