Walking Directory Trees with Ignore Rules in a Python CLI

Linters, formatters, sync tools, code generators, search tools — a large share of CLIs start by walking a directory tree. And every one of them meets the same expectations from users: do not descend into .git, .venv or node_modules; respect .gitignore; let me add my own exclusions; and do it fast even in a repository with two hundred thousand files. Path.rglob("*") gets none of that right: it visits everything, including the virtual environment with its tens of thousands of files, and filtering afterwards wastes all that work. This guide builds a walker that prunes ignored directories before entering them, understands gitignore syntax — including negation — through the pathspec library, supports a tool-specific ignore file and command-line excludes, and offers git ls-files as an exact alternative inside repositories. It belongs to the filesystem topic.

Prerequisites

• Python 3.10+ and pathspec (uv add pathspec), the gitignore-matching library used by Black and many other tools.
• Familiarity with cross-platform paths with pathlib.

Prune, do not filter

The single most important performance decision is when ignore rules apply. Filtering after a full traversal still reads every directory under node_modules. os.walk lets you edit the list of subdirectories in place before it descends, so an ignored directory is never opened at all — in a typical JavaScript-adjacent Python repository, that is the difference between a walk that takes seconds and one that takes milliseconds.

The recipe

# src/mytool/walk.py
from __future__ import annotations

import os
from collections.abc import Iterator, Sequence
from pathlib import Path

import pathspec

ALWAYS_SKIP = [".git/", "__pycache__/", ".venv/", "node_modules/"]


def load_spec(root: Path, extra: Sequence[str] = (),
              ignore_files: Sequence[str] = (".gitignore", ".mytoolignore")) -> pathspec.GitIgnoreSpec:
    """Combine built-in skips, ignore files at the root, and --exclude patterns."""
    lines = list(ALWAYS_SKIP)
    for name in ignore_files:
        f = root / name
        if f.is_file():
            lines += f.read_text(encoding="utf-8").splitlines()
    lines += list(extra)                       # command-line excludes come last and win
    return pathspec.GitIgnoreSpec.from_lines(lines)


def walk_files(root: Path, spec: pathspec.GitIgnoreSpec, *,
               follow_symlinks: bool = False) -> Iterator[Path]:
    root = root.resolve()
    for dirpath, dirnames, filenames in os.walk(root, followlinks=follow_symlinks):
        rel_dir = Path(dirpath).relative_to(root)
        # Prune in place: os.walk never descends into directories removed here.
        dirnames[:] = sorted(d for d in dirnames
                             if not spec.match_file(f"{(rel_dir / d).as_posix()}/"))
        for name in sorted(filenames):
            rel = (rel_dir / name).as_posix()
            if not spec.match_file(rel):
                yield root / rel

A few details are easy to get wrong:

• Directories are matched with a trailing slash. A gitignore pattern such as build/ matches only directories; pathspec knows a path is a directory by the slash, so the pruning check appends one. Without it, build/ would never match and the walker would descend into every build directory.
• Paths are relative and POSIX-style. Gitignore patterns are written with forward slashes relative to the root, so matching uses as_posix() relative paths on every platform, including Windows.
• Order matters for negation. Gitignore semantics are "last matching pattern wins", so !logs/keep.log re-includes a file excluded by logs/*.log above it. Putting command-line --exclude patterns last lets them override the ignore files.
• Sorting makes output deterministic, which keeps tests and diffs stable regardless of filesystem order.

Given a tree with a .gitignore of build/, *.pyc, logs/*.log and !logs/keep.log, the walker yields exactly .gitignore, README.md, logs/keep.log and src/pkg/a.py — never entering .git, build or node_modules.

Wiring it into a command

# src/mytool/cli.py
from pathlib import Path
from typing import Annotated

import typer

from mytool.walk import load_spec, walk_files

app = typer.Typer()


@app.callback()
def main() -> None:
    """File tool."""


@app.command()
def files(
    root: Annotated[Path, typer.Argument(exists=True, file_okay=False)] = Path("."),
    exclude: Annotated[list[str], typer.Option("--exclude", "-x", help="Gitignore-style pattern; repeatable.")] = [],
    no_ignore: Annotated[bool, typer.Option("--no-ignore", help="Ignore .gitignore and .mytoolignore.")] = False,
) -> None:
    """List the files mytool would process."""
    spec = load_spec(root, exclude, ignore_files=() if no_ignore else (".gitignore", ".mytoolignore"))
    for path in walk_files(root, spec):
        typer.echo(path.relative_to(root.resolve()).as_posix())

A files (or --list-files) command that shows exactly what the tool will process is the best debugging aid you can give users of an ignore-aware CLI; Ruff, Black and many others offer one.

Inside a git repository: ask git

pathspec reads the root .gitignore, but git's real rules are richer: nested .gitignore files in subdirectories, .git/info/exclude, the user's global excludes file, and files that are tracked despite matching an ignore pattern. When the root is a git repository and you want git's exact view, ask git:

import subprocess
from pathlib import Path


def git_files(root: Path) -> list[Path] | None:
    """Tracked plus untracked-but-not-ignored files, or None if not a git repo."""
    result = subprocess.run(
        ["git", "ls-files", "--cached", "--others", "--exclude-standard", "-z"],
        cwd=root, capture_output=True,
    )
    if result.returncode != 0:
        return None
    return [root / p for p in result.stdout.decode("utf-8").split("\0") if p]

-z separates names with NUL bytes so file names containing newlines or unusual characters survive intact. Fall back to the pathspec walker when git is missing or the directory is not a repository, and apply your tool-specific ignore file on top in both cases. The safe-subprocess practices from calling external commands safely with subprocess apply.

Choosing the root

Ignore files are interpreted relative to the directory that contains them, so the walker needs to know where the project root is — not just where the user happens to be. Running mytool files src/ from the repository root should still apply the root .gitignore, with patterns matched against paths like src/pkg/a.py, not pkg/a.py. The usual solution is to find the root first by walking up from the target to the nearest directory containing .git or the tool's config file, as described in discovering project config files by walking up directories, load the spec there, and then walk only the requested subtree while computing relative paths from the root. Getting this right is what makes mytool check src/ and mytool check . agree about which files under src/ are ignored — a mismatch users notice immediately and find very confusing.

UX considerations

• Give users a way to see the decision. A files command or --verbose output listing skipped directories ends most "why was my file ignored?" questions.
• Offer --no-ignore for the occasional need to process everything, as ripgrep does.
• Use a tool-specific ignore file (.mytoolignore) for exclusions that are about your tool and not about git — generated fixtures, vendored code you do not want linted.
• Do not follow symlinks by default. Following them can escape the root or loop forever; make it an explicit option.
• Report unreadable directories, do not crash. os.walk takes an onerror callback; log a warning and continue rather than aborting a walk over a permission error.

Testing the behaviour

Build small trees in tmp_path and assert on the exact list of files — including that ignored directories were never entered:

# tests/test_walk.py
import os
from pathlib import Path

from mytool.walk import load_spec, walk_files


def make_tree(root: Path) -> None:
    for rel in ["src/pkg/a.py", "src/pkg/b.pyc", "build/out.o", ".git/HEAD",
                "logs/app.log", "logs/keep.log", "node_modules/x/i.js", "README.md"]:
        path = root / rel
        path.parent.mkdir(parents=True, exist_ok=True)
        path.write_text("")
    (root / ".gitignore").write_text("build/\n*.pyc\nlogs/*.log\n!logs/keep.log\n")


def listed(root: Path, *extra: str) -> list[str]:
    spec = load_spec(root, extra)
    return [p.relative_to(root.resolve()).as_posix() for p in walk_files(root, spec)]


def test_gitignore_rules_and_negation(tmp_path):
    make_tree(tmp_path)
    assert listed(tmp_path) == [".gitignore", "README.md", "logs/keep.log", "src/pkg/a.py"]


def test_command_line_excludes_apply_last(tmp_path):
    make_tree(tmp_path)
    assert listed(tmp_path, "src/") == [".gitignore", "README.md", "logs/keep.log"]


def test_ignored_directories_are_never_entered(tmp_path, monkeypatch):
    make_tree(tmp_path)
    visited = []
    real_walk = os.walk

    def spy(top, *args, **kwargs):
        for entry in real_walk(top, *args, **kwargs):
            visited.append(Path(entry[0]).name)
            yield entry

    monkeypatch.setattr("mytool.walk.os.walk", spy)
    listed(tmp_path)
    assert "node_modules" not in visited and "build" not in visited

The last test is the performance guarantee in executable form: it fails if someone "simplifies" the walker to filter after traversal.

Conclusion

Walking a tree well means respecting the same rules users already wrote for git, and paying nothing for directories they excluded. Prune ignored directories in place during os.walk, match with pathspec's gitignore semantics using POSIX-style relative paths and a trailing slash for directories, layer built-in skips, ignore files and command-line excludes in that order, and use git ls-files when you need git's exact view. Show users what will be processed, and test both the file list and the pruning.

Frequently asked questions

Related

• Up: Filesystem paths and atomic writes
• Cross-platform paths with pathlib
• Handling file permissions and umask in CLIs
• Storing CLI state in SQLite
• Wrapping git and other tools from a Python CLI