Long-Running and Watch-Mode Python CLIs

Most CLI commands run for a second and exit. Some do not. A build --watch that rebuilds every time a file changes. A sync command that runs every night from cron. A queue consumer or log shipper packaged as a CLI subcommand and run under systemd, Docker or Kubernetes. A tail-like command that follows an event stream until someone stops it. These long-running commands face problems short ones never meet: they are stopped by signals rather than by finishing, they run without a person watching, they are started by machines with minimal environments, and nobody notices when they quietly stop doing their job.

This topic covers building those commands properly in Python: shutting down gracefully when asked, watching files without rebuilding in loops, running reliably on a schedule, and exposing health so that something notices when they stall. It completes the CLI Runtime & Systems Integration section, and it leans on the others — file locking to prevent overlapping runs, cancellation for async loops, and secrets from the environment for unattended runs.

TL;DR

• Handle SIGTERM like Ctrl+C. Supervisors stop you with SIGTERM, and Python's default is to die without running finally blocks. Install a handler that sets a stop event.
• Structure the loop around one bounded unit of work, and make every wait wake up on the stop event.
• Debounce and filter file events in watch mode, ignore your own outputs, and keep watching after a failed rebuild.
• Make scheduled commands non-interactive, idempotent and overlap-safe, with absolute paths and meaningful exit codes. Prefer systemd timers where available.
• Emit a heartbeat after each unit of work and provide a health command so probes and monitors can tell "running" from "working".

Three shapes of long-running command

It helps to recognise which shape you are building, because each has a different priority:

A watch loop is started and stopped by a developer at a terminal; responsiveness and readable output matter most. A supervised service is started and stopped by systemd, a container runtime or Kubernetes, which communicate through signals and expect a timely exit. A scheduled job runs to completion on a timer, unattended; it must be safe to repeat, must never overlap with itself, and must fail loudly enough that someone notices. Many real tools are all three at once — the same sync command is run by hand, in --watch mode during development and nightly from a timer — so it is worth designing the core loop once, correctly.

The shape of a good loop

Almost every long-running command is a loop: wait for something to do, do it, repeat. The details of that loop determine how the command behaves when it is stopped, when it fails and when it stalls.

# src/mytool/loop.py
from __future__ import annotations

import logging
import signal
import threading
import time
from collections.abc import Callable
from pathlib import Path

log = logging.getLogger(__name__)


class Stopper:
    """Turns SIGINT/SIGTERM into a stop event the loop can wait on."""

    def __init__(self) -> None:
        self.event = threading.Event()
        self.signum: int | None = None

    def install(self) -> None:
        for sig in (signal.SIGINT, signal.SIGTERM):
            signal.signal(sig, self._handle)

    def _handle(self, signum: int, frame: object) -> None:
        if self.event.is_set():                    # second signal: stop waiting politely
            raise KeyboardInterrupt
        self.signum = signum
        self.event.set()

    @property
    def exit_code(self) -> int:
        return 128 + self.signum if self.signum else 0


def run_forever(unit: Callable[[], int], *, interval: float, stopper: Stopper,
                heartbeat: Path | None = None) -> int:
    """Call `unit` every `interval` seconds until stopped; return the exit code."""
    while not stopper.event.is_set():
        started = time.monotonic()
        try:
            done = unit()
            log.info("processed %d items", done)
        except Exception:
            log.exception("unit of work failed; will retry next interval")
        if heartbeat is not None:
            heartbeat.touch()
        remaining = interval - (time.monotonic() - started)
        stopper.event.wait(max(0.0, remaining))    # wakes immediately on a signal
    log.info("stopping after signal %s", stopper.signum)
    return stopper.exit_code

Four properties make this loop well-behaved:

• The wait is interruptible. event.wait(timeout) returns the moment the stop event is set, so a command with a five-minute interval still stops within milliseconds. time.sleep(300) would make shutdown wait for the rest of the interval — or be killed by the supervisor first.
• Units of work are bounded. A signal is only noticed between units. If a unit can take ten minutes, the loop cannot honour a thirty-second grace period; split the work, or check the stop event inside it.
• One failure does not end the loop. An unexpected error is logged with its traceback and the loop carries on at the next interval — the right default for a service. For a scheduled one-shot job, the opposite is right: fail and exit non-zero.
• Progress is visible. A heartbeat file is touched after every unit, which is what health checks read.

Being stopped: signals

A command at a terminal is stopped with Ctrl+C, which sends SIGINT and, by default, raises KeyboardInterrupt. A command under a supervisor is stopped with SIGTERM, and Python's default action for SIGTERM is to terminate immediately — no exception, no finally blocks, no flushing of buffered output, no removal of lock files or temporary directories.

The Stopper above treats both signals the same way: it sets an event, the loop finishes its current unit and returns, and the command exits with 128 + signal — 130 for SIGINT, 143 for SIGTERM — the codes supervisors and shells expect. A second signal while shutting down raises KeyboardInterrupt to abandon slow cleanup, the same escape hatch asyncio.run provides. Supervisors escalate to SIGKILL after a grace period — 10 seconds for docker stop, 30 for Kubernetes, 90 for systemd by default — so cleanup must fit well inside the shortest. Handling SIGTERM and graceful shutdown covers the details, including containers where your CLI runs as PID 1.

Watch mode

--watch is one of the most-loved features a developer tool can have: save a file and see the result a fraction of a second later. Implementing it with a polling loop over os.stat is slow and CPU-hungry; implementing it naively with OS file events leads to rebuilding five times per save, or rebuilding forever because each build writes files the watcher then notices. The watchfiles package, built on Rust's notify library, handles the platform-specific event APIs, debounces bursts of events into batches, and supports filters:

from pathlib import Path

from watchfiles import DefaultFilter, watch


class SourceFilter(DefaultFilter):
    def __init__(self, output_dir: Path) -> None:
        super().__init__()
        self.output_dir = output_dir.resolve()

    def __call__(self, change, path: str) -> bool:
        return super().__call__(change, path) and not Path(path).resolve().is_relative_to(self.output_dir)


def watch_and_build(src: Path, out: Path, build) -> None:
    build()
    for changes in watch(src, watch_filter=SourceFilter(out), debounce=200):
        try:
            build()
        except Exception as exc:          # a broken file must not end the watch
            print(f"error: {exc} — still watching")

DefaultFilter already ignores .git, __pycache__, virtual environments and common editor swap files; the subclass adds your own output directory, which prevents the infinite rebuild loop. Building a watch mode with watchfiles develops this into a complete --watch flag with timestamped output, incremental rebuilds and tests.

Running on a schedule

The most common long-running arrangement is not a process that runs forever but one that runs again and again: a nightly sync, an hourly report, a cleanup every Sunday. The scheduler — cron, a systemd timer, a CI schedule, Kubernetes CronJobs — starts your command in an environment that is nothing like your interactive shell: a minimal PATH, no shell profile, no terminal, a different working directory, often a different user. Commands that work perfectly by hand fail there in predictable ways.

Before scheduling a command, it should be:

• Non-interactive: never prompt; detect the missing terminal and fail with a message instead, as in prompting for passwords securely.
• Idempotent: safe to run twice with the same result, because schedulers retry and humans rerun.
• Overlap-safe: if a run takes longer than the interval, the next one must not race it — a lock with --no-wait is the usual answer.
• Explicit about paths: absolute paths or paths from config, never "the current directory".
• Honest in its exit code: non-zero on any failure, so the scheduler, the journal or an alerting wrapper can notice.

systemd timers add journald logging, catch-up of missed runs and built-in protection against overlap, and are preferable to cron where available. Running a CLI on a schedule with cron and systemd has complete, copy-pasteable configurations for both.

Knowing it still works

A long-running process that is alive but stuck — waiting forever on a hung connection, spinning on an error — looks healthy to everything that only checks whether the process exists. A scheduled job that silently stopped being scheduled looks like nothing at all. Both need a signal of progress, not mere existence:

• A heartbeat — a file touched, a timestamp written or a metric updated after every unit of work — shows the loop is making progress. A mytool health --max-age 120 command that checks the heartbeat's age gives Docker HEALTHCHECK, Kubernetes liveness probes and monitoring scripts a simple exit-code API.
• A dead-man switch — an external service pinged at the end of each successful scheduled run, which alerts when a ping does not arrive — catches the failure no in-process check can: the job never running at all.

Health checks and heartbeats for long-running CLIs implements both.

Crashes, restarts and picking up where you left off

A long-running command will be restarted: after a crash, after a deploy, after the machine reboots, after the supervisor decides it is unhealthy. Whether that restart is harmless or costly depends on decisions made in the loop long before anything goes wrong.

Record progress after each unit, not at the end. A command that processes 10,000 records and writes its checkpoint only when it finishes loses everything on a crash at record 9,000. Write the checkpoint — the last processed ID, cursor or timestamp — after each unit, atomically, using the pattern from writing files atomically in Python CLIs. A restart then resumes from the last completed unit.

Make units idempotent. A crash between doing the work and recording the checkpoint means the unit will be done again. If "done again" means "a second email sent" or "the same payment charged twice", you need idempotency keys or a check before acting; if it means "the same file uploaded again", it is merely wasteful.

Back off after repeated failures. A command that crashes on start because a dependency is down will be restarted immediately by most supervisors, crash again, and spin. systemd's RestartSec= and StartLimitBurst= or Kubernetes' crash-loop back-off handle this outside the process; inside it, apply the same capped exponential backoff described in retries and backoff for CLI HTTP calls to the loop's own retry of a failing unit.

Fail loudly on configuration errors. There is a difference between "the API is temporarily unavailable" (keep looping, retry) and "the token is invalid" or "the config file does not parse" (no amount of retrying will help). Exit non-zero for the second kind so the supervisor's restart limits and your alerting can see it, rather than logging the same error every thirty seconds forever.

Logging for commands nobody watches

Interactive output habits work against long-running commands. Progress bars and spinners fill log files with carriage-return redraws; colour codes appear as \x1b[32m in the journal; and the default print buffering means a crash can lose the last minutes of output entirely. Three adjustments help:

• Detect the environment. When stderr is not a terminal, switch progress bars off and use plain, timestamped log lines. The mechanics are in detecting a TTY and adapting output.
• Log units of work, not ticks. One line per unit — "synced 318 items (2 skipped) in 41s" — is the right granularity. A line per loop iteration when nothing happened drowns the useful ones.
• Flush promptly. Run with PYTHONUNBUFFERED=1 under supervisors, or configure logging handlers that flush after each record (the standard StreamHandler does), so the journal shows what happened right up to the moment of failure.

For services whose logs are shipped to a central system, structured JSON logging makes each line a queryable record.

Testing long-running commands

Long-running code is tested by making "long" short and "forever" finite:

• Inject the unit of work and count calls, so tests control what each iteration does.
• Stop the loop from the test by setting the stop event from a timer thread or from inside the unit itself after N calls.
• Use tiny intervals — the loop's behaviour does not change between 0.01 seconds and five minutes.
• Test real signals once, in a subprocess, to prove the handler is installed and the exit code is right.

import threading

from mytool.loop import Stopper, run_forever


def test_loop_runs_until_stopped(tmp_path):
    stopper = Stopper()
    calls = []

    def unit() -> int:
        calls.append(1)
        if len(calls) == 3:
            stopper.signum = 15
            stopper.event.set()
        return 1

    code = run_forever(unit, interval=0.01, stopper=stopper, heartbeat=tmp_path / "hb")
    assert len(calls) == 3
    assert code == 143
    assert (tmp_path / "hb").exists()


def test_failures_do_not_end_the_loop():
    stopper = Stopper()
    calls = []

    def unit() -> int:
        calls.append(1)
        if len(calls) < 3:
            raise RuntimeError("transient")
        stopper.event.set()
        return 0

    run_forever(unit, interval=0.01, stopper=stopper)
    assert len(calls) == 3

Key takeaways

• Treat SIGTERM exactly like Ctrl+C: set a stop event, finish the current unit, clean up, exit 128 + signal.
• Build loops from bounded units of work and interruptible waits.
• In watch mode, debounce, filter out your own outputs and keep watching after failures.
• Before scheduling, make commands non-interactive, idempotent, overlap-safe and path-explicit.
• Prefer systemd timers to cron where available for logging, catch-up and overlap protection.
• Publish progress with a heartbeat and a health command; use a dead-man switch for scheduled jobs.

Frequently asked questions

Related

• Up: CLI Runtime & Systems Integration
• Down: Handling SIGTERM and graceful shutdown
• Down: Building a watch mode with watchfiles
• Down: Running a CLI on a schedule with cron and systemd
• Down: Health checks and heartbeats for long-running CLIs
• Sideways: Concurrency and async in Python CLIs
• Sideways: Error handling and exit codes