Property-Based Testing CLI Arguments with Hypothesis

Example-based tests check the inputs you thought of: --timeout 30s, --timeout 5m, --timeout nonsense. Users — and scripts, and other programs feeding your CLI — produce the inputs you did not think of: --timeout 0., a name with a trailing newline, a path containing =, an empty string, a number with a thousand digits. Many of those crash a command with a traceback instead of a clean usage error. Property-based testing turns the question around: instead of listing inputs, you describe the space of inputs and state properties that must hold for all of them, and the Hypothesis library searches that space for a counterexample — then shrinks it to the simplest input that still fails. This guide applies it to the argument-parsing and output edges of a CLI, where it finds real bugs quickly. It belongs to the testing Python CLI applications topic.

Prerequisites

• Python 3.10+, pytest and Hypothesis (uv add --dev hypothesis).
• A Typer or Click CLI with commands testable through CliRunner, as in testing Click commands with CliRunner.

How property-based testing works

A Hypothesis test is an ordinary pytest function decorated with @given(...), which describes how to generate its arguments using strategies — st.text(), st.integers(), st.from_regex(...), and combinations of them. Hypothesis runs the test many times (100 by default) with generated values, biased towards edge cases such as empty strings, boundary numbers and unusual Unicode. When an assertion fails, it shrinks the failing input step by step to a minimal example and reports it. Failures are stored in a local database and replayed first on the next run, so a bug, once found, stays found until fixed.

The code under test

Here is a small command with a duration parser and a JSON-emitting mode — the kind of code that looks obviously correct:

# src/mytool/durations.py
import re

UNITS = {"ms": 0.001, "s": 1.0, "m": 60.0, "h": 3600.0}
PATTERN = re.compile(r"(\d+(?:\.\d+)?)(ms|s|m|h)")


def parse_duration(text: str) -> float:
    """'1.5h', '30s', '250ms' -> seconds. Raises ValueError on bad input."""
    m = PATTERN.fullmatch(text.strip())
    if not m:
        raise ValueError(f"invalid duration {text!r}")
    return float(m[1]) * UNITS[m[2]]


def format_duration(seconds: float) -> str:
    """The inverse: seconds -> shortest exact representation."""
    for unit in ("h", "m", "s"):
        value = seconds / UNITS[unit]
        if value >= 1 and value == int(value):
            return f"{int(value)}{unit}"
    return f"{round(seconds * 1000)}ms"

# src/mytool/cli.py
import json
from typing import Annotated

import typer

from mytool.durations import format_duration, parse_duration

app = typer.Typer()


@app.callback()
def main() -> None:
    """Scheduling helpers."""


@app.command()
def wait(
    duration: Annotated[str, typer.Argument(help="How long, e.g. 30s or 1.5h.")],
    as_json: Annotated[bool, typer.Option("--json")] = False,
) -> None:
    """Report how long DURATION is."""
    try:
        seconds = parse_duration(duration)
    except ValueError as exc:
        raise typer.BadParameter(str(exc), param_hint="DURATION")
    if as_json:
        typer.echo(json.dumps({"input": duration, "seconds": seconds}))
    else:
        typer.echo(f"{format_duration(seconds)} ({seconds:g} seconds)")

The recipe: properties worth testing

# tests/test_properties.py
import json
import math

from hypothesis import given, settings
from hypothesis import strategies as st
from typer.testing import CliRunner

from mytool.cli import app
from mytool.durations import format_duration, parse_duration

runner = CliRunner()

# Any printable text a user or script might pass as an argument.
arg_text = st.text(alphabet=st.characters(blacklist_categories=("Cs",), blacklist_characters="\x00"),
                   max_size=40)
# Syntactically valid durations, including edge cases like '0s' and '007m'.
valid_duration = st.from_regex(r"\A[0-9]{1,6}(\.[0-9]{1,3})?(ms|s|m|h)\Z")


@given(arg_text)
@settings(max_examples=300)
def test_never_crashes(text):
    """Property 1: any argument produces a result or a usage error, never a traceback."""
    result = runner.invoke(app, ["wait", "--", text])
    assert result.exit_code in (0, 2), result.output
    assert result.exception is None or isinstance(result.exception, SystemExit)


@given(valid_duration)
def test_valid_input_is_accepted(text):
    """Property 2: everything the documented grammar allows parses to a finite, non-negative number."""
    value = parse_duration(text)
    assert math.isfinite(value) and value >= 0


@given(valid_duration)
def test_round_trip(text):
    """Property 3: formatting a parsed duration and parsing it again gives the same value."""
    seconds = parse_duration(text)
    assert parse_duration(format_duration(seconds)) == seconds


@given(valid_duration)
def test_json_output_always_parses(text):
    """Property 4: --json output is valid JSON with the documented keys."""
    result = runner.invoke(app, ["wait", "--json", "--", text])
    assert result.exit_code == 0
    data = json.loads(result.stdout)
    assert set(data) == {"input", "seconds"}

What each property buys you

Never crashes is the single most productive property for a CLI. It asserts nothing about correctness — only that every possible argument ends in success or a clean usage error (exit 2), never an unhandled exception. In real tools it regularly finds inputs that slip past validation: whitespace-only strings, values that convert to infinity, characters a regular expression accepts but a converter rejects. The -- before the generated text makes sure it is treated as an argument even when it starts with a dash.

Valid input is accepted tests the parser against its own grammar. Generating from a regular expression that mirrors the documentation keeps the test honest about what users are promised.

Round trip tests two functions against each other: whatever the parser accepts, the formatter must write in a form the parser reads back to the same value. Round-trip properties are powerful because they need no oracle — you never compute the expected answer yourself.

JSON always parses protects scripts: whatever the input, --json output is valid JSON with the documented keys, and nothing else is mixed into stdout. The contract behind it is described in emitting JSON output for scripting.

What Hypothesis found

Against the code above, three of the four properties pass. The round-trip property fails within a fraction of a second, and Hypothesis shrinks the failure to a tiny, readable input:

0.1ms is valid according to the documented grammar, parses to 0.0001 seconds, and is then formatted as 0ms because format_duration rounds to whole milliseconds. A user who runs mytool wait 0.1ms sees "0ms" echoed back — and any code that stores durations by formatting them silently loses information. Nobody writing examples by hand would think to try a sub-millisecond value; Hypothesis tries it early because it deliberately explores small and fractional numbers.

The fix is to decide what the tool's resolution really is and make both functions agree on it. Here, durations are whole milliseconds: parse exactly with Decimal, reject anything finer, and format from an integer. Restricting the pattern to ASCII digits ([0-9] rather than \d, which in Python also matches other scripts' digits) removes a second, subtler class of surprise at the same time:

# src/mytool/durations.py (fixed)
import re
from decimal import Decimal, InvalidOperation

UNIT_MS = {"ms": 1, "s": 1000, "m": 60_000, "h": 3_600_000}
PATTERN = re.compile(r"([0-9]+(?:\.[0-9]+)?)(ms|s|m|h)")   # ASCII digits only


def parse_ms(text: str) -> int:
    """'1.5h', '30s', '250ms' -> whole milliseconds. Raises ValueError on bad input."""
    m = PATTERN.fullmatch(text.strip())
    if not m:
        raise ValueError(f"invalid duration {text!r}")
    try:
        ms = Decimal(m[1]) * UNIT_MS[m[2]]
    except InvalidOperation:
        raise ValueError(f"invalid duration {text!r}") from None
    if ms != ms.to_integral_value():
        raise ValueError(f"{text!r} is finer than one millisecond")
    return int(ms)


def format_ms(ms: int) -> str:
    """Whole milliseconds -> the largest unit that represents them exactly."""
    for unit in ("h", "m", "s"):
        if ms >= UNIT_MS[unit] and ms % UNIT_MS[unit] == 0:
            return f"{ms // UNIT_MS[unit]}{unit}"
    return f"{ms}ms"

With integers there is no floating-point rounding anywhere, so the round-trip property can be strengthened to cover every possible value, not just parsed text:

@given(st.integers(min_value=0, max_value=10**12))
def test_format_parse_round_trip_for_all_values(ms):
    assert parse_ms(format_ms(ms)) == ms

Hypothesis also stores the original counterexample in its local database and replays it first on the next run, so the regression is checked every time; adding @example("0.1ms") to the round-trip test makes it visible in the code as well.

UX considerations

Property-based tests improve the user experience indirectly, by making every weird input end in a helpful error:

• No tracebacks for bad input. The never-crash property is effectively a guarantee that users see "Invalid value for 'DURATION': invalid duration '…'" rather than a stack trace. See friendly error messages and tracebacks.
• Documented grammar matches reality. Generating valid inputs from the documented format keeps help text and behaviour aligned.
• Script-safe output. Output-format properties protect everyone who pipes your tool into something else.

Testing the behaviour

Keep property tests fast and deterministic enough for CI:

• Bound the input size (max_size=40); CLI arguments are short, and huge inputs only slow the search.
• Use settings profiles: a small max_examples for local runs, a larger one in a nightly CI job. Register them in conftest.py with settings.register_profile("ci", max_examples=1000) and select one with --hypothesis-profile=ci.
• Commit nothing from .hypothesis/ — the failure database is local — but turn every interesting failure into an explicit example with @example("0.") so it is documented in the test file.
• Avoid filesystem-heavy work per example. A property that creates files for every one of 300 examples is slow; test path-handling logic on pure functions, and keep one or two example-based tests for the filesystem itself.

Conclusion

Property-based testing is unusually effective at the edges of a CLI, where arbitrary text becomes typed values. State a few properties — never crashes, accepts everything the grammar allows, formatter and parser round-trip, machine output always parses — and let Hypothesis search for counterexamples and shrink them to something readable. It finds the Unicode digits, empty strings and overflowing numbers that example-based tests never think to try, and every bug it finds becomes a permanent regression test.

Frequently asked questions

Related

• Up: Testing Python CLI applications
• End-to-end testing an installed CLI
• Snapshot testing CLI output
• Writing custom Click parameter types
• Advanced argument validation strategies