Smoke-Testing the Built Wheel of a Python CLI in CI

Every test passed. The release went out. The first user to run mytool init got FileNotFoundError: templates/default.toml, because the template directory was never included in the wheel. Or the entry point said mytool.cli:mian. Or rich was listed under development dependencies, where it happened to be installed for every test run. These bugs share a cause: the tests ran against your source checkout, where everything is present and importable, while users install a wheel, which contains only what the build configuration says. A smoke test closes that gap by installing the artefact you are about to publish into a clean environment and running the command the way a user would. This guide builds that test for a CLI, including a doctor command that makes it thorough, and wires it into CI between the build and publish jobs. It is part of the CI/CD pipelines topic.

Prerequisites

• A CLI that builds with uv build into dist/. See building wheels and sdists for Python CLIs.
• A console-script entry point in pyproject.toml ([project.scripts]).
• A CI pipeline with a build job that uploads dist/ as an artefact, as in the topic overview.

What unit tests cannot see

When pytest runs in your repository, import mytool resolves to src/mytool (or the editable install of it), every non-Python file in the tree is on disk next to the code, and the environment contains every development dependency. None of that is true for a user.

Each of those bugs is invisible to any test that imports from the checkout, and each is caught instantly by installing the wheel somewhere clean and running the command once. That is the whole idea of a smoke test: not a second test suite, just enough real usage of the real artefact to prove the package is whole.

The recipe, part 1: a doctor command

mytool --version proves the entry point and the top-level import work. It does not prove that the templates, schemas, plugins and lazily imported modules are all present. A small doctor subcommand that touches every packaged resource makes the smoke test meaningful — and is useful to users debugging their installation too:

# src/mytool/doctor.py
from __future__ import annotations

import importlib
import json
from importlib import metadata, resources

import typer

LAZY_MODULES = ["mytool.commands.deploy", "mytool.commands.report", "mytool.render"]


def check_templates() -> str:
    root = resources.files("mytool") / "templates"
    names = sorted(p.name for p in root.iterdir() if p.name.endswith(".toml"))
    if not names:
        raise RuntimeError("no templates packaged")
    return f"{len(names)} found"


def check_schema() -> str:
    schema = json.loads((resources.files("mytool") / "schema.json").read_text(encoding="utf-8"))
    return f"ok ({len(schema.get('properties', {}))} properties)"


def check_modules() -> str:
    for name in LAZY_MODULES:
        importlib.import_module(name)
    return f"{len(LAZY_MODULES)} imported"


def check_plugins() -> str:
    eps = metadata.entry_points(group="mytool.plugins")
    for ep in eps:
        ep.load()
    return f"{len(eps)} loaded"


CHECKS = {"templates": check_templates, "config schema": check_schema,
          "lazy modules": check_modules, "plugins": check_plugins}


def doctor() -> None:
    """Check that this installation is complete."""
    failed = False
    for label, check in CHECKS.items():
        try:
            typer.echo(f"{label:<14} {check()}")
        except Exception as exc:
            failed = True
            typer.secho(f"{label:<14} FAILED: {type(exc).__name__}: {exc}", fg="red", err=True)
    typer.echo(f"mytool {metadata.version('mytool')}")
    raise typer.Exit(1 if failed else 0)

Register it with app.command()(doctor). Three details matter. Resources are read with importlib.resources.files(), which works whether the package is a directory, a zip or a frozen binary — reading Path(__file__).parent / "templates" works from the checkout and fails in some install layouts; see bundling data files with importlib.resources. Lazily imported subcommand modules are imported explicitly, because a CLI that lazy-loads subcommands would otherwise never import them during --help. And the version comes from installed metadata, which proves the distribution's metadata is intact.

The recipe, part 2: the CI job

The smoke job downloads the artefact built earlier, installs it into an isolated environment with no development dependencies, and runs it from a directory outside the checkout:

  smoke:
    needs: build
    strategy:
      matrix:
        os: [ubuntu-latest, windows-latest, macos-latest]
    runs-on: ${{ matrix.os }}
    steps:
      # Deliberately no checkout: the source tree must not be importable.
      - uses: actions/download-artifact@v4
        with:
          name: dist
          path: dist
      - uses: astral-sh/setup-uv@v6
        with:
          python-version: "3.12"
      - name: Install the wheel as a tool
        shell: bash
        run: uv tool install dist/*.whl
      - name: Run it like a user would
        shell: bash
        working-directory: ${{ runner.temp }}
        run: |
          mytool --version
          mytool --help > /dev/null
          mytool doctor
      - name: The sdist must build and work too
        shell: bash
        run: |
          uv tool uninstall mytool
          uv tool install dist/*.tar.gz
          cd "$RUNNER_TEMP" && mytool doctor

Leaving out actions/checkout is the strongest guarantee that nothing from the repository is on the import path. uv tool install creates a dedicated environment containing your package and its runtime dependencies only, and puts the mytool launcher on PATH — exactly what a user gets from uv tool install mytool or pipx install mytool. Running from $RUNNER_TEMP avoids Python's habit of importing from the current directory.

The sdist step catches a different class of bug: a source distribution that is missing files needed to build (a README.md referenced by metadata, a LICENSE, a build-time data file). Downstream packagers — Linux distributions, conda-forge, Homebrew — build from the sdist, so it is worth proving it works.

UX considerations

The smoke test protects users directly, and a few choices make it more useful:

• Ship doctor as a public command. When a user reports "it doesn't work", asking for mytool doctor output tells you in seconds whether the installation is broken, the environment is odd, or the bug is real.
• Keep it fast and offline. A doctor command that calls the network fails in air-gapped CI and on planes. Check local resources; offer a separate --online flag for connectivity checks if they are useful.
• Make failures specific. "templates FAILED: FileNotFoundError: templates" points straight at the packaging configuration.
• Test the oldest supported Python in the smoke job too if your matrix is large; a wheel tagged py3-none-any should install everywhere requires-python allows.
• Keep the list of lazy modules honest. Generate it from your command registry rather than maintaining it by hand, so new subcommands are covered automatically.

Testing the behaviour

The doctor command itself deserves ordinary unit tests, and you can reproduce the whole CI smoke test locally before pushing:

# Build, then install into a throwaway tool environment and run from elsewhere.
uv build
uv tool install --force dist/mytool-*.whl
(cd "$(mktemp -d)" && mytool --version && mytool doctor)
uv tool uninstall mytool

To confirm the smoke test actually catches what it should, break the packaging on purpose once — exclude the templates directory in the build configuration, or misspell the entry point — rebuild, and watch doctor fail. A smoke test that has never been seen to fail has not been proven to work.

For a pytest-based variant, mark end-to-end tests that call the installed command with @pytest.mark.installed, skip them unless shutil.which("mytool") resolves outside the repository, and run pytest -m installed in the smoke job after installing the wheel. The patterns for that are in end-to-end testing an installed CLI.

Conclusion

Unit tests prove your code works; a smoke test proves your package contains your code. Build once, install the artefact into a clean environment with only runtime dependencies, run it from outside the checkout on each operating system, and give it a doctor command that touches every resource, lazy module and plugin. It adds a minute to the pipeline and removes an entire category of "works on my machine" release bugs.

Frequently asked questions

Related

• Up: CI/CD pipelines for Python CLIs
• Publishing to PyPI with trusted publishing
• Automating releases from git tags
• Bundling data files with importlib.resources
• End-to-end testing an installed CLI