Testing Plugins Against the Host CLI

A plugin is code that only works inside someone else's program. Unit tests of its functions say little about whether it works when the host CLI discovers it, mounts its commands, passes it configuration and renders its output — and the host evolves on its own schedule, so a plugin that works today can break with the host's next minor release. Plugin testing therefore has two halves. The host should ship tools that make plugins easy to test: a way to build the CLI with a given plugin mounted, and a set of contract tests every plugin can run. The plugin should use them, verify its own packaging registers the entry point, and run its tests against every host version it claims to support. This guide builds both halves for a Typer-based host and plugin. It belongs to the plugin architectures topic.

Prerequisites

• A host CLI that mounts plugins from an entry-point group, as in discovering plugins with entry points.
• A plugin package written as in writing a plugin for an existing CLI.

What to test, and where

1. Plugin logic — ordinary unit tests in the plugin's repository, no host involved.
2. The plugin inside the host — the host's real command tree with the plugin mounted, invoked through CliRunner. This catches option clashes, help rendering problems and assumptions about the context.
3. Packaging — the installed plugin's entry point actually points at the right object, so discovery finds it.
4. Compatibility — the same tests against the oldest and newest host versions the plugin supports.

The host makes layers 2 and 4 cheap by shipping a small testing module.

The recipe: the host side

# src/mytool/testing.py
"""Helpers for testing plugins against mytool. Part of the public plugin API."""
from __future__ import annotations

from collections.abc import Mapping

import typer
from typer.testing import CliRunner, Result

from mytool.cli import build_app


def app_with_plugins(plugins: Mapping[str, typer.Typer], *, discover: bool = False) -> typer.Typer:
    """Build the real mytool command tree with the given plugin apps mounted."""
    return build_app(extra_plugins=dict(plugins), discover=discover)


def invoke(app: typer.Typer, *args: str, env: Mapping[str, str] | None = None) -> Result:
    return CliRunner().invoke(app, list(args), env=dict(env or {}))


def check_plugin_contract(name: str, plugin_app: typer.Typer) -> list[str]:
    """Return a list of contract violations for a plugin app (empty means OK)."""
    problems = []
    app = app_with_plugins({name: plugin_app})
    help_result = invoke(app, name, "--help")
    if help_result.exit_code != 0:
        problems.append(f"'{name} --help' exited with {help_result.exit_code}")
    for info in plugin_app.registered_commands:
        cmd = info.name or info.callback.__name__.replace("_", "-")
        if not (info.help or info.callback.__doc__):
            problems.append(f"command {cmd!r} has no help text")
        result = invoke(app, name, cmd, "--help")
        if result.exit_code != 0:
            problems.append(f"'{name} {cmd} --help' exited with {result.exit_code}")
    return problems

# src/mytool/cli.py (excerpt)
from importlib.metadata import entry_points

import typer

BUILTINS = {"status", "plugins"}


def build_app(extra_plugins: dict[str, typer.Typer] | None = None,
              discover: bool = True) -> typer.Typer:
    app = typer.Typer()

    @app.callback()
    def main() -> None:
        """mytool."""

    @app.command()
    def status() -> None:
        """Show status."""
        typer.echo("ok")

    plugins = dict(extra_plugins or {})
    if discover:
        for ep in entry_points(group="mytool.commands"):
            plugins.setdefault(ep.name, ep.load())
    for name, plugin_app in sorted(plugins.items()):
        if name not in BUILTINS:
            app.add_typer(plugin_app, name=name)
    return app

The essential design decision is that the command tree is built by a function, not at import time, so tests can build it with exactly the plugins they want — and with discovery switched off, so a developer's other installed plugins do not leak into the test. check_plugin_contract encodes the host's expectations once — every command has help, help renders without error — so every plugin checks the same rules. Treat mytool.testing as public API: document it, and keep it stable across minor versions like the rest of the plugin interface described in versioning a plugin API.

The recipe: the plugin side

# tests/test_in_host.py (in the mytool-aws plugin repository)
from importlib.metadata import entry_points

from mytool.testing import app_with_plugins, check_plugin_contract, invoke

from mytool_aws.cli import app as aws_app


def test_plugin_meets_the_host_contract():
    assert check_plugin_contract("aws", aws_app) == []


def test_command_runs_inside_the_host():
    app = app_with_plugins({"aws": aws_app})
    result = invoke(app, "aws", "buckets", "--prefix", "logs-",
                    env={"MYTOOL_AWS_FAKE": "1"})
    assert result.exit_code == 0
    assert "logs-2026" in result.stdout


def test_entry_point_is_registered():
    eps = {ep.name: ep for ep in entry_points(group="mytool.commands")}
    assert "aws" in eps, "is the plugin installed (uv sync / pip install -e .)?"
    assert eps["aws"].load() is aws_app

The last test is the packaging check. It fails if the [project.entry-points."mytool.commands"] table in pyproject.toml has a typo, points at the wrong attribute, or uses the wrong group name — mistakes that pass every other test because those tests import the plugin directly.

A compatibility matrix

A plugin declares the host versions it supports in its dependencies (mytool>=1.4,<2). CI should test the edges of that range, not just whatever version the lock file picked:

# .github/workflows/test.yml (in the plugin repository)
jobs:
  test:
    strategy:
      matrix:
        host: ["mytool==1.4.*", "mytool<2"]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v6
      - run: uv run --with "${{ matrix.host }}" pytest -q

uv run --with adds the requested host version on top of the project environment for that run, so one job tests the oldest supported host and another the newest release. When the newest-host job fails, either the plugin needs a fix or the host broke its contract — both worth knowing before users find out. Hosts can return the favour: a scheduled job in the host repository that installs popular plugins and runs their contract checks catches accidental API breaks before a release.

Testing what the plugin receives from the host

Plugins rarely live on options alone: they read the host's configuration, use its HTTP client or credentials, and write output through its formatting helpers. Those touch points are the most fragile part of the contract, so the host's testing module should let plugin tests supply them explicitly instead of reaching for real files and environment variables. A typical pattern is a context object the host passes to plugins — on ctx.obj, as in sharing state with Click context objects — and a helper that builds one for tests:

# assumes a context-aware variant of the helpers above
from mytool.testing import app_with_plugins, invoke, make_context

ctx = make_context(config={"region": "eu-west-1"}, output_format="json")
result = invoke(app_with_plugins({"aws": aws_app}, context=ctx), "aws", "buckets")

The plugin then tests real interactions — "when the host is in JSON mode, my command returns records rather than printing a table" — without knowing how the host loads configuration internally. When the host changes that internal machinery, plugin tests keep passing because they only depend on the published context shape. Keep the context small and versioned; every field on it is something plugins will start to rely on.

UX considerations

The users of this tooling are plugin authors, often outside your team:

• Document the testing module next to the plugin guide, with a copy-paste example like the plugin-side tests above.
• Make contract failures specific. "command 'buckets' has no help text" is actionable; "contract failed" is not.
• Keep discovery out of tests by default. A test that silently picks up every plugin installed on the developer's machine is flaky in the most confusing way.
• Version the contract. If the host adds a rule, announce it, and consider making it a warning for one minor release before it fails.

Testing the behaviour

The host should test its own testing helpers — they are API — including that the contract check actually catches violations:

# tests/test_testing_helpers.py (in the host repository)
import typer

from mytool.testing import app_with_plugins, check_plugin_contract, invoke


def make_plugin(with_docs: bool) -> typer.Typer:
    plugin = typer.Typer()

    @plugin.command()
    def ping() -> None:
        typer.echo("pong")

    if with_docs:
        ping.__doc__ = "Reply with pong."
    return plugin


def test_mounted_plugin_runs():
    result = invoke(app_with_plugins({"net": make_plugin(True)}), "net", "ping")
    assert result.stdout == "pong\n"


def test_contract_passes_for_documented_plugin():
    assert check_plugin_contract("net", make_plugin(True)) == []


def test_contract_flags_missing_help():
    assert check_plugin_contract("net", make_plugin(False)) == ["command 'ping' has no help text"]


def test_builtins_cannot_be_shadowed():
    app = app_with_plugins({"status": make_plugin(True)})
    assert invoke(app, "status").stdout == "ok\n"

Conclusion

Plugins are only as reliable as their integration with the host. Give plugin authors a host-provided testing module that builds the real command tree with chosen plugins mounted and checks a written contract; in each plugin, test inside that tree, verify the entry point resolves to the right object, and run the suite against the oldest and newest supported host versions. Hosts that also run popular plugins' checks before releasing catch their own breaking changes first.

Frequently asked questions

Related

• Up: Plugin architectures for extensible CLIs
• Writing a plugin for an existing CLI
• Isolating plugin failures
• Versioning a plugin API
• Testing Click commands with CliRunner