Python CLI Toolcraft

Architecture

Lazy Loading Subcommands for Faster Startup

Speed up a multi-command Python CLI by lazy-loading subcommands: defer heavy imports until a command runs with a custom Click lazy group.

Updated

When a multi-command CLI constructs its command group, it typically imports every subcommand module — and every heavy dependency those modules pull in — before it even looks at what the user typed. This guide shows you how to defer that: a custom Click Group that imports a subcommand's module only when that subcommand is actually invoked, so --help and completion stay instant no matter how heavy your commands are.

TL;DR

  • The problem is not slow commands — it's that constructing the CLI eagerly imports every command module, so one command's import pandas taxes all the others.
  • Subclass click.Group and override list_commands (cheap, just names) and get_command (imports the module on demand). This is a LazyGroup.
  • Register subcommands as "module.path:attr" strings in a dict — the string is what lets you name a command without importing it.
  • Typer has no built-in lazy group, but you can wrap the underlying Click object or split into deferred sub-apps.
  • Verify the win with python -X importtime: heavy modules should be absent from --help and present only when their command runs.

The problem: one command taxes them all

Here is a conventional Click CLI. Each command lives in its own module and is added to the group at import time:

# cli.py — the eager version
import click
from .commands.convert import convert   # imports pandas at module top
from .commands.fetch import fetch        # imports requests
from .commands.report import report      # imports matplotlib

@click.group()
def cli() -> None:
    ...

cli.add_command(convert)
cli.add_command(fetch)
cli.add_command(report)

The three from .commands... import lines run the moment cli.py is imported, which is the moment the CLI starts. So mycli --help — a request for static text — imports pandas, requests, and matplotlib. On a typical machine that is several hundred milliseconds of work to print help that needed none of it. Every subcommand pays for every other subcommand's dependencies. As the CLI grows, startup degrades linearly.

Deferring the imports inside each command function helps only partly: importing the command module still runs that module's top-level import statements. To truly avoid the cost you must not import the module at all until the command is invoked.

A custom Click LazyGroup

Click looks up subcommands through two Group methods: list_commands(ctx) returns the names to display (for --help), and get_command(ctx, name) resolves one name to a Command object (for dispatch). The trick is to make list_commands cheap — it only needs strings — and to do the actual import inside get_command, which Click calls only for the command the user actually ran.

# lazy_group.py
from __future__ import annotations

import importlib
import click


class LazyGroup(click.Group):
    """A click.Group whose subcommands are imported on first use.

    lazy_subcommands maps a command name to a "module.path:attribute"
    string. The module is imported only when that command is invoked.
    """

    def __init__(
        self,
        *args,
        lazy_subcommands: dict[str, str] | None = None,
        **kwargs,
    ) -> None:
        super().__init__(*args, **kwargs)
        self._lazy: dict[str, str] = lazy_subcommands or {}

    def list_commands(self, ctx: click.Context) -> list[str]:
        # Eagerly-added commands + lazy names, without importing anything.
        return sorted({*super().list_commands(ctx), *self._lazy})

    def get_command(self, ctx: click.Context, name: str) -> click.Command | None:
        if name in self._lazy:
            return self._load(name)
        return super().get_command(ctx, name)

    def _load(self, name: str) -> click.Command:
        module_path, _, attr = self._lazy[name].partition(":")
        module = importlib.import_module(module_path)
        cmd = getattr(module, attr)
        if not isinstance(cmd, click.Command):
            raise TypeError(f"{self._lazy[name]!r} is not a click.Command")
        return cmd

Wiring the CLI to use it means passing the class and a registry of string paths — and crucially, not importing the command modules:

# cli.py — the lazy version. Note: no `from .commands... import ...`
import click
from .lazy_group import LazyGroup


@click.group(
    cls=LazyGroup,
    lazy_subcommands={
        "convert": "myapp.commands.convert:convert",
        "fetch": "myapp.commands.fetch:fetch",
        "report": "myapp.commands.report:report",
    },
)
def cli() -> None:
    """myapp — does three heavy things, but starts instantly."""


if __name__ == "__main__":
    cli()

Each command module is written exactly as before — an ordinary @click.command() with its heavy imports at the top:

# myapp/commands/convert.py
import click
import pandas as pd        # stays at module top; only imported when convert runs


@click.command()
@click.argument("path")
def convert(path: str) -> None:
    """Convert a CSV to Parquet."""
    df = pd.read_csv(path)
    df.to_parquet(path.replace(".csv", ".parquet"))
    click.echo(f"wrote {path.replace('.csv', '.parquet')}")

Now mycli --help calls list_commands, which returns ["convert", "fetch", "report"] as plain strings — no module is imported, no pandas, no matplotlib. Only mycli convert data.csv triggers get_command("convert"), which imports myapp.commands.convert and its pandas at that moment. The user pays for exactly the command they ran.

A self-contained version you can run now

To see the behavior without a package, register modules from the standard library and print when each is loaded:

import importlib
import click


class LazyGroup(click.Group):
    def __init__(self, *args, lazy_subcommands=None, **kwargs):
        super().__init__(*args, **kwargs)
        self._lazy = lazy_subcommands or {}

    def list_commands(self, ctx):
        return sorted({*super().list_commands(ctx), *self._lazy})

    def get_command(self, ctx, name):
        if name in self._lazy:
            module_path, _, attr = self._lazy[name].partition(":")
            print(f"[lazy] importing {module_path} for '{name}'")
            return getattr(importlib.import_module(module_path), attr)
        return super().get_command(ctx, name)


@click.group(cls=LazyGroup, lazy_subcommands={"hi": "_lazy_cmds:hi"})
def cli():
    ...

With a sibling _lazy_cmds.py defining a hi command, running python -m app --help prints the command list with no [lazy] importing line, while python -m app hi prints the import line first — proof the module loads only on invocation.

The string-path registry pattern

The heart of the technique is that a command is registered as a string, "module.path:attr", not an imported object. A string can name a target without triggering its import; importlib.import_module resolves it later. This is the same module:attribute convention Python uses for console-script entry points, which is no coincidence — see best practices for Python CLI entry points.

Keep the registry in one place — the lazy_subcommands dict on your top-level group — so there is a single readable manifest of every command and where it lives. For a large CLI you can build the dict programmatically (e.g. from a directory listing), but an explicit dict is easier to reason about and, importantly, still imports nothing at construction time. The one rule: never import a command module from cli.py. The instant you do, that command is eager again and the string registry buys you nothing.

Doing the same in Typer

Typer builds on Click but does not expose a lazy-group option directly. Two workable approaches:

  • Reuse the Click LazyGroup. A typer.Typer is convertible to a Click object with typer.main.get_command(app). For a Typer-first app you can define your top-level group as a Click LazyGroup and mount Typer sub-apps under it, or keep the whole top level in Click and only use Typer inside individual (lazily imported) command modules.
  • Split into deferred sub-apps. Typer's app.add_typer(sub_app, name=...) still imports sub_app at call time, so it is not lazy by itself. Wrap the registration so the sub-app is built by a factory that is only called from a Click get_command override.

If you are choosing between the two frameworks for a startup-sensitive tool, Click's explicit Group subclassing makes lazy loading a first-class, well-supported pattern, which is a real point in its favor — weigh it in Typer vs Click: when to use each.

Measuring the win with -X importtime

Don't take the speedup on faith — prove it. python -X importtime prints every import and its cost, so you can confirm heavy modules are absent from the help path:

# Heavy modules should NOT appear here.
$ python -X importtime -m myapp --help 2>&1 | grep -E "pandas|matplotlib" || echo "no heavy imports on --help"
no heavy imports on --help

# They SHOULD appear only when the command runs.
$ python -X importtime -m myapp convert data.csv 2>&1 | grep -c pandas
1

The first command confirms --help no longer drags in pandas; the second confirms it loads exactly when convert runs. For the full profiling workflow — reading the cumulative column, visualizing with tuna, and timing wall-clock with hyperfine — see profiling Python CLI startup time.

Production notes

  • list_commands must stay cheap. If you ever compute command names by importing modules and inspecting them, you have reintroduced the eager cost on --help. Keep names as static strings.
  • Completion calls list_commands too. Shell completion of subcommand names goes through the same path, so lazy loading keeps tab completion instant. Completing arguments, however, invokes get_command, so a command whose options need heavy imports will pay on argument completion — keep option definitions light.
  • Import errors surface late. With eager imports a broken command fails at startup; with lazy loading it fails only when invoked. That is usually what you want (one broken command doesn't down the whole CLI), but add a test that imports every registered module so CI still catches breakage early.
  • Bytecode caching matters in benchmarks. First run compiles .pyc files; measure a warm run for realistic numbers.
  • Structure first. Per-command lazy loading assumes one module per command. If your CLI is still a monolith, split it following how to structure a large Python CLI project before adding a LazyGroup.