Your CLI declares requires-python = ">=3.10", and your tests pass on the 3.13 you develop with. Then a user on 3.10 reports AttributeError: type object 'Path' has no attribute 'walk', and a Windows user reports that every path in the output has the wrong separator. Neither bug was hard to fix; both were impossible to see from your machine. A CI test matrix runs your suite on the combinations your users actually have. This guide builds one in GitHub Actions with uv — every supported Python on Linux, the extremes on Windows and macOS, an extra job at your lowest declared dependency versions — explains the fail-fast and caching choices, and shows how to reproduce any failing job locally. It is part of the CI/CD pipelines topic.
Prerequisites
- A CLI project managed with uv, with a
uv.lockcommitted and tests intests/runnable withuv run pytest. See uv for Python CLI dependency management. - Development dependencies in a
devdependency group (uv add --dev pytest). - A GitHub repository with Actions enabled.
Choosing the matrix
Start from your support policy. requires-python states the oldest version you support; the newest is the latest stable CPython release. Everything in between is supported implicitly. Each combination you test costs runner time, and Windows and macOS runners are slower and — for private repositories — billed at a multiple of Linux minutes. The trade-off that serves most CLI projects:
Version-specific bugs (new syntax, removed deprecations, stdlib additions) show up on Linux as reliably as anywhere, so test every version there. Platform-specific bugs (paths, encodings, subprocess behaviour, terminal handling) show up on any Python version, so Windows and macOS only need the oldest and newest — the oldest because it is where "I used something too new" appears, the newest because it is where deprecations become errors. Nine jobs instead of fifteen, with almost no loss of coverage.
The recipe
# .github/workflows/tests.yml
name: tests
on:
push:
branches: [main]
pull_request:
schedule:
- cron: "0 6 * * 1" # weekly, to catch breakage from new releases
permissions:
contents: read
concurrency:
group: tests-${{ github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
jobs:
test:
name: py${{ matrix.python }} · ${{ matrix.os }}${{ matrix.resolution == 'lowest-direct' && ' · lowest deps' || '' }}
runs-on: ${{ matrix.os }}
strategy:
fail-fast: ${{ github.event_name == 'pull_request' }}
matrix:
os: [ubuntu-latest]
python: ["3.10", "3.11", "3.12", "3.13", "3.14"]
resolution: [locked]
include:
- { os: windows-latest, python: "3.10", resolution: locked }
- { os: windows-latest, python: "3.14", resolution: locked }
- { os: macos-latest, python: "3.10", resolution: locked }
- { os: macos-latest, python: "3.14", resolution: locked }
- { os: ubuntu-latest, python: "3.10", resolution: lowest-direct }
env:
PYTHONUTF8: "0" # test the default encoding behaviour users get
FORCE_COLOR: "0"
steps:
- uses: actions/checkout@v4
- uses: astral-sh/setup-uv@v6
with:
python-version: ${{ matrix.python }}
enable-cache: true
- name: Install (locked)
if: matrix.resolution == 'locked'
run: uv sync --locked --group dev
- name: Install (lowest direct dependencies)
if: matrix.resolution == 'lowest-direct'
run: uv sync --resolution lowest-direct --group dev
- name: Test
run: uv run pytest -q --durations=10
summary:
if: always()
needs: test
runs-on: ubuntu-latest
steps:
- run: test "${{ needs.test.result }}" = "success"
Why each piece is there
setup-uv with python-version. uv installs the requested interpreter itself (a managed CPython build) and sets it for the job, so there is no separate setup-python step and the same versions are available on every OS. Passing python-version also sets UV_PYTHON, which uv sync and uv run then use.
uv sync --locked. The locked jobs install exactly what uv.lock specifies and fail if the lockfile is out of date with pyproject.toml. That catches the common mistake of editing dependencies without re-locking, and guarantees CI tests what developers run.
The lowest-direct job. --resolution lowest-direct installs the lowest version of each direct dependency your constraints allow. If you declared click>=8.0 but used a feature added in 8.1, this job fails and the others do not. It is the only way to keep lower bounds honest, and a CLI's lower bounds matter because users install it into environments with other packages pinning shared dependencies.
Encoding and colour environment. Setting PYTHONUTF8=0 explicitly tests the default behaviour on Windows, where the locale encoding is often cp1252 — the source of many Windows-only CLI bugs covered in fixing Unicode and encoding errors on Windows. FORCE_COLOR=0 keeps colour codes out of assertions on output.
Readable job names. The name: expression produces "py3.10 · windows-latest" in the UI, so a failing combination is identifiable at a glance.
A summary job. Branch protection needs stable check names, and matrix job names change when you add a Python version. Requiring the single summary job — which fails unless every matrix job succeeded — means you never have to edit protection rules when the matrix changes.
Fail-fast, concurrency and schedules
By default, a failure in one matrix job cancels the others. That is what you want on a pull request that is still being worked on — fast feedback, fewer wasted minutes — and exactly what you do not want on main, where a Windows-only failure could hide a separate macOS failure you would otherwise have fixed in the same change.
The concurrency block applies the same logic to whole runs: pushing a new commit to a pull request cancels the now-obsolete run for the previous commit. The weekly schedule catches breakage caused by the world rather than by your code — a dependency release, a new runner image, a Python patch release — while the project is quiet.
UX considerations
The "users" of a CI configuration are the contributors reading its results, and a few habits make those results easier to act on:
- Show slow tests.
--durations=10prints the ten slowest tests, which keeps suite time visible before it becomes a problem. - Make failures obvious in the summary. Actions shows each failing job's name; with descriptive names, "py3.10 · windows-latest" tells a contributor where to look before they open a log.
- Skip what cannot run, loudly. Tests that need POSIX signals or a specific tool should use
pytest.mark.skipifwith a reason, so the Windows job reports them as skipped rather than failing — and the reasons appear inpytest -rsoutput. - Document how to reproduce. A sentence in
CONTRIBUTING.md— "to reproduce a CI job, runuv run --python 3.10 pytest" — saves every contributor from reverse-engineering the workflow.
Testing the behaviour
The best way to test the matrix is to run it locally before pushing. uv makes this nearly free, because it downloads missing interpreters on demand and keeps a separate environment per version:
# Every locked version, in isolated environments, without touching your .venv
for v in 3.10 3.11 3.12 3.13 3.14; do
echo "== Python $v"
uv run --isolated --python "$v" --group dev pytest -q || break
done
# The lowest-dependency job
uv run --isolated --python 3.10 --resolution lowest-direct --group dev pytest -q
--isolated runs each in a temporary environment so your project .venv stays on your preferred version. For Windows- and macOS-specific failures, the CI job's log is usually enough; when it is not, adding a temporary tmate or SSH-debug step to a single matrix job, or a workflow_dispatch trigger to re-run only that combination, is faster than guessing.
You can also verify the workflow file itself before pushing: actionlint catches expression typos, unknown keys and shell mistakes in workflow YAML, and runs happily as a pre-commit hook.
Conclusion
A test matrix is the cheapest way to meet your users' environments before they meet your bugs. Test every supported Python on Linux, the oldest and newest on Windows and macOS, and one job at your lowest declared dependency versions; install with uv sync --locked; cancel sibling jobs only on pull requests; and require a single summary check so the matrix can evolve freely. With uv, the same matrix runs locally in a loop, so most failures never reach CI at all.
Frequently asked questions
Should I test pre-release Pythons?
Yes, as an allowed-to-fail job once the first release candidate is out. Add continue-on-error: ${{ contains(matrix.python, 'rc') }} or a separate job with python-version: "3.15" and allow-prereleases, so failures are visible without blocking merges.
Why not use actions/setup-python?
It works fine alongside uv, and some teams prefer it. setup-uv with python-version is one step instead of two, uses the same managed interpreters developers get locally with uv, and keeps interpreter selection consistent across operating systems.
How do I test against free-threaded Python?
uv can install free-threaded builds (3.14t). Add one Linux job with that version to learn early whether your dependencies support it. Mark it allowed-to-fail until they do.
When should I drop an old Python version from the matrix?
When it reaches end of life (five years after release, each October), or earlier if keeping it costs real effort. Drop it in the same change that raises requires-python and removes it from the trove classifiers, and mention it in the changelog: pip and uv will then keep users on that old interpreter on your last compatible release instead of installing one that fails at import time.
My tests pass locally but fail only on the Windows runner. Where do I start?
Look first at paths (separators, case, drive letters), text encoding (open() without encoding=), line endings in expected-output fixtures, and subprocess calls to programs that are .cmd shims on Windows. Those four account for the large majority of Windows-only CLI failures.