Fixing Unicode and Encoding Errors on Windows in Python CLIs

A Windows user reports that mytool status > status.txt crashes with UnicodeEncodeError: 'charmap' codec can't encode character '\u2713'. You cannot reproduce it: on your Mac the command works, and on the user's machine it works too — until they redirect the output. Another user's config file with a German comment is read as Größe. A third sees ? in place of every accented letter in the output of a program your tool runs. These are the three faces of the same problem: text crossing a boundary where Python has to choose an encoding, and choosing one — on Windows — that cannot represent the text. This guide explains exactly when Windows Python uses a legacy code page, and fixes each boundary: standard streams, files, subprocess output and symbols the user's console cannot display. It belongs to the cross-platform terminal compatibility topic.

Prerequisites

• A Python CLI that runs on Windows, or that you want to run there.
• A Windows machine or a Windows CI runner for final verification — the behaviour cannot be reproduced exactly elsewhere, although the tests below simulate it faithfully.

Where the errors come from

Python strings are Unicode. Whenever text leaves the process — to a console, a file, a pipe — it must be encoded into bytes, and whenever it enters — from a file, a pipe, another program — it must be decoded. Each boundary has its own encoding:

• The interactive Windows console is not the problem. Since Python 3.6, writing to a real console uses the Unicode console API, so print("✓") in Windows Terminal or cmd.exe works.
• Redirected standard streams are the problem. When stdout is a file or pipe, Python encodes with the locale's ANSI code page — cp1252 in Western Europe and the Americas, others elsewhere — which covers a couple of hundred characters. Anything outside it raises UnicodeEncodeError by default.
• open() without encoding= uses the same locale encoding on Windows, so files written on one machine are misread on another, and files with characters outside the code page cannot be written at all.
• Subprocess output arrives as bytes in whatever encoding the other program chose — often the OEM code page (cp437, cp850) for older console tools — and text=True decodes it with the locale encoding, which may differ again.

The redirect case is why these bugs escape testing: the author tries the command interactively, it works, and only scripts and CI — which always redirect — hit the crash.

The recipe

Put the encoding decisions in one small module and call it at the very start of the entry point:

# src/mytool/encoding.py
from __future__ import annotations

import sys
from typing import TextIO

SYMBOLS = {"✓": "[ok]", "✗": "[x]", "→": "->", "…": "...", "•": "*"}


def ensure_utf8_streams() -> None:
    """Make stdout/stderr UTF-8 with replacement, whatever the platform decided."""
    for name in ("stdout", "stderr"):
        stream = getattr(sys, name)
        encoding = (getattr(stream, "encoding", None) or "").lower().replace("-", "")
        if encoding != "utf8" and hasattr(stream, "reconfigure"):
            stream.reconfigure(encoding="utf-8", errors="replace")


def can_encode(text: str, stream: TextIO | None = None) -> bool:
    encoding = getattr(stream or sys.stdout, "encoding", None) or "ascii"
    try:
        text.encode(encoding)
    except (UnicodeEncodeError, LookupError):
        return False
    return True


def symbol(char: str, stream: TextIO | None = None) -> str:
    """Return char if the stream can show it, else an ASCII stand-in."""
    return char if can_encode(char, stream) else SYMBOLS.get(char, "?")

# src/mytool/__main__.py (and the console-script entry function)
from mytool.encoding import ensure_utf8_streams


def main() -> None:
    ensure_utf8_streams()          # before anything is printed
    from mytool.cli import app
    app()

What each piece does:

• ensure_utf8_streams() reconfigures stdout and stderr to UTF-8 with errors="replace". reconfigure (Python 3.7+) changes the encoding of an existing text stream in place, before anything has been written. UTF-8 can represent every character, so encoding never fails; errors="replace" guarantees that even a lone surrogate or other malformed string becomes ? rather than a traceback. Consumers reading redirected output — other programs, editors — overwhelmingly expect UTF-8 today.
• symbol() is for decorative characters. When output goes somewhere that genuinely cannot display ✓ — a legacy console font, a stream you chose not to reconfigure — the ASCII stand-in keeps the meaning. Use it for status marks, arrows and bullets, never for user data, which must be passed through faithfully.

Files: always say which encoding

Every open(), Path.read_text() and Path.write_text() in your tool should pass encoding="utf-8":

from pathlib import Path

config = Path("mytool.toml").read_text(encoding="utf-8")
Path("report.csv").write_text(csv_text, encoding="utf-8", newline="")

For files users edit in Windows tools, accept a byte-order mark by reading with encoding="utf-8-sig", which handles files with or without one. For TOML specifically, tomllib requires binary mode (open(path, "rb")) and handles UTF-8 itself, which removes the question entirely — see reading TOML config with tomllib. Ruff's PLW1514 rule flags any text-mode open() without an encoding, which is the easiest way to find them all; see configuring Ruff for a CLI project.

Output from other programs

When your CLI runs other tools, decode their output deliberately:

import subprocess

proc = subprocess.run(["git", "log", "-1", "--format=%an"], capture_output=True, check=True)
author = proc.stdout.decode("utf-8", errors="replace").strip()

Modern tools (git, Go and Rust programs, Python with UTF-8 mode) usually emit UTF-8; older Windows console programs may emit the OEM code page. Decoding as UTF-8 with errors="replace" never crashes and is right for the modern majority; if you wrap a specific legacy tool, decode with its known encoding instead. The subprocess side of this is covered in calling external commands safely with subprocess.

Python's UTF-8 mode

Setting PYTHONUTF8=1 (or running python -X utf8) makes the whole interpreter default to UTF-8 for files and streams on every platform. It is an excellent setting for users and CI, and Python 3.15 is planned to make it the default. But a CLI cannot rely on it: users run whatever Python they have, and launchers generated by pip, pipx and uv do not set it. Explicit encodings in your own code work regardless.

UX considerations

• Never crash on output. A tool that fails because it could not print a check mark has its priorities backwards. Replacement characters are an acceptable worst case; tracebacks are not.
• Keep user data intact. Symbols may fall back to ASCII; names, paths and messages from users must round-trip exactly, which UTF-8 guarantees.
• Do not depend on emoji. Even in UTF-8, older console fonts render many emoji as boxes. Plain symbols (✓, ✗, →) have much better font coverage, and a word next to each keeps meaning clear.
• Document the escape hatch. Mention PYTHONUTF8=1 in troubleshooting docs for users hitting encoding issues in other Python tools too.

Testing the behaviour

You can reproduce the Windows redirect behaviour on any platform by wrapping a bytes buffer in a cp1252 text stream — exactly what Windows Python does for a redirected stdout:

# tests/test_encoding.py
import io
import sys

import pytest

from mytool import encoding


def cp1252_stream() -> io.TextIOWrapper:
    """What Windows gives a redirected stdout on a Western-European locale."""
    return io.TextIOWrapper(io.BytesIO(), encoding="cp1252", errors="strict")


def test_the_original_failure_is_real():
    stream = cp1252_stream()
    with pytest.raises(UnicodeEncodeError):
        print("✓ deployed", file=stream, flush=True)


def test_reconfigure_fixes_it(monkeypatch):
    stream = cp1252_stream()
    monkeypatch.setattr(sys, "stdout", stream)
    encoding.ensure_utf8_streams()
    print("✓ déployé → prod", flush=True)
    assert stream.buffer.getvalue().decode("utf-8") == "✓ déployé → prod\n"


def test_symbol_falls_back_on_narrow_streams():
    assert encoding.symbol("✓", cp1252_stream()) == "[ok]"
    assert encoding.symbol("é", cp1252_stream()) == "é"            # cp1252 has é
    utf8 = io.TextIOWrapper(io.BytesIO(), encoding="utf-8")
    assert encoding.symbol("✓", utf8) == "✓"


def test_reading_bytes_from_another_program():
    raw = "Größe: 5 MB\n".encode("utf-8")
    assert raw.decode("utf-8", errors="replace") == "Größe: 5 MB\n"
    assert "�" in b"\xff broken".decode("utf-8", errors="replace")

The first test is worth keeping even though it tests Python rather than your code: it documents why the fix exists, and it fails loudly if a future Python changes the behaviour. For end-to-end confidence, add a Windows CI job that runs a command with output redirected (mytool status > out.txt) and with PYTHONUTF8 unset, as described in testing a CLI across Python versions with GitHub Actions.

Conclusion

Encoding errors in Python CLIs on Windows come from a small number of boundaries: redirected standard streams, files opened without an encoding, and output from other programs. Reconfigure stdout and stderr to UTF-8 with errors="replace" at the start of the entry point, pass encoding="utf-8" to every file operation, decode subprocess output explicitly, and give decorative symbols ASCII fallbacks. Then simulate a cp1252 stream in tests and run one redirected command on a Windows runner, and the "works on my machine" class of Unicode bugs is closed.

Frequently asked questions

Related

• Up: Cross-platform terminal compatibility
• Respecting NO_COLOR and FORCE_COLOR
• Detecting CI environments and non-interactive shells
• Cross-platform paths with pathlib
• Handling broken pipe and SIGPIPE