Emitting JSON Output for Scripting

The moment someone wants to automate your tool, they parse whatever it prints. A --json flag turns that from a fragile exercise in awk into a documented interface — and moves your human-readable output back into the category of things you are free to improve.

TL;DR

• One envelope shape across every command, not a shape per command.
• Include a version field so consumers can branch when the shape changes.
• Use NDJSON (one object per line) for streams; one document for bounded results.
• JSON goes to stdout; warnings, progress and errors stay on stderr.
• Signal failure with the exit code, not a "status": "error" field.

Prerequisites

A command that produces structured results, and the stream discipline from working with stdin, stdout and pipes: results on stdout, everything else on stderr.

One envelope, every command

A tool where status --json returns a bare list and list --json returns {"items": [...]} forces every consumer to write a parser per command. One shape removes that:

# src/mytool/output.py
import json
import sys
from dataclasses import dataclass, field
from typing import Any

SCHEMA_VERSION = 1

@dataclass
class Envelope:
    data: Any
    warnings: list[str] = field(default_factory=list)

    def emit(self) -> None:
        json.dump(
            {"version": SCHEMA_VERSION, "data": self.data, "warnings": self.warnings},
            sys.stdout,
            default=str,          # dates and paths serialise predictably
        )
        sys.stdout.write("\n")

@app.command()
def status(json_output: Annotated[bool, typer.Option("--json")] = False) -> None:
    rows = core.collect_status()
    if json_output:
        Envelope(data=[row.as_dict() for row in rows]).emit()
        return
    render_table(rows)

The version field costs nothing now and is the only thing that lets you evolve the shape later without breaking every consumer at once. default=str is a small mercy: without it, a datetime or a Path in your data raises TypeError at the worst possible moment.

Streams: NDJSON

For a bounded result — the status of four environments — one document is right, and it can be pretty-printed when a human is reading it. For a stream of many records, newline-delimited JSON is better: one object per line, no enclosing array, no closing bracket to wait for.

@app.command()
def export(
    period: str,
    json_output: Annotated[bool, typer.Option("--json")] = False,
) -> None:
    for row in core.collect(period):              # a generator
        if json_output:
            sys.stdout.write(json.dumps(row.as_dict(), default=str) + "\n")
        else:
            out.print(format_row(row))

Two properties follow immediately. Memory stays flat regardless of how many records there are. And a consumer — jq --unbuffered, another instance of your tool, a script reading line by line — can start working before your command finishes, which matters when the export takes ten minutes.

Say which you emit in the help text. "One JSON object per line" is a one-line promise that saves a consumer an experiment.

Versioning the shape

The JSON output is an interface with the same rules as your flags. Some changes are safe and some are not:

Adding a key is safe. Consumers that do not know about it ignore it, which is why documenting "new keys may be added" up front is worth doing — it makes the addition unambiguously non-breaking.

Renaming or removing a key is breaking. Deprecate the way you would a flag: emit both for a release or two, note it in the changelog, then drop the old one at a major version.

Changing a value's type or units is the worst case, exactly as with flags — nothing looks different and every consumer silently misreads it. Add a new key instead.

When the shape does need to change incompatibly, the version field is what lets a consumer cope:

SCHEMA_VERSION = 2      # data.items[] gained "region"; "location" removed

Bump it, document what changed in the changelog, and — if the change is significant — consider supporting --json-version 1 for a transition period.

What does not belong in the JSON

Two things people put in the payload that belong elsewhere.

Failure status. A shell script branches on the exit code, not on a field:

if mytool status --json > status.json; then …

A command that exits 0 with {"status": "error"} breaks that idiom and every set -e script in the world. Use the exit code, and put the message on stderr.

Progress and warnings. Progress belongs on stderr. Warnings may appear in the envelope's warnings list — machine-readable and useful — but they should also be printed on stderr for the human, and neither should ever appear on stdout as loose text.

err.print(f"[yellow]{len(skipped)} records skipped[/]")            # for the human
Envelope(data=rows, warnings=[f"{len(skipped)} records skipped"]).emit()   # for the script

UX considerations

Pretty-print for humans, compact for pipes. When stdout is a terminal, indented JSON with colour is much easier to read; when it is piped, compact output is smaller and faster to parse. Rich's print_json handles the terminal case, and the decision follows the same isatty check as everything else.

Keep key names stable and boring. created_at rather than createdAt or ctime; the same name for the same concept in every command. Consumers write code against these strings, and consistency is worth more than elegance.

Sort keys when the output might be diffed. json.dumps(..., sort_keys=True) makes two runs comparable, which matters when people use your export to detect drift between environments.

Document one example. A single realistic document in the help epilog or the reference tells a consumer more than a schema description of the same size.

Testing the behaviour

The valuable assertion is that stdout parses on its own — because that is what breaks when somebody adds a stray print:

def test_json_output_is_parseable_and_stdout_is_clean(cli):
    result = cli.invoke(app, ["status", "--json"])

    assert result.exit_code == 0
    payload = json.loads(result.stdout)          # fails loudly on any non-JSON line
    assert payload["version"] == 1
    assert isinstance(payload["data"], list)

def test_progress_does_not_contaminate_stdout(cli):
    result = cli.invoke(app, ["export", "7d", "--json"])

    for line in result.stdout.splitlines():
        json.loads(line)                          # NDJSON: every line stands alone
    assert "exported" in result.stderr

def test_failure_uses_the_exit_code_not_a_field(cli):
    result = cli.invoke(app, ["status", "--json"], env={"MYTOOL_ENDPOINT": "http://127.0.0.1:1"})

    assert result.exit_code == 69
    assert result.stdout == ""                    # no half-written payload

That last test encodes a decision worth making explicitly: on failure, emit nothing on stdout rather than a partial document. A consumer that receives half a JSON object cannot tell whether the data was incomplete or the parse failed.

Conclusion

A --json flag is a small feature with a large effect: it gives scripts a stable interface and gives you back the freedom to change human-readable output. Keep one envelope, version it, choose NDJSON for streams, keep stdout clean, and let the exit code carry failure.

Frequently asked questions

Related

• Up: Working with stdin, stdout and pipes
• Sideways: Reading piped input in Python CLIs
• Sideways: Detecting a TTY and adapting output
• Related: Structured JSON logging in Python CLIs
• Related: Versioning and deprecating CLI flags