[{"data":1,"prerenderedAt":3756},["ShallowReactive",2],{"page-\u002Fadvanced-input-parsing-user-experience\u002Fworking-with-stdin-stdout-and-pipes\u002Fprocessing-large-files-and-ndjson-streams\u002F":3,"content-directory":3209},{"id":4,"title":5,"body":6,"date":3193,"description":3194,"difficulty":3195,"draft":3196,"extension":3197,"meta":3198,"navigation":152,"path":3199,"seo":3200,"stem":3201,"tags":3202,"updated":3193,"__hash__":3208},"content\u002Fadvanced-input-parsing-user-experience\u002Fworking-with-stdin-stdout-and-pipes\u002Fprocessing-large-files-and-ndjson-streams\u002Findex.md","Processing Large Files and NDJSON Streams in Python CLIs",{"type":7,"value":8,"toc":3174},"minimark",[9,48,53,67,71,75,86,89,92,96,111,1232,1239,1789,1792,1881,1886,1904,1914,1933,1942,1948,1952,1955,1970,1974,2038,2042,2049,3012,3026,3030,3045,3049,3053,3067,3071,3084,3088,3099,3103,3121,3125,3137,3141,3170],[10,11,12,13,17,18,21,22,25,26,29,30,33,34,37,38,41,42,47],"p",{},"Event exports, audit logs, API dumps and log archives increasingly arrive as NDJSON — newline-delimited JSON, one object per line, often gzipped and often far larger than the laptop it is being inspected on. A CLI that handles them with ",[14,15,16],"code",{},"json.load(f)"," or ",[14,19,20],{},"f.read().splitlines()"," works on the sample file and then dies on the real one, taking several times the file's size in memory before it prints anything. The alternative costs nothing in code: read one line, handle it, write the result, repeat. Memory stays flat whatever the input size, the first results appear immediately, and the command composes with ",[14,23,24],{},"head",", ",[14,27,28],{},"grep"," and ",[14,31,32],{},"jq"," in a pipeline. This guide builds a small ",[14,35,36],{},"filter"," command that streams NDJSON from files, ",[14,39,40],{},".gz"," archives or stdin, reports malformed lines precisely, offers an explicit lenient mode, and proves with a test that its memory use does not grow with the input. It belongs to the ",[43,44,46],"a",{"href":45},"\u002Fadvanced-input-parsing-user-experience\u002Fworking-with-stdin-stdout-and-pipes\u002F","working with stdin, stdout and pipes topic",".",[49,50,52],"h2",{"id":51},"prerequisites","Prerequisites",[54,55,56,60],"ul",{},[57,58,59],"li",{},"Python 3.10+ and a Typer or Click CLI.",[57,61,62,63,47],{},"Familiarity with ",[43,64,66],{"href":65},"\u002Fadvanced-input-parsing-user-experience\u002Fworking-with-stdin-stdout-and-pipes\u002Freading-piped-input-in-python-clis\u002F","reading piped input in Python CLIs",[49,68,70],{"id":69},"stream-do-not-load","Stream, do not load",[72,73],"inline-diagram",{"name":74},"nd-stream",[10,76,77,78,81,82,85],{},"A Python file object is an iterator over lines, reading from disk in buffered chunks. ",[14,79,80],{},"for line in fh"," therefore holds one line in memory at a time, however big the file. Parse it with ",[14,83,84],{},"json.loads",", decide whether it matches, write the result straight to stdout, and let the object go. Every stage of the command is a generator or a loop, and nothing ever builds a list of all records.",[72,87],{"name":88},"nd-bars",[10,90,91],{},"The difference is not a percentage but orders of magnitude. Loading a file as text costs its size again as a Python string; parsing everything into dictionaries costs several times more, because every key, value and dictionary carries object overhead. Streaming costs the size of the longest line plus buffers. The test at the end of this guide measures it: a peak of about 25 KB while filtering an 18 MB file.",[49,93,95],{"id":94},"the-recipe","The recipe",[10,97,98,99,102,103,106,107,110],{},"The streaming logic lives in a module that knows nothing about the CLI framework. ",[14,100,101],{},"open_input"," handles the three kinds of input, ",[14,104,105],{},"records"," parses lines lazily and deals with bad ones, and ",[14,108,109],{},"filter_records"," connects them to an output stream:",[112,113,118],"pre",{"className":114,"code":115,"language":116,"meta":117,"style":117},"language-python shiki shiki-themes github-light github-dark","# src\u002Fmytool\u002Fstream.py\nfrom __future__ import annotations\n\nimport gzip\nimport io\nimport json\nimport sys\nfrom collections.abc import Iterator\nfrom contextlib import contextmanager\nfrom dataclasses import dataclass, field\nfrom pathlib import Path\nfrom typing import IO, Any\n\n\nclass BadRecord(Exception):\n    def __init__(self, source: str, line_no: int, reason: str) -> None:\n        super().__init__(f\"{source}:{line_no}: {reason}\")\n        self.source, self.line_no, self.reason = source, line_no, reason\n\n\n@dataclass\nclass Stats:\n    read: int = 0\n    written: int = 0\n    skipped: int = 0\n    first_errors: list[str] = field(default_factory=list)\n\n\n@contextmanager\ndef open_input(name: str) -> Iterator[IO[str]]:\n    \"\"\"'-' is stdin; .gz files are decompressed on the fly. Text, UTF-8, read lazily.\"\"\"\n    if name == \"-\":\n        fh = io.TextIOWrapper(sys.stdin.buffer, encoding=\"utf-8\")\n        try:\n            yield fh\n        finally:\n            fh.detach()                     # leave the real stdin open\n    elif name.endswith(\".gz\"):\n        with gzip.open(name, \"rt\", encoding=\"utf-8\") as fh:\n            yield fh\n    else:\n        with Path(name).open(encoding=\"utf-8\") as fh:\n            yield fh\n\n\ndef records(fh: IO[str], source: str, *, skip_invalid: bool, stats: Stats) -> Iterator[dict[str, Any]]:\n    for line_no, line in enumerate(fh, start=1):\n        if not line.strip():\n            continue\n        stats.read += 1\n        try:\n            obj = json.loads(line)\n            if not isinstance(obj, dict):\n                raise ValueError(\"expected a JSON object\")\n        except ValueError as exc:          # json.JSONDecodeError is a ValueError\n            if not skip_invalid:\n                raise BadRecord(source, line_no, str(exc)) from None\n            stats.skipped += 1\n            if len(stats.first_errors) \u003C 5:\n                stats.first_errors.append(f\"{source}:{line_no}: {exc}\")\n            continue\n        yield obj\n\n\ndef filter_records(inputs: list[str], out: IO[str], *, where: dict[str, Any],\n                   fields: list[str] | None, skip_invalid: bool) -> Stats:\n    stats = Stats()\n    for name in inputs:\n        with open_input(name) as fh:\n            for obj in records(fh, name, skip_invalid=skip_invalid, stats=stats):\n                if all(obj.get(k) == v for k, v in where.items()):\n                    if fields:\n                        obj = {k: obj.get(k) for k in fields}\n                    out.write(json.dumps(obj, separators=(\",\", \":\")) + \"\\n\")\n                    stats.written += 1\n    return stats\n","python","",[14,119,120,129,147,154,163,171,179,187,200,213,226,239,255,260,265,284,319,374,400,405,410,416,426,440,452,464,491,496,501,507,535,541,558,579,587,596,604,613,627,656,663,671,691,698,703,708,749,777,789,795,807,814,825,844,860,877,887,905,915,934,971,976,985,990,995,1030,1053,1064,1076,1088,1118,1146,1155,1176,1213,1223],{"__ignoreMap":117},[121,122,125],"span",{"class":123,"line":124},"line",1,[121,126,128],{"class":127},"sJ8bj","# src\u002Fmytool\u002Fstream.py\n",[121,130,132,136,140,143],{"class":123,"line":131},2,[121,133,135],{"class":134},"szBVR","from",[121,137,139],{"class":138},"sj4cs"," __future__",[121,141,142],{"class":134}," import",[121,144,146],{"class":145},"sVt8B"," annotations\n",[121,148,150],{"class":123,"line":149},3,[121,151,153],{"emptyLinePlaceholder":152},true,"\n",[121,155,157,160],{"class":123,"line":156},4,[121,158,159],{"class":134},"import",[121,161,162],{"class":145}," gzip\n",[121,164,166,168],{"class":123,"line":165},5,[121,167,159],{"class":134},[121,169,170],{"class":145}," io\n",[121,172,174,176],{"class":123,"line":173},6,[121,175,159],{"class":134},[121,177,178],{"class":145}," json\n",[121,180,182,184],{"class":123,"line":181},7,[121,183,159],{"class":134},[121,185,186],{"class":145}," sys\n",[121,188,190,192,195,197],{"class":123,"line":189},8,[121,191,135],{"class":134},[121,193,194],{"class":145}," collections.abc ",[121,196,159],{"class":134},[121,198,199],{"class":145}," Iterator\n",[121,201,203,205,208,210],{"class":123,"line":202},9,[121,204,135],{"class":134},[121,206,207],{"class":145}," contextlib ",[121,209,159],{"class":134},[121,211,212],{"class":145}," contextmanager\n",[121,214,216,218,221,223],{"class":123,"line":215},10,[121,217,135],{"class":134},[121,219,220],{"class":145}," dataclasses ",[121,222,159],{"class":134},[121,224,225],{"class":145}," dataclass, field\n",[121,227,229,231,234,236],{"class":123,"line":228},11,[121,230,135],{"class":134},[121,232,233],{"class":145}," pathlib ",[121,235,159],{"class":134},[121,237,238],{"class":145}," Path\n",[121,240,242,244,247,249,252],{"class":123,"line":241},12,[121,243,135],{"class":134},[121,245,246],{"class":145}," typing ",[121,248,159],{"class":134},[121,250,251],{"class":138}," IO",[121,253,254],{"class":145},", Any\n",[121,256,258],{"class":123,"line":257},13,[121,259,153],{"emptyLinePlaceholder":152},[121,261,263],{"class":123,"line":262},14,[121,264,153],{"emptyLinePlaceholder":152},[121,266,268,271,275,278,281],{"class":123,"line":267},15,[121,269,270],{"class":134},"class",[121,272,274],{"class":273},"sScJk"," BadRecord",[121,276,277],{"class":145},"(",[121,279,280],{"class":138},"Exception",[121,282,283],{"class":145},"):\n",[121,285,287,290,293,296,299,302,305,308,310,313,316],{"class":123,"line":286},16,[121,288,289],{"class":134},"    def",[121,291,292],{"class":138}," __init__",[121,294,295],{"class":145},"(self, source: ",[121,297,298],{"class":138},"str",[121,300,301],{"class":145},", line_no: ",[121,303,304],{"class":138},"int",[121,306,307],{"class":145},", reason: ",[121,309,298],{"class":138},[121,311,312],{"class":145},") -> ",[121,314,315],{"class":138},"None",[121,317,318],{"class":145},":\n",[121,320,322,325,328,331,333,336,340,343,346,349,352,354,357,359,362,364,367,369,371],{"class":123,"line":321},17,[121,323,324],{"class":138},"        super",[121,326,327],{"class":145},"().",[121,329,330],{"class":138},"__init__",[121,332,277],{"class":145},[121,334,335],{"class":134},"f",[121,337,339],{"class":338},"sZZnC","\"",[121,341,342],{"class":138},"{",[121,344,345],{"class":145},"source",[121,347,348],{"class":138},"}",[121,350,351],{"class":338},":",[121,353,342],{"class":138},[121,355,356],{"class":145},"line_no",[121,358,348],{"class":138},[121,360,361],{"class":338},": ",[121,363,342],{"class":138},[121,365,366],{"class":145},"reason",[121,368,348],{"class":138},[121,370,339],{"class":338},[121,372,373],{"class":145},")\n",[121,375,377,380,383,386,389,391,394,397],{"class":123,"line":376},18,[121,378,379],{"class":138},"        self",[121,381,382],{"class":145},".source, ",[121,384,385],{"class":138},"self",[121,387,388],{"class":145},".line_no, ",[121,390,385],{"class":138},[121,392,393],{"class":145},".reason ",[121,395,396],{"class":134},"=",[121,398,399],{"class":145}," source, line_no, reason\n",[121,401,403],{"class":123,"line":402},19,[121,404,153],{"emptyLinePlaceholder":152},[121,406,408],{"class":123,"line":407},20,[121,409,153],{"emptyLinePlaceholder":152},[121,411,413],{"class":123,"line":412},21,[121,414,415],{"class":273},"@dataclass\n",[121,417,419,421,424],{"class":123,"line":418},22,[121,420,270],{"class":134},[121,422,423],{"class":273}," Stats",[121,425,318],{"class":145},[121,427,429,432,434,437],{"class":123,"line":428},23,[121,430,431],{"class":145},"    read: ",[121,433,304],{"class":138},[121,435,436],{"class":134}," =",[121,438,439],{"class":138}," 0\n",[121,441,443,446,448,450],{"class":123,"line":442},24,[121,444,445],{"class":145},"    written: ",[121,447,304],{"class":138},[121,449,436],{"class":134},[121,451,439],{"class":138},[121,453,455,458,460,462],{"class":123,"line":454},25,[121,456,457],{"class":145},"    skipped: ",[121,459,304],{"class":138},[121,461,436],{"class":134},[121,463,439],{"class":138},[121,465,467,470,472,475,477,480,484,486,489],{"class":123,"line":466},26,[121,468,469],{"class":145},"    first_errors: list[",[121,471,298],{"class":138},[121,473,474],{"class":145},"] ",[121,476,396],{"class":134},[121,478,479],{"class":145}," field(",[121,481,483],{"class":482},"s4XuR","default_factory",[121,485,396],{"class":134},[121,487,488],{"class":138},"list",[121,490,373],{"class":145},[121,492,494],{"class":123,"line":493},27,[121,495,153],{"emptyLinePlaceholder":152},[121,497,499],{"class":123,"line":498},28,[121,500,153],{"emptyLinePlaceholder":152},[121,502,504],{"class":123,"line":503},29,[121,505,506],{"class":273},"@contextmanager\n",[121,508,510,513,516,519,521,524,527,530,532],{"class":123,"line":509},30,[121,511,512],{"class":134},"def",[121,514,515],{"class":273}," open_input",[121,517,518],{"class":145},"(name: ",[121,520,298],{"class":138},[121,522,523],{"class":145},") -> Iterator[",[121,525,526],{"class":138},"IO",[121,528,529],{"class":145},"[",[121,531,298],{"class":138},[121,533,534],{"class":145},"]]:\n",[121,536,538],{"class":123,"line":537},31,[121,539,540],{"class":338},"    \"\"\"'-' is stdin; .gz files are decompressed on the fly. Text, UTF-8, read lazily.\"\"\"\n",[121,542,544,547,550,553,556],{"class":123,"line":543},32,[121,545,546],{"class":134},"    if",[121,548,549],{"class":145}," name ",[121,551,552],{"class":134},"==",[121,554,555],{"class":338}," \"-\"",[121,557,318],{"class":145},[121,559,561,564,566,569,572,574,577],{"class":123,"line":560},33,[121,562,563],{"class":145},"        fh ",[121,565,396],{"class":134},[121,567,568],{"class":145}," io.TextIOWrapper(sys.stdin.buffer, ",[121,570,571],{"class":482},"encoding",[121,573,396],{"class":134},[121,575,576],{"class":338},"\"utf-8\"",[121,578,373],{"class":145},[121,580,582,585],{"class":123,"line":581},34,[121,583,584],{"class":134},"        try",[121,586,318],{"class":145},[121,588,590,593],{"class":123,"line":589},35,[121,591,592],{"class":134},"            yield",[121,594,595],{"class":145}," fh\n",[121,597,599,602],{"class":123,"line":598},36,[121,600,601],{"class":134},"        finally",[121,603,318],{"class":145},[121,605,607,610],{"class":123,"line":606},37,[121,608,609],{"class":145},"            fh.detach()                     ",[121,611,612],{"class":127},"# leave the real stdin open\n",[121,614,616,619,622,625],{"class":123,"line":615},38,[121,617,618],{"class":134},"    elif",[121,620,621],{"class":145}," name.endswith(",[121,623,624],{"class":338},"\".gz\"",[121,626,283],{"class":145},[121,628,630,633,636,639,641,643,645,647,650,653],{"class":123,"line":629},39,[121,631,632],{"class":134},"        with",[121,634,635],{"class":145}," gzip.open(name, ",[121,637,638],{"class":338},"\"rt\"",[121,640,25],{"class":145},[121,642,571],{"class":482},[121,644,396],{"class":134},[121,646,576],{"class":338},[121,648,649],{"class":145},") ",[121,651,652],{"class":134},"as",[121,654,655],{"class":145}," fh:\n",[121,657,659,661],{"class":123,"line":658},40,[121,660,592],{"class":134},[121,662,595],{"class":145},[121,664,666,669],{"class":123,"line":665},41,[121,667,668],{"class":134},"    else",[121,670,318],{"class":145},[121,672,674,676,679,681,683,685,687,689],{"class":123,"line":673},42,[121,675,632],{"class":134},[121,677,678],{"class":145}," Path(name).open(",[121,680,571],{"class":482},[121,682,396],{"class":134},[121,684,576],{"class":338},[121,686,649],{"class":145},[121,688,652],{"class":134},[121,690,655],{"class":145},[121,692,694,696],{"class":123,"line":693},43,[121,695,592],{"class":134},[121,697,595],{"class":145},[121,699,701],{"class":123,"line":700},44,[121,702,153],{"emptyLinePlaceholder":152},[121,704,706],{"class":123,"line":705},45,[121,707,153],{"emptyLinePlaceholder":152},[121,709,711,713,716,719,721,723,725,728,730,732,735,738,741,744,746],{"class":123,"line":710},46,[121,712,512],{"class":134},[121,714,715],{"class":273}," records",[121,717,718],{"class":145},"(fh: ",[121,720,526],{"class":138},[121,722,529],{"class":145},[121,724,298],{"class":138},[121,726,727],{"class":145},"], source: ",[121,729,298],{"class":138},[121,731,25],{"class":145},[121,733,734],{"class":134},"*",[121,736,737],{"class":145},", skip_invalid: ",[121,739,740],{"class":138},"bool",[121,742,743],{"class":145},", stats: Stats) -> Iterator[dict[",[121,745,298],{"class":138},[121,747,748],{"class":145},", Any]]:\n",[121,750,752,755,758,761,764,767,770,772,775],{"class":123,"line":751},47,[121,753,754],{"class":134},"    for",[121,756,757],{"class":145}," line_no, line ",[121,759,760],{"class":134},"in",[121,762,763],{"class":138}," enumerate",[121,765,766],{"class":145},"(fh, ",[121,768,769],{"class":482},"start",[121,771,396],{"class":134},[121,773,774],{"class":138},"1",[121,776,283],{"class":145},[121,778,780,783,786],{"class":123,"line":779},48,[121,781,782],{"class":134},"        if",[121,784,785],{"class":134}," not",[121,787,788],{"class":145}," line.strip():\n",[121,790,792],{"class":123,"line":791},49,[121,793,794],{"class":134},"            continue\n",[121,796,798,801,804],{"class":123,"line":797},50,[121,799,800],{"class":145},"        stats.read ",[121,802,803],{"class":134},"+=",[121,805,806],{"class":138}," 1\n",[121,808,810,812],{"class":123,"line":809},51,[121,811,584],{"class":134},[121,813,318],{"class":145},[121,815,817,820,822],{"class":123,"line":816},52,[121,818,819],{"class":145},"            obj ",[121,821,396],{"class":134},[121,823,824],{"class":145}," json.loads(line)\n",[121,826,828,831,833,836,839,842],{"class":123,"line":827},53,[121,829,830],{"class":134},"            if",[121,832,785],{"class":134},[121,834,835],{"class":138}," isinstance",[121,837,838],{"class":145},"(obj, ",[121,840,841],{"class":138},"dict",[121,843,283],{"class":145},[121,845,847,850,853,855,858],{"class":123,"line":846},54,[121,848,849],{"class":134},"                raise",[121,851,852],{"class":138}," ValueError",[121,854,277],{"class":145},[121,856,857],{"class":338},"\"expected a JSON object\"",[121,859,373],{"class":145},[121,861,863,866,868,871,874],{"class":123,"line":862},55,[121,864,865],{"class":134},"        except",[121,867,852],{"class":138},[121,869,870],{"class":134}," as",[121,872,873],{"class":145}," exc:          ",[121,875,876],{"class":127},"# json.JSONDecodeError is a ValueError\n",[121,878,880,882,884],{"class":123,"line":879},56,[121,881,830],{"class":134},[121,883,785],{"class":134},[121,885,886],{"class":145}," skip_invalid:\n",[121,888,890,892,895,897,900,902],{"class":123,"line":889},57,[121,891,849],{"class":134},[121,893,894],{"class":145}," BadRecord(source, line_no, ",[121,896,298],{"class":138},[121,898,899],{"class":145},"(exc)) ",[121,901,135],{"class":134},[121,903,904],{"class":138}," None\n",[121,906,908,911,913],{"class":123,"line":907},58,[121,909,910],{"class":145},"            stats.skipped ",[121,912,803],{"class":134},[121,914,806],{"class":138},[121,916,918,920,923,926,929,932],{"class":123,"line":917},59,[121,919,830],{"class":134},[121,921,922],{"class":138}," len",[121,924,925],{"class":145},"(stats.first_errors) ",[121,927,928],{"class":134},"\u003C",[121,930,931],{"class":138}," 5",[121,933,318],{"class":145},[121,935,937,940,942,944,946,948,950,952,954,956,958,960,962,965,967,969],{"class":123,"line":936},60,[121,938,939],{"class":145},"                stats.first_errors.append(",[121,941,335],{"class":134},[121,943,339],{"class":338},[121,945,342],{"class":138},[121,947,345],{"class":145},[121,949,348],{"class":138},[121,951,351],{"class":338},[121,953,342],{"class":138},[121,955,356],{"class":145},[121,957,348],{"class":138},[121,959,361],{"class":338},[121,961,342],{"class":138},[121,963,964],{"class":145},"exc",[121,966,348],{"class":138},[121,968,339],{"class":338},[121,970,373],{"class":145},[121,972,974],{"class":123,"line":973},61,[121,975,794],{"class":134},[121,977,979,982],{"class":123,"line":978},62,[121,980,981],{"class":134},"        yield",[121,983,984],{"class":145}," obj\n",[121,986,988],{"class":123,"line":987},63,[121,989,153],{"emptyLinePlaceholder":152},[121,991,993],{"class":123,"line":992},64,[121,994,153],{"emptyLinePlaceholder":152},[121,996,998,1000,1003,1006,1008,1011,1013,1015,1017,1020,1022,1025,1027],{"class":123,"line":997},65,[121,999,512],{"class":134},[121,1001,1002],{"class":273}," filter_records",[121,1004,1005],{"class":145},"(inputs: list[",[121,1007,298],{"class":138},[121,1009,1010],{"class":145},"], out: ",[121,1012,526],{"class":138},[121,1014,529],{"class":145},[121,1016,298],{"class":138},[121,1018,1019],{"class":145},"], ",[121,1021,734],{"class":134},[121,1023,1024],{"class":145},", where: dict[",[121,1026,298],{"class":138},[121,1028,1029],{"class":145},", Any],\n",[121,1031,1033,1036,1038,1040,1043,1046,1048,1050],{"class":123,"line":1032},66,[121,1034,1035],{"class":145},"                   fields: list[",[121,1037,298],{"class":138},[121,1039,474],{"class":145},[121,1041,1042],{"class":134},"|",[121,1044,1045],{"class":138}," None",[121,1047,737],{"class":145},[121,1049,740],{"class":138},[121,1051,1052],{"class":145},") -> Stats:\n",[121,1054,1056,1059,1061],{"class":123,"line":1055},67,[121,1057,1058],{"class":145},"    stats ",[121,1060,396],{"class":134},[121,1062,1063],{"class":145}," Stats()\n",[121,1065,1067,1069,1071,1073],{"class":123,"line":1066},68,[121,1068,754],{"class":134},[121,1070,549],{"class":145},[121,1072,760],{"class":134},[121,1074,1075],{"class":145}," inputs:\n",[121,1077,1079,1081,1084,1086],{"class":123,"line":1078},69,[121,1080,632],{"class":134},[121,1082,1083],{"class":145}," open_input(name) ",[121,1085,652],{"class":134},[121,1087,655],{"class":145},[121,1089,1091,1094,1097,1099,1102,1105,1107,1110,1113,1115],{"class":123,"line":1090},70,[121,1092,1093],{"class":134},"            for",[121,1095,1096],{"class":145}," obj ",[121,1098,760],{"class":134},[121,1100,1101],{"class":145}," records(fh, name, ",[121,1103,1104],{"class":482},"skip_invalid",[121,1106,396],{"class":134},[121,1108,1109],{"class":145},"skip_invalid, ",[121,1111,1112],{"class":482},"stats",[121,1114,396],{"class":134},[121,1116,1117],{"class":145},"stats):\n",[121,1119,1121,1124,1127,1130,1132,1135,1138,1141,1143],{"class":123,"line":1120},71,[121,1122,1123],{"class":134},"                if",[121,1125,1126],{"class":138}," all",[121,1128,1129],{"class":145},"(obj.get(k) ",[121,1131,552],{"class":134},[121,1133,1134],{"class":145}," v ",[121,1136,1137],{"class":134},"for",[121,1139,1140],{"class":145}," k, v ",[121,1142,760],{"class":134},[121,1144,1145],{"class":145}," where.items()):\n",[121,1147,1149,1152],{"class":123,"line":1148},72,[121,1150,1151],{"class":134},"                    if",[121,1153,1154],{"class":145}," fields:\n",[121,1156,1158,1161,1163,1166,1168,1171,1173],{"class":123,"line":1157},73,[121,1159,1160],{"class":145},"                        obj ",[121,1162,396],{"class":134},[121,1164,1165],{"class":145}," {k: obj.get(k) ",[121,1167,1137],{"class":134},[121,1169,1170],{"class":145}," k ",[121,1172,760],{"class":134},[121,1174,1175],{"class":145}," fields}\n",[121,1177,1179,1182,1185,1187,1189,1192,1194,1197,1200,1203,1206,1209,1211],{"class":123,"line":1178},74,[121,1180,1181],{"class":145},"                    out.write(json.dumps(obj, ",[121,1183,1184],{"class":482},"separators",[121,1186,396],{"class":134},[121,1188,277],{"class":145},[121,1190,1191],{"class":338},"\",\"",[121,1193,25],{"class":145},[121,1195,1196],{"class":338},"\":\"",[121,1198,1199],{"class":145},")) ",[121,1201,1202],{"class":134},"+",[121,1204,1205],{"class":338}," \"",[121,1207,1208],{"class":138},"\\n",[121,1210,339],{"class":338},[121,1212,373],{"class":145},[121,1214,1216,1219,1221],{"class":123,"line":1215},75,[121,1217,1218],{"class":145},"                    stats.written ",[121,1220,803],{"class":134},[121,1222,806],{"class":138},[121,1224,1226,1229],{"class":123,"line":1225},76,[121,1227,1228],{"class":134},"    return",[121,1230,1231],{"class":145}," stats\n",[10,1233,1234,1235,1238],{},"The command is a thin wrapper that parses options, calls the streaming function with ",[14,1236,1237],{},"sys.stdout",", and turns problems into messages and exit codes:",[112,1240,1242],{"className":114,"code":1241,"language":116,"meta":117,"style":117},"# src\u002Fmytool\u002Fcli.py\nfrom __future__ import annotations\n\nimport sys\nfrom typing import Annotated\n\nimport typer\n\nfrom mytool.stream import BadRecord, filter_records\n\napp = typer.Typer()\n\n\n@app.callback()\ndef main() -> None:\n    \"\"\"Work with NDJSON event files.\"\"\"\n\n\n@app.command(\"filter\")\ndef filter_cmd(\n    inputs: Annotated[list[str], typer.Argument(help=\"NDJSON files (.gz ok); '-' for stdin.\")] = None,\n    where: Annotated[list[str], typer.Option(\"--where\", help=\"KEY=VALUE to match.\")] = None,\n    fields: Annotated[str, typer.Option(\"--fields\", help=\"Comma-separated keys to keep.\")] = \"\",\n    skip_invalid: Annotated[bool, typer.Option(\"--skip-invalid\", help=\"Skip malformed lines.\")] = False,\n) -> None:\n    \"\"\"Stream matching records to stdout, one JSON object per line.\"\"\"\n    conditions = dict(w.split(\"=\", 1) for w in (where or []))\n    try:\n        stats = filter_records(inputs or [\"-\"], sys.stdout, where=conditions,\n                               fields=[f for f in fields.split(\",\") if f] or None,\n                               skip_invalid=skip_invalid)\n    except BadRecord as exc:\n        typer.echo(f\"error: {exc} (use --skip-invalid to skip bad lines)\", err=True)\n        raise typer.Exit(65) from None          # EX_DATAERR\n    if stats.skipped:\n        typer.echo(f\"skipped {stats.skipped} of {stats.read} lines:\", err=True)\n        for line in stats.first_errors:\n            typer.echo(f\"  {line}\", err=True)\n",[14,1243,1244,1249,1259,1263,1269,1280,1284,1291,1295,1307,1311,1321,1325,1329,1337,1351,1356,1360,1364,1376,1386,1414,1444,1475,1505,1513,1518,1556,1563,1592,1628,1638,1651,1682,1702,1709,1748,1761],{"__ignoreMap":117},[121,1245,1246],{"class":123,"line":124},[121,1247,1248],{"class":127},"# src\u002Fmytool\u002Fcli.py\n",[121,1250,1251,1253,1255,1257],{"class":123,"line":131},[121,1252,135],{"class":134},[121,1254,139],{"class":138},[121,1256,142],{"class":134},[121,1258,146],{"class":145},[121,1260,1261],{"class":123,"line":149},[121,1262,153],{"emptyLinePlaceholder":152},[121,1264,1265,1267],{"class":123,"line":156},[121,1266,159],{"class":134},[121,1268,186],{"class":145},[121,1270,1271,1273,1275,1277],{"class":123,"line":165},[121,1272,135],{"class":134},[121,1274,246],{"class":145},[121,1276,159],{"class":134},[121,1278,1279],{"class":145}," Annotated\n",[121,1281,1282],{"class":123,"line":173},[121,1283,153],{"emptyLinePlaceholder":152},[121,1285,1286,1288],{"class":123,"line":181},[121,1287,159],{"class":134},[121,1289,1290],{"class":145}," typer\n",[121,1292,1293],{"class":123,"line":189},[121,1294,153],{"emptyLinePlaceholder":152},[121,1296,1297,1299,1302,1304],{"class":123,"line":202},[121,1298,135],{"class":134},[121,1300,1301],{"class":145}," mytool.stream ",[121,1303,159],{"class":134},[121,1305,1306],{"class":145}," BadRecord, filter_records\n",[121,1308,1309],{"class":123,"line":215},[121,1310,153],{"emptyLinePlaceholder":152},[121,1312,1313,1316,1318],{"class":123,"line":228},[121,1314,1315],{"class":145},"app ",[121,1317,396],{"class":134},[121,1319,1320],{"class":145}," typer.Typer()\n",[121,1322,1323],{"class":123,"line":241},[121,1324,153],{"emptyLinePlaceholder":152},[121,1326,1327],{"class":123,"line":257},[121,1328,153],{"emptyLinePlaceholder":152},[121,1330,1331,1334],{"class":123,"line":262},[121,1332,1333],{"class":273},"@app.callback",[121,1335,1336],{"class":145},"()\n",[121,1338,1339,1341,1344,1347,1349],{"class":123,"line":267},[121,1340,512],{"class":134},[121,1342,1343],{"class":273}," main",[121,1345,1346],{"class":145},"() -> ",[121,1348,315],{"class":138},[121,1350,318],{"class":145},[121,1352,1353],{"class":123,"line":286},[121,1354,1355],{"class":338},"    \"\"\"Work with NDJSON event files.\"\"\"\n",[121,1357,1358],{"class":123,"line":321},[121,1359,153],{"emptyLinePlaceholder":152},[121,1361,1362],{"class":123,"line":376},[121,1363,153],{"emptyLinePlaceholder":152},[121,1365,1366,1369,1371,1374],{"class":123,"line":402},[121,1367,1368],{"class":273},"@app.command",[121,1370,277],{"class":145},[121,1372,1373],{"class":338},"\"filter\"",[121,1375,373],{"class":145},[121,1377,1378,1380,1383],{"class":123,"line":407},[121,1379,512],{"class":134},[121,1381,1382],{"class":273}," filter_cmd",[121,1384,1385],{"class":145},"(\n",[121,1387,1388,1391,1393,1396,1399,1401,1404,1407,1409,1411],{"class":123,"line":412},[121,1389,1390],{"class":145},"    inputs: Annotated[list[",[121,1392,298],{"class":138},[121,1394,1395],{"class":145},"], typer.Argument(",[121,1397,1398],{"class":482},"help",[121,1400,396],{"class":134},[121,1402,1403],{"class":338},"\"NDJSON files (.gz ok); '-' for stdin.\"",[121,1405,1406],{"class":145},")] ",[121,1408,396],{"class":134},[121,1410,1045],{"class":138},[121,1412,1413],{"class":145},",\n",[121,1415,1416,1419,1421,1424,1427,1429,1431,1433,1436,1438,1440,1442],{"class":123,"line":418},[121,1417,1418],{"class":145},"    where: Annotated[list[",[121,1420,298],{"class":138},[121,1422,1423],{"class":145},"], typer.Option(",[121,1425,1426],{"class":338},"\"--where\"",[121,1428,25],{"class":145},[121,1430,1398],{"class":482},[121,1432,396],{"class":134},[121,1434,1435],{"class":338},"\"KEY=VALUE to match.\"",[121,1437,1406],{"class":145},[121,1439,396],{"class":134},[121,1441,1045],{"class":138},[121,1443,1413],{"class":145},[121,1445,1446,1449,1451,1454,1457,1459,1461,1463,1466,1468,1470,1473],{"class":123,"line":428},[121,1447,1448],{"class":145},"    fields: Annotated[",[121,1450,298],{"class":138},[121,1452,1453],{"class":145},", typer.Option(",[121,1455,1456],{"class":338},"\"--fields\"",[121,1458,25],{"class":145},[121,1460,1398],{"class":482},[121,1462,396],{"class":134},[121,1464,1465],{"class":338},"\"Comma-separated keys to keep.\"",[121,1467,1406],{"class":145},[121,1469,396],{"class":134},[121,1471,1472],{"class":338}," \"\"",[121,1474,1413],{"class":145},[121,1476,1477,1480,1482,1484,1487,1489,1491,1493,1496,1498,1500,1503],{"class":123,"line":442},[121,1478,1479],{"class":145},"    skip_invalid: Annotated[",[121,1481,740],{"class":138},[121,1483,1453],{"class":145},[121,1485,1486],{"class":338},"\"--skip-invalid\"",[121,1488,25],{"class":145},[121,1490,1398],{"class":482},[121,1492,396],{"class":134},[121,1494,1495],{"class":338},"\"Skip malformed lines.\"",[121,1497,1406],{"class":145},[121,1499,396],{"class":134},[121,1501,1502],{"class":138}," False",[121,1504,1413],{"class":145},[121,1506,1507,1509,1511],{"class":123,"line":454},[121,1508,312],{"class":145},[121,1510,315],{"class":138},[121,1512,318],{"class":145},[121,1514,1515],{"class":123,"line":466},[121,1516,1517],{"class":338},"    \"\"\"Stream matching records to stdout, one JSON object per line.\"\"\"\n",[121,1519,1520,1523,1525,1528,1531,1534,1536,1538,1540,1542,1545,1547,1550,1553],{"class":123,"line":493},[121,1521,1522],{"class":145},"    conditions ",[121,1524,396],{"class":134},[121,1526,1527],{"class":138}," dict",[121,1529,1530],{"class":145},"(w.split(",[121,1532,1533],{"class":338},"\"=\"",[121,1535,25],{"class":145},[121,1537,774],{"class":138},[121,1539,649],{"class":145},[121,1541,1137],{"class":134},[121,1543,1544],{"class":145}," w ",[121,1546,760],{"class":134},[121,1548,1549],{"class":145}," (where ",[121,1551,1552],{"class":134},"or",[121,1554,1555],{"class":145}," []))\n",[121,1557,1558,1561],{"class":123,"line":498},[121,1559,1560],{"class":134},"    try",[121,1562,318],{"class":145},[121,1564,1565,1568,1570,1573,1575,1578,1581,1584,1587,1589],{"class":123,"line":503},[121,1566,1567],{"class":145},"        stats ",[121,1569,396],{"class":134},[121,1571,1572],{"class":145}," filter_records(inputs ",[121,1574,1552],{"class":134},[121,1576,1577],{"class":145}," [",[121,1579,1580],{"class":338},"\"-\"",[121,1582,1583],{"class":145},"], sys.stdout, ",[121,1585,1586],{"class":482},"where",[121,1588,396],{"class":134},[121,1590,1591],{"class":145},"conditions,\n",[121,1593,1594,1597,1599,1602,1604,1607,1609,1612,1614,1616,1619,1622,1624,1626],{"class":123,"line":509},[121,1595,1596],{"class":482},"                               fields",[121,1598,396],{"class":134},[121,1600,1601],{"class":145},"[f ",[121,1603,1137],{"class":134},[121,1605,1606],{"class":145}," f ",[121,1608,760],{"class":134},[121,1610,1611],{"class":145}," fields.split(",[121,1613,1191],{"class":338},[121,1615,649],{"class":145},[121,1617,1618],{"class":134},"if",[121,1620,1621],{"class":145}," f] ",[121,1623,1552],{"class":134},[121,1625,1045],{"class":138},[121,1627,1413],{"class":145},[121,1629,1630,1633,1635],{"class":123,"line":537},[121,1631,1632],{"class":482},"                               skip_invalid",[121,1634,396],{"class":134},[121,1636,1637],{"class":145},"skip_invalid)\n",[121,1639,1640,1643,1646,1648],{"class":123,"line":543},[121,1641,1642],{"class":134},"    except",[121,1644,1645],{"class":145}," BadRecord ",[121,1647,652],{"class":134},[121,1649,1650],{"class":145}," exc:\n",[121,1652,1653,1656,1658,1661,1663,1665,1667,1670,1672,1675,1677,1680],{"class":123,"line":560},[121,1654,1655],{"class":145},"        typer.echo(",[121,1657,335],{"class":134},[121,1659,1660],{"class":338},"\"error: ",[121,1662,342],{"class":138},[121,1664,964],{"class":145},[121,1666,348],{"class":138},[121,1668,1669],{"class":338}," (use --skip-invalid to skip bad lines)\"",[121,1671,25],{"class":145},[121,1673,1674],{"class":482},"err",[121,1676,396],{"class":134},[121,1678,1679],{"class":138},"True",[121,1681,373],{"class":145},[121,1683,1684,1687,1690,1693,1695,1697,1699],{"class":123,"line":581},[121,1685,1686],{"class":134},"        raise",[121,1688,1689],{"class":145}," typer.Exit(",[121,1691,1692],{"class":138},"65",[121,1694,649],{"class":145},[121,1696,135],{"class":134},[121,1698,1045],{"class":138},[121,1700,1701],{"class":127},"          # EX_DATAERR\n",[121,1703,1704,1706],{"class":123,"line":589},[121,1705,546],{"class":134},[121,1707,1708],{"class":145}," stats.skipped:\n",[121,1710,1711,1713,1715,1718,1720,1723,1725,1728,1730,1733,1735,1738,1740,1742,1744,1746],{"class":123,"line":598},[121,1712,1655],{"class":145},[121,1714,335],{"class":134},[121,1716,1717],{"class":338},"\"skipped ",[121,1719,342],{"class":138},[121,1721,1722],{"class":145},"stats.skipped",[121,1724,348],{"class":138},[121,1726,1727],{"class":338}," of ",[121,1729,342],{"class":138},[121,1731,1732],{"class":145},"stats.read",[121,1734,348],{"class":138},[121,1736,1737],{"class":338}," lines:\"",[121,1739,25],{"class":145},[121,1741,1674],{"class":482},[121,1743,396],{"class":134},[121,1745,1679],{"class":138},[121,1747,373],{"class":145},[121,1749,1750,1753,1756,1758],{"class":123,"line":606},[121,1751,1752],{"class":134},"        for",[121,1754,1755],{"class":145}," line ",[121,1757,760],{"class":134},[121,1759,1760],{"class":145}," stats.first_errors:\n",[121,1762,1763,1766,1768,1771,1773,1775,1777,1779,1781,1783,1785,1787],{"class":123,"line":615},[121,1764,1765],{"class":145},"            typer.echo(",[121,1767,335],{"class":134},[121,1769,1770],{"class":338},"\"  ",[121,1772,342],{"class":138},[121,1774,123],{"class":145},[121,1776,348],{"class":138},[121,1778,339],{"class":338},[121,1780,25],{"class":145},[121,1782,1674],{"class":482},[121,1784,396],{"class":134},[121,1786,1679],{"class":138},[121,1788,373],{"class":145},[10,1790,1791],{},"In use, it behaves like any other filter in a pipeline:",[112,1793,1797],{"className":1794,"code":1795,"language":1796,"meta":117,"style":117},"language-bash shiki shiki-themes github-light github-dark","mytool filter events.ndjson.gz --where level=error --fields ts,user | head -20\nkubectl logs deploy\u002Fapi | mytool filter --where level=error\nmytool filter a.ndjson b.ndjson.gz - \u003C c.ndjson > errors.ndjson\n","bash",[14,1798,1799,1831,1854],{"__ignoreMap":117},[121,1800,1801,1804,1807,1810,1813,1816,1819,1822,1825,1828],{"class":123,"line":124},[121,1802,1803],{"class":273},"mytool",[121,1805,1806],{"class":338}," filter",[121,1808,1809],{"class":338}," events.ndjson.gz",[121,1811,1812],{"class":138}," --where",[121,1814,1815],{"class":338}," level=error",[121,1817,1818],{"class":138}," --fields",[121,1820,1821],{"class":338}," ts,user",[121,1823,1824],{"class":134}," |",[121,1826,1827],{"class":273}," head",[121,1829,1830],{"class":138}," -20\n",[121,1832,1833,1836,1839,1842,1844,1847,1849,1851],{"class":123,"line":131},[121,1834,1835],{"class":273},"kubectl",[121,1837,1838],{"class":338}," logs",[121,1840,1841],{"class":338}," deploy\u002Fapi",[121,1843,1824],{"class":134},[121,1845,1846],{"class":273}," mytool",[121,1848,1806],{"class":338},[121,1850,1812],{"class":138},[121,1852,1853],{"class":338}," level=error\n",[121,1855,1856,1858,1860,1863,1866,1869,1872,1875,1878],{"class":123,"line":149},[121,1857,1803],{"class":273},[121,1859,1806],{"class":338},[121,1861,1862],{"class":338}," a.ndjson",[121,1864,1865],{"class":338}," b.ndjson.gz",[121,1867,1868],{"class":338}," -",[121,1870,1871],{"class":134}," \u003C",[121,1873,1874],{"class":338}," c.ndjson",[121,1876,1877],{"class":134}," >",[121,1879,1880],{"class":338}," errors.ndjson\n",[1882,1883,1885],"h3",{"id":1884},"why-it-is-built-this-way","Why it is built this way",[10,1887,1888,1892,1893,1895,1896,1899,1900,47],{},[1889,1890,1891],"strong",{},"Generators all the way down."," ",[14,1894,105],{}," is a generator, so parsing happens only as fast as the consumer asks for records. When ",[14,1897,1898],{},"head -20"," has what it needs and closes the pipe, the command stops reading — a 20 GB file costs the same as a 20 KB one. Handling the resulting broken pipe cleanly is covered in ",[43,1901,1903],{"href":1902},"\u002Fadvanced-input-parsing-user-experience\u002Fworking-with-stdin-stdout-and-pipes\u002Fhandling-broken-pipe-and-sigpipe\u002F","handling broken pipe and SIGPIPE",[10,1905,1906,1909,1910,1913],{},[1889,1907,1908],{},"Explicit UTF-8, and stdin in binary."," NDJSON is UTF-8 by definition, so the files are opened with ",[14,1911,1912],{},"encoding=\"utf-8\""," rather than the platform default, which differs on Windows. Stdin is wrapped from its binary buffer for the same reason, and detached afterwards so the real stdin is not closed.",[10,1915,1916,1892,1919,1922,1923,25,1926,1929,1930,47],{},[1889,1917,1918],{},"Compression is transparent.",[14,1920,1921],{},"gzip.open(..., \"rt\")"," decompresses as it reads, so archives never need to be unpacked to disk first. The same pattern extends to ",[14,1924,1925],{},".bz2",[14,1927,1928],{},".xz"," and, with a third-party package, ",[14,1931,1932],{},".zst",[10,1934,1935,1892,1938,1941],{},[1889,1936,1937],{},"Compact output.",[14,1939,1940],{},"separators=(\",\", \":\")"," writes each object without spaces, which matters when the output is gigabytes, and keeps one object per line so the output is NDJSON too — ready for the next command in the pipeline.",[10,1943,1944,1947],{},[1889,1945,1946],{},"Blank lines are ignored."," Many producers end files with an empty line or separate batches with one; treating those as errors would make the tool fussy for no benefit.",[49,1949,1951],{"id":1950},"bad-records-stop-or-skip","Bad records: stop or skip?",[72,1953],{"name":1954},"nd-errors",[10,1956,1957,1958,1961,1962,1965,1966,1969],{},"A malformed line in the middle of a large file is common: a truncated write, a log line that is not JSON, a stray array. The default here is to ",[1889,1959,1960],{},"stop"," with the file name and line number and exit 65 (",[14,1963,1964],{},"EX_DATAERR","), because silently dropping records is how data goes missing without anybody noticing. When the user knows the input is messy and wants the good records anyway, ",[14,1967,1968],{},"--skip-invalid"," skips bad lines, counts them, and reports the count and the first few locations on stderr so the output stays clean. Leniency is opt-in and visible — never the default.",[49,1971,1973],{"id":1972},"ux-considerations","UX considerations",[54,1975,1976,1990,2004,2015,2028],{},[57,1977,1978,1981,1982,1985,1986,47],{},[1889,1979,1980],{},"Default to stdin."," With no file arguments the command reads stdin, and ",[14,1983,1984],{},"-"," means stdin among other files, following the conventions in ",[43,1987,1989],{"href":1988},"\u002Fmodern-python-cli-frameworks-architecture\u002Fdesigning-cli-interfaces-and-conventions\u002Ffollowing-posix-and-gnu-argument-conventions\u002F","following POSIX and GNU argument conventions",[57,1991,1992,1995,1996,1999,2000,47],{},[1889,1993,1994],{},"Results on stdout, everything else on stderr."," The skip summary and errors go to stderr, so ",[14,1997,1998],{},"> out.ndjson"," captures only records. Output rules are covered in ",[43,2001,2003],{"href":2002},"\u002Fadvanced-input-parsing-user-experience\u002Fworking-with-stdin-stdout-and-pipes\u002Femitting-json-output-for-scripting\u002F","emitting JSON output for scripting",[57,2005,2006,2009,2010,2014],{},[1889,2007,2008],{},"Progress only on a terminal."," For long runs over files, a byte-based progress bar on stderr helps — but only when stderr is a TTY; see ",[43,2011,2013],{"href":2012},"\u002Fadvanced-input-parsing-user-experience\u002Fworking-with-stdin-stdout-and-pipes\u002Fdetecting-tty-and-adapting-output\u002F","detecting TTY and adapting output",". Stdin has no known length, so show a record count instead.",[57,2016,2017,2020,2021,2023,2024,2027],{},[1889,2018,2019],{},"Faster parsing is a drop-in."," If profiling shows ",[14,2022,84],{}," dominating, ",[14,2025,2026],{},"orjson.loads"," is several times faster and accepts the same lines; keep the streaming structure and swap only the parser.",[57,2029,2030,2033,2034,47],{},[1889,2031,2032],{},"Parallelism rarely pays."," Reading and parsing a stream is usually I\u002FO-bound or bottlenecked on one core's JSON parsing; splitting a file across processes adds complexity and reorders output. Measure first, as in ",[43,2035,2037],{"href":2036},"\u002Fcli-runtime-systems-integration\u002Fconcurrency-and-async-in-python-clis\u002Fmultiprocessing-for-cpu-bound-cli-tasks\u002F","multiprocessing for CPU-bound CLI tasks",[49,2039,2041],{"id":2040},"testing-the-behaviour","Testing the behaviour",[10,2043,2044,2045,2048],{},"The tests cover filtering and projection, stdin and gzip input together, both error modes, laziness and — the property that matters most — flat memory, measured with ",[14,2046,2047],{},"tracemalloc"," on an 18 MB file:",[112,2050,2052],{"className":114,"code":2051,"language":116,"meta":117,"style":117},"# tests\u002Ftest_stream.py\nimport gzip\nimport io\nimport json\nimport tracemalloc\n\nimport pytest\nfrom typer.testing import CliRunner\n\nfrom mytool.cli import app\nfrom mytool.stream import BadRecord, Stats, filter_records, records\n\nrunner = CliRunner()\nEVENTS = [{\"level\": \"info\", \"user\": \"ana\"}, {\"level\": \"error\", \"user\": \"bo\"}, {\"level\": \"error\", \"user\": \"cy\"}]\nNDJSON = \"\".join(json.dumps(e) + \"\\n\" for e in EVENTS)\n\n\ndef test_filters_and_projects(tmp_path):\n    path = tmp_path \u002F \"events.ndjson\"\n    path.write_text(NDJSON)\n    result = runner.invoke(app, [\"filter\", str(path), \"--where\", \"level=error\", \"--fields\", \"user\"])\n    assert result.exit_code == 0\n    assert result.stdout.splitlines() == ['{\"user\":\"bo\"}', '{\"user\":\"cy\"}']\n\n\ndef test_reads_stdin_and_gzip(tmp_path):\n    gz = tmp_path \u002F \"events.ndjson.gz\"\n    with gzip.open(gz, \"wt\", encoding=\"utf-8\") as fh:\n        fh.write(NDJSON)\n    result = runner.invoke(app, [\"filter\", \"-\", str(gz), \"--where\", \"user=ana\"], input=NDJSON)\n    assert result.stdout.count('\"ana\"') == 2\n\n\ndef test_bad_line_stops_with_its_line_number(tmp_path):\n    path = tmp_path \u002F \"bad.ndjson\"\n    path.write_text(NDJSON + \"{not json\\n\")\n    result = runner.invoke(app, [\"filter\", str(path)])\n    assert result.exit_code == 65\n    assert f\"{path}:4:\" in result.stderr\n\n\ndef test_skip_invalid_counts_and_reports(tmp_path):\n    path = tmp_path \u002F \"bad.ndjson\"\n    path.write_text(\"[1, 2]\\n\" + NDJSON + \"{oops\\n\")\n    result = runner.invoke(app, [\"filter\", str(path), \"--skip-invalid\"])\n    assert result.exit_code == 0 and len(result.stdout.splitlines()) == 3\n    assert \"skipped 2 of 5 lines\" in result.stderr\n\n\ndef test_records_is_lazy():\n    stats = Stats()\n    gen = records(io.StringIO('{\"a\": 1}\\n{broken\\n'), \"x\", skip_invalid=False, stats=stats)\n    assert next(gen) == {\"a\": 1}            # the bad second line has not been read yet\n    with pytest.raises(BadRecord):\n        next(gen)\n\n\ndef test_memory_stays_flat(tmp_path):\n    path = tmp_path \u002F \"big.ndjson\"\n    with path.open(\"w\") as fh:\n        for i in range(200_000):\n            fh.write(json.dumps({\"i\": i, \"level\": \"info\", \"msg\": \"x\" * 50}) + \"\\n\")\n    tracemalloc.start()\n    filter_records([str(path)], io.StringIO(), where={\"level\": \"none\"}, fields=None, skip_invalid=False)\n    _, peak = tracemalloc.get_traced_memory()\n    tracemalloc.stop()\n    assert path.stat().st_size > 15_000_000\n    assert peak \u003C 1_000_000                 # measured: about 25 KB for an 18 MB file\n",[14,2053,2054,2059,2065,2071,2077,2084,2088,2095,2107,2111,2123,2134,2138,2148,2215,2248,2252,2256,2266,2282,2291,2328,2340,2362,2366,2370,2379,2393,2418,2427,2466,2483,2487,2491,2500,2513,2531,2548,2559,2584,2588,2592,2601,2613,2640,2660,2684,2695,2699,2703,2713,2721,2768,2796,2803,2811,2815,2819,2828,2841,2857,2876,2921,2926,2969,2979,2984,2997],{"__ignoreMap":117},[121,2055,2056],{"class":123,"line":124},[121,2057,2058],{"class":127},"# tests\u002Ftest_stream.py\n",[121,2060,2061,2063],{"class":123,"line":131},[121,2062,159],{"class":134},[121,2064,162],{"class":145},[121,2066,2067,2069],{"class":123,"line":149},[121,2068,159],{"class":134},[121,2070,170],{"class":145},[121,2072,2073,2075],{"class":123,"line":156},[121,2074,159],{"class":134},[121,2076,178],{"class":145},[121,2078,2079,2081],{"class":123,"line":165},[121,2080,159],{"class":134},[121,2082,2083],{"class":145}," tracemalloc\n",[121,2085,2086],{"class":123,"line":173},[121,2087,153],{"emptyLinePlaceholder":152},[121,2089,2090,2092],{"class":123,"line":181},[121,2091,159],{"class":134},[121,2093,2094],{"class":145}," pytest\n",[121,2096,2097,2099,2102,2104],{"class":123,"line":189},[121,2098,135],{"class":134},[121,2100,2101],{"class":145}," typer.testing ",[121,2103,159],{"class":134},[121,2105,2106],{"class":145}," CliRunner\n",[121,2108,2109],{"class":123,"line":202},[121,2110,153],{"emptyLinePlaceholder":152},[121,2112,2113,2115,2118,2120],{"class":123,"line":215},[121,2114,135],{"class":134},[121,2116,2117],{"class":145}," mytool.cli ",[121,2119,159],{"class":134},[121,2121,2122],{"class":145}," app\n",[121,2124,2125,2127,2129,2131],{"class":123,"line":228},[121,2126,135],{"class":134},[121,2128,1301],{"class":145},[121,2130,159],{"class":134},[121,2132,2133],{"class":145}," BadRecord, Stats, filter_records, records\n",[121,2135,2136],{"class":123,"line":241},[121,2137,153],{"emptyLinePlaceholder":152},[121,2139,2140,2143,2145],{"class":123,"line":257},[121,2141,2142],{"class":145},"runner ",[121,2144,396],{"class":134},[121,2146,2147],{"class":145}," CliRunner()\n",[121,2149,2150,2153,2155,2158,2161,2163,2166,2168,2171,2173,2176,2179,2181,2183,2186,2188,2190,2192,2195,2197,2199,2201,2203,2205,2207,2209,2212],{"class":123,"line":262},[121,2151,2152],{"class":138},"EVENTS",[121,2154,436],{"class":134},[121,2156,2157],{"class":145}," [{",[121,2159,2160],{"class":338},"\"level\"",[121,2162,361],{"class":145},[121,2164,2165],{"class":338},"\"info\"",[121,2167,25],{"class":145},[121,2169,2170],{"class":338},"\"user\"",[121,2172,361],{"class":145},[121,2174,2175],{"class":338},"\"ana\"",[121,2177,2178],{"class":145},"}, {",[121,2180,2160],{"class":338},[121,2182,361],{"class":145},[121,2184,2185],{"class":338},"\"error\"",[121,2187,25],{"class":145},[121,2189,2170],{"class":338},[121,2191,361],{"class":145},[121,2193,2194],{"class":338},"\"bo\"",[121,2196,2178],{"class":145},[121,2198,2160],{"class":338},[121,2200,361],{"class":145},[121,2202,2185],{"class":338},[121,2204,25],{"class":145},[121,2206,2170],{"class":338},[121,2208,361],{"class":145},[121,2210,2211],{"class":338},"\"cy\"",[121,2213,2214],{"class":145},"}]\n",[121,2216,2217,2220,2222,2224,2227,2229,2231,2233,2235,2238,2241,2243,2246],{"class":123,"line":267},[121,2218,2219],{"class":138},"NDJSON",[121,2221,436],{"class":134},[121,2223,1472],{"class":338},[121,2225,2226],{"class":145},".join(json.dumps(e) ",[121,2228,1202],{"class":134},[121,2230,1205],{"class":338},[121,2232,1208],{"class":138},[121,2234,339],{"class":338},[121,2236,2237],{"class":134}," for",[121,2239,2240],{"class":145}," e ",[121,2242,760],{"class":134},[121,2244,2245],{"class":138}," EVENTS",[121,2247,373],{"class":145},[121,2249,2250],{"class":123,"line":286},[121,2251,153],{"emptyLinePlaceholder":152},[121,2253,2254],{"class":123,"line":321},[121,2255,153],{"emptyLinePlaceholder":152},[121,2257,2258,2260,2263],{"class":123,"line":376},[121,2259,512],{"class":134},[121,2261,2262],{"class":273}," test_filters_and_projects",[121,2264,2265],{"class":145},"(tmp_path):\n",[121,2267,2268,2271,2273,2276,2279],{"class":123,"line":402},[121,2269,2270],{"class":145},"    path ",[121,2272,396],{"class":134},[121,2274,2275],{"class":145}," tmp_path ",[121,2277,2278],{"class":134},"\u002F",[121,2280,2281],{"class":338}," \"events.ndjson\"\n",[121,2283,2284,2287,2289],{"class":123,"line":407},[121,2285,2286],{"class":145},"    path.write_text(",[121,2288,2219],{"class":138},[121,2290,373],{"class":145},[121,2292,2293,2296,2298,2301,2303,2305,2307,2310,2312,2314,2317,2319,2321,2323,2325],{"class":123,"line":412},[121,2294,2295],{"class":145},"    result ",[121,2297,396],{"class":134},[121,2299,2300],{"class":145}," runner.invoke(app, [",[121,2302,1373],{"class":338},[121,2304,25],{"class":145},[121,2306,298],{"class":138},[121,2308,2309],{"class":145},"(path), ",[121,2311,1426],{"class":338},[121,2313,25],{"class":145},[121,2315,2316],{"class":338},"\"level=error\"",[121,2318,25],{"class":145},[121,2320,1456],{"class":338},[121,2322,25],{"class":145},[121,2324,2170],{"class":338},[121,2326,2327],{"class":145},"])\n",[121,2329,2330,2333,2336,2338],{"class":123,"line":418},[121,2331,2332],{"class":134},"    assert",[121,2334,2335],{"class":145}," result.exit_code ",[121,2337,552],{"class":134},[121,2339,439],{"class":138},[121,2341,2342,2344,2347,2349,2351,2354,2356,2359],{"class":123,"line":428},[121,2343,2332],{"class":134},[121,2345,2346],{"class":145}," result.stdout.splitlines() ",[121,2348,552],{"class":134},[121,2350,1577],{"class":145},[121,2352,2353],{"class":338},"'{\"user\":\"bo\"}'",[121,2355,25],{"class":145},[121,2357,2358],{"class":338},"'{\"user\":\"cy\"}'",[121,2360,2361],{"class":145},"]\n",[121,2363,2364],{"class":123,"line":442},[121,2365,153],{"emptyLinePlaceholder":152},[121,2367,2368],{"class":123,"line":454},[121,2369,153],{"emptyLinePlaceholder":152},[121,2371,2372,2374,2377],{"class":123,"line":466},[121,2373,512],{"class":134},[121,2375,2376],{"class":273}," test_reads_stdin_and_gzip",[121,2378,2265],{"class":145},[121,2380,2381,2384,2386,2388,2390],{"class":123,"line":493},[121,2382,2383],{"class":145},"    gz ",[121,2385,396],{"class":134},[121,2387,2275],{"class":145},[121,2389,2278],{"class":134},[121,2391,2392],{"class":338}," \"events.ndjson.gz\"\n",[121,2394,2395,2398,2401,2404,2406,2408,2410,2412,2414,2416],{"class":123,"line":498},[121,2396,2397],{"class":134},"    with",[121,2399,2400],{"class":145}," gzip.open(gz, ",[121,2402,2403],{"class":338},"\"wt\"",[121,2405,25],{"class":145},[121,2407,571],{"class":482},[121,2409,396],{"class":134},[121,2411,576],{"class":338},[121,2413,649],{"class":145},[121,2415,652],{"class":134},[121,2417,655],{"class":145},[121,2419,2420,2423,2425],{"class":123,"line":503},[121,2421,2422],{"class":145},"        fh.write(",[121,2424,2219],{"class":138},[121,2426,373],{"class":145},[121,2428,2429,2431,2433,2435,2437,2439,2441,2443,2445,2448,2450,2452,2455,2457,2460,2462,2464],{"class":123,"line":509},[121,2430,2295],{"class":145},[121,2432,396],{"class":134},[121,2434,2300],{"class":145},[121,2436,1373],{"class":338},[121,2438,25],{"class":145},[121,2440,1580],{"class":338},[121,2442,25],{"class":145},[121,2444,298],{"class":138},[121,2446,2447],{"class":145},"(gz), ",[121,2449,1426],{"class":338},[121,2451,25],{"class":145},[121,2453,2454],{"class":338},"\"user=ana\"",[121,2456,1019],{"class":145},[121,2458,2459],{"class":482},"input",[121,2461,396],{"class":134},[121,2463,2219],{"class":138},[121,2465,373],{"class":145},[121,2467,2468,2470,2473,2476,2478,2480],{"class":123,"line":537},[121,2469,2332],{"class":134},[121,2471,2472],{"class":145}," result.stdout.count(",[121,2474,2475],{"class":338},"'\"ana\"'",[121,2477,649],{"class":145},[121,2479,552],{"class":134},[121,2481,2482],{"class":138}," 2\n",[121,2484,2485],{"class":123,"line":543},[121,2486,153],{"emptyLinePlaceholder":152},[121,2488,2489],{"class":123,"line":560},[121,2490,153],{"emptyLinePlaceholder":152},[121,2492,2493,2495,2498],{"class":123,"line":581},[121,2494,512],{"class":134},[121,2496,2497],{"class":273}," test_bad_line_stops_with_its_line_number",[121,2499,2265],{"class":145},[121,2501,2502,2504,2506,2508,2510],{"class":123,"line":589},[121,2503,2270],{"class":145},[121,2505,396],{"class":134},[121,2507,2275],{"class":145},[121,2509,2278],{"class":134},[121,2511,2512],{"class":338}," \"bad.ndjson\"\n",[121,2514,2515,2517,2519,2522,2525,2527,2529],{"class":123,"line":598},[121,2516,2286],{"class":145},[121,2518,2219],{"class":138},[121,2520,2521],{"class":134}," +",[121,2523,2524],{"class":338}," \"{not json",[121,2526,1208],{"class":138},[121,2528,339],{"class":338},[121,2530,373],{"class":145},[121,2532,2533,2535,2537,2539,2541,2543,2545],{"class":123,"line":606},[121,2534,2295],{"class":145},[121,2536,396],{"class":134},[121,2538,2300],{"class":145},[121,2540,1373],{"class":338},[121,2542,25],{"class":145},[121,2544,298],{"class":138},[121,2546,2547],{"class":145},"(path)])\n",[121,2549,2550,2552,2554,2556],{"class":123,"line":615},[121,2551,2332],{"class":134},[121,2553,2335],{"class":145},[121,2555,552],{"class":134},[121,2557,2558],{"class":138}," 65\n",[121,2560,2561,2563,2566,2568,2570,2573,2575,2578,2581],{"class":123,"line":629},[121,2562,2332],{"class":134},[121,2564,2565],{"class":134}," f",[121,2567,339],{"class":338},[121,2569,342],{"class":138},[121,2571,2572],{"class":145},"path",[121,2574,348],{"class":138},[121,2576,2577],{"class":338},":4:\"",[121,2579,2580],{"class":134}," in",[121,2582,2583],{"class":145}," result.stderr\n",[121,2585,2586],{"class":123,"line":658},[121,2587,153],{"emptyLinePlaceholder":152},[121,2589,2590],{"class":123,"line":665},[121,2591,153],{"emptyLinePlaceholder":152},[121,2593,2594,2596,2599],{"class":123,"line":673},[121,2595,512],{"class":134},[121,2597,2598],{"class":273}," test_skip_invalid_counts_and_reports",[121,2600,2265],{"class":145},[121,2602,2603,2605,2607,2609,2611],{"class":123,"line":693},[121,2604,2270],{"class":145},[121,2606,396],{"class":134},[121,2608,2275],{"class":145},[121,2610,2278],{"class":134},[121,2612,2512],{"class":338},[121,2614,2615,2617,2620,2622,2624,2626,2629,2631,2634,2636,2638],{"class":123,"line":700},[121,2616,2286],{"class":145},[121,2618,2619],{"class":338},"\"[1, 2]",[121,2621,1208],{"class":138},[121,2623,339],{"class":338},[121,2625,2521],{"class":134},[121,2627,2628],{"class":138}," NDJSON",[121,2630,2521],{"class":134},[121,2632,2633],{"class":338}," \"{oops",[121,2635,1208],{"class":138},[121,2637,339],{"class":338},[121,2639,373],{"class":145},[121,2641,2642,2644,2646,2648,2650,2652,2654,2656,2658],{"class":123,"line":705},[121,2643,2295],{"class":145},[121,2645,396],{"class":134},[121,2647,2300],{"class":145},[121,2649,1373],{"class":338},[121,2651,25],{"class":145},[121,2653,298],{"class":138},[121,2655,2309],{"class":145},[121,2657,1486],{"class":338},[121,2659,2327],{"class":145},[121,2661,2662,2664,2666,2668,2671,2674,2676,2679,2681],{"class":123,"line":710},[121,2663,2332],{"class":134},[121,2665,2335],{"class":145},[121,2667,552],{"class":134},[121,2669,2670],{"class":138}," 0",[121,2672,2673],{"class":134}," and",[121,2675,922],{"class":138},[121,2677,2678],{"class":145},"(result.stdout.splitlines()) ",[121,2680,552],{"class":134},[121,2682,2683],{"class":138}," 3\n",[121,2685,2686,2688,2691,2693],{"class":123,"line":751},[121,2687,2332],{"class":134},[121,2689,2690],{"class":338}," \"skipped 2 of 5 lines\"",[121,2692,2580],{"class":134},[121,2694,2583],{"class":145},[121,2696,2697],{"class":123,"line":779},[121,2698,153],{"emptyLinePlaceholder":152},[121,2700,2701],{"class":123,"line":791},[121,2702,153],{"emptyLinePlaceholder":152},[121,2704,2705,2707,2710],{"class":123,"line":797},[121,2706,512],{"class":134},[121,2708,2709],{"class":273}," test_records_is_lazy",[121,2711,2712],{"class":145},"():\n",[121,2714,2715,2717,2719],{"class":123,"line":809},[121,2716,1058],{"class":145},[121,2718,396],{"class":134},[121,2720,1063],{"class":145},[121,2722,2723,2726,2728,2731,2734,2736,2739,2741,2744,2747,2750,2752,2754,2756,2759,2761,2763,2765],{"class":123,"line":816},[121,2724,2725],{"class":145},"    gen ",[121,2727,396],{"class":134},[121,2729,2730],{"class":145}," records(io.StringIO(",[121,2732,2733],{"class":338},"'{\"a\": 1}",[121,2735,1208],{"class":138},[121,2737,2738],{"class":338},"{broken",[121,2740,1208],{"class":138},[121,2742,2743],{"class":338},"'",[121,2745,2746],{"class":145},"), ",[121,2748,2749],{"class":338},"\"x\"",[121,2751,25],{"class":145},[121,2753,1104],{"class":482},[121,2755,396],{"class":134},[121,2757,2758],{"class":138},"False",[121,2760,25],{"class":145},[121,2762,1112],{"class":482},[121,2764,396],{"class":134},[121,2766,2767],{"class":145},"stats)\n",[121,2769,2770,2772,2775,2778,2780,2783,2786,2788,2790,2793],{"class":123,"line":827},[121,2771,2332],{"class":134},[121,2773,2774],{"class":138}," next",[121,2776,2777],{"class":145},"(gen) ",[121,2779,552],{"class":134},[121,2781,2782],{"class":145}," {",[121,2784,2785],{"class":338},"\"a\"",[121,2787,361],{"class":145},[121,2789,774],{"class":138},[121,2791,2792],{"class":145},"}            ",[121,2794,2795],{"class":127},"# the bad second line has not been read yet\n",[121,2797,2798,2800],{"class":123,"line":846},[121,2799,2397],{"class":134},[121,2801,2802],{"class":145}," pytest.raises(BadRecord):\n",[121,2804,2805,2808],{"class":123,"line":862},[121,2806,2807],{"class":138},"        next",[121,2809,2810],{"class":145},"(gen)\n",[121,2812,2813],{"class":123,"line":879},[121,2814,153],{"emptyLinePlaceholder":152},[121,2816,2817],{"class":123,"line":889},[121,2818,153],{"emptyLinePlaceholder":152},[121,2820,2821,2823,2826],{"class":123,"line":907},[121,2822,512],{"class":134},[121,2824,2825],{"class":273}," test_memory_stays_flat",[121,2827,2265],{"class":145},[121,2829,2830,2832,2834,2836,2838],{"class":123,"line":917},[121,2831,2270],{"class":145},[121,2833,396],{"class":134},[121,2835,2275],{"class":145},[121,2837,2278],{"class":134},[121,2839,2840],{"class":338}," \"big.ndjson\"\n",[121,2842,2843,2845,2848,2851,2853,2855],{"class":123,"line":936},[121,2844,2397],{"class":134},[121,2846,2847],{"class":145}," path.open(",[121,2849,2850],{"class":338},"\"w\"",[121,2852,649],{"class":145},[121,2854,652],{"class":134},[121,2856,655],{"class":145},[121,2858,2859,2861,2864,2866,2869,2871,2874],{"class":123,"line":973},[121,2860,1752],{"class":134},[121,2862,2863],{"class":145}," i ",[121,2865,760],{"class":134},[121,2867,2868],{"class":138}," range",[121,2870,277],{"class":145},[121,2872,2873],{"class":138},"200_000",[121,2875,283],{"class":145},[121,2877,2878,2881,2884,2887,2889,2891,2893,2895,2898,2900,2902,2905,2908,2911,2913,2915,2917,2919],{"class":123,"line":978},[121,2879,2880],{"class":145},"            fh.write(json.dumps({",[121,2882,2883],{"class":338},"\"i\"",[121,2885,2886],{"class":145},": i, ",[121,2888,2160],{"class":338},[121,2890,361],{"class":145},[121,2892,2165],{"class":338},[121,2894,25],{"class":145},[121,2896,2897],{"class":338},"\"msg\"",[121,2899,361],{"class":145},[121,2901,2749],{"class":338},[121,2903,2904],{"class":134}," *",[121,2906,2907],{"class":138}," 50",[121,2909,2910],{"class":145},"}) ",[121,2912,1202],{"class":134},[121,2914,1205],{"class":338},[121,2916,1208],{"class":138},[121,2918,339],{"class":338},[121,2920,373],{"class":145},[121,2922,2923],{"class":123,"line":987},[121,2924,2925],{"class":145},"    tracemalloc.start()\n",[121,2927,2928,2931,2933,2936,2938,2940,2942,2944,2946,2949,2952,2955,2957,2959,2961,2963,2965,2967],{"class":123,"line":992},[121,2929,2930],{"class":145},"    filter_records([",[121,2932,298],{"class":138},[121,2934,2935],{"class":145},"(path)], io.StringIO(), ",[121,2937,1586],{"class":482},[121,2939,396],{"class":134},[121,2941,342],{"class":145},[121,2943,2160],{"class":338},[121,2945,361],{"class":145},[121,2947,2948],{"class":338},"\"none\"",[121,2950,2951],{"class":145},"}, ",[121,2953,2954],{"class":482},"fields",[121,2956,396],{"class":134},[121,2958,315],{"class":138},[121,2960,25],{"class":145},[121,2962,1104],{"class":482},[121,2964,396],{"class":134},[121,2966,2758],{"class":138},[121,2968,373],{"class":145},[121,2970,2971,2974,2976],{"class":123,"line":997},[121,2972,2973],{"class":145},"    _, peak ",[121,2975,396],{"class":134},[121,2977,2978],{"class":145}," tracemalloc.get_traced_memory()\n",[121,2980,2981],{"class":123,"line":1032},[121,2982,2983],{"class":145},"    tracemalloc.stop()\n",[121,2985,2986,2988,2991,2994],{"class":123,"line":1055},[121,2987,2332],{"class":134},[121,2989,2990],{"class":145}," path.stat().st_size ",[121,2992,2993],{"class":134},">",[121,2995,2996],{"class":138}," 15_000_000\n",[121,2998,2999,3001,3004,3006,3009],{"class":123,"line":1066},[121,3000,2332],{"class":134},[121,3002,3003],{"class":145}," peak ",[121,3005,928],{"class":134},[121,3007,3008],{"class":138}," 1_000_000",[121,3010,3011],{"class":127},"                 # measured: about 25 KB for an 18 MB file\n",[10,3013,3014,3017,3018,3021,3022,3025],{},[14,3015,3016],{},"test_records_is_lazy"," proves the generator yields the first record before it has read the broken second line; a version that parsed everything up front would raise immediately. ",[14,3019,3020],{},"test_memory_stays_flat"," is the regression test for \"someone added ",[14,3023,3024],{},"list(...)"," for convenience\": a change that loads the file would push the peak into tens of megabytes and fail. It writes 200,000 records, so it takes about a second — mark it as slow if your suite is strict about speed.",[49,3027,3029],{"id":3028},"conclusion","Conclusion",[10,3031,3032,3033,3035,3036,3038,3039,3041,3042,3044],{},"Handling large inputs in a CLI is a matter of never holding more than one record: iterate over lines, parse each with ",[14,3034,84],{},", write results immediately, and let generators connect the stages so the command stops as soon as its consumer does. Open files with explicit UTF-8, decompress ",[14,3037,40],{}," on the fly, treat ",[14,3040,1984],{}," and no arguments as stdin, stop on malformed lines with a file and line number by default, and make skipping explicit with a counted summary. A ",[14,3043,2047],{}," test keeps memory flat through future changes, and the command stays usable on files far larger than the machine's memory.",[49,3046,3048],{"id":3047},"frequently-asked-questions","Frequently asked questions",[1882,3050,3052],{"id":3051},"what-if-the-input-is-one-big-json-array-rather-than-ndjson","What if the input is one big JSON array rather than NDJSON?",[10,3054,3055,3058,3059,3062,3063,3066],{},[14,3056,3057],{},"json.load"," must then read the whole document. For arrays too large for memory, use an incremental parser such as ",[14,3060,3061],{},"ijson",", which yields array items one at a time; or convert once with ",[14,3064,3065],{},"jq -c '.[]' big.json > big.ndjson"," and stream from then on.",[1882,3068,3070],{"id":3069},"should-the-output-preserve-key-order-and-formatting","Should the output preserve key order and formatting?",[10,3072,3073,3074,3076,3077,3080,3081,3083],{},"Key order is preserved (",[14,3075,841],{}," keeps insertion order and ",[14,3078,3079],{},"json.dumps"," writes it). Formatting is normalised to compact JSON; if byte-for-byte passthrough of matching lines matters, write the original ",[14,3082,123],{}," instead of re-serialising the parsed object.",[1882,3085,3087],{"id":3086},"how-do-i-handle-lines-with-invalid-utf-8","How do I handle lines with invalid UTF-8?",[10,3089,3090,3091,3094,3095,3098],{},"By default decoding raises ",[14,3092,3093],{},"UnicodeDecodeError",", which stops the command. For dirty sources, open with ",[14,3096,3097],{},"errors=\"replace\""," and let the JSON parser decide whether the result is still valid; report the replacement in the skip summary.",[1882,3100,3102],{"id":3101},"can-the-command-read-from-s3-or-http-directly","Can the command read from S3 or HTTP directly?",[10,3104,3105,3106,3109,3110,3113,3114,3117,3118,3120],{},"Keep the command reading streams and let the transport produce one: ",[14,3107,3108],{},"aws s3 cp s3:\u002F\u002Fbucket\u002Fkey - | mytool filter",", or an ",[14,3111,3112],{},"httpx"," streaming response whose ",[14,3115,3116],{},"iter_lines()"," feeds ",[14,3119,105],{},". The streaming core stays the same.",[1882,3122,3124],{"id":3123},"how-do-i-show-which-file-a-bad-line-came-from-when-reading-several-files","How do I show which file a bad line came from when reading several files?",[10,3126,3127,3128,3130,3131,3133,3134,3136],{},"Pass the file name into ",[14,3129,105],{}," as ",[14,3132,345],{},", as the recipe does, and include it in every error message. With stdin, ",[14,3135,1984],{}," is the name users recognise.",[49,3138,3140],{"id":3139},"related","Related",[54,3142,3143,3149,3154,3159,3164],{},[57,3144,3145,3146],{},"Up: ",[43,3147,3148],{"href":45},"Working with stdin, stdout and pipes",[57,3150,3151],{},[43,3152,3153],{"href":65},"Reading piped input in Python CLIs",[57,3155,3156],{},[43,3157,3158],{"href":1902},"Handling broken pipe and SIGPIPE",[57,3160,3161],{},[43,3162,3163],{"href":2002},"Emitting JSON output for scripting",[57,3165,3166],{},[43,3167,3169],{"href":3168},"\u002Fcli-runtime-systems-integration\u002Fcalling-http-apis-from-python-clis\u002Fdownloading-files-with-progress-in-python\u002F","Downloading files with progress in Python",[3171,3172,3173],"style",{},"html pre.shiki code .sJ8bj, html code.shiki .sJ8bj{--shiki-default:#6A737D;--shiki-dark:#6A737D}html pre.shiki code .szBVR, html code.shiki .szBVR{--shiki-default:#D73A49;--shiki-dark:#F97583}html pre.shiki code .sj4cs, html code.shiki .sj4cs{--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .sVt8B, html code.shiki .sVt8B{--shiki-default:#24292E;--shiki-dark:#E1E4E8}html pre.shiki code .sScJk, html code.shiki .sScJk{--shiki-default:#6F42C1;--shiki-dark:#B392F0}html pre.shiki code .sZZnC, html code.shiki .sZZnC{--shiki-default:#032F62;--shiki-dark:#9ECBFF}html pre.shiki code .s4XuR, html code.shiki .s4XuR{--shiki-default:#E36209;--shiki-dark:#FFAB70}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":117,"searchDepth":131,"depth":131,"links":3175},[3176,3177,3178,3181,3182,3183,3184,3185,3192],{"id":51,"depth":131,"text":52},{"id":69,"depth":131,"text":70},{"id":94,"depth":131,"text":95,"children":3179},[3180],{"id":1884,"depth":149,"text":1885},{"id":1950,"depth":131,"text":1951},{"id":1972,"depth":131,"text":1973},{"id":2040,"depth":131,"text":2041},{"id":3028,"depth":131,"text":3029},{"id":3047,"depth":131,"text":3048,"children":3186},[3187,3188,3189,3190,3191],{"id":3051,"depth":149,"text":3052},{"id":3069,"depth":149,"text":3070},{"id":3086,"depth":149,"text":3087},{"id":3101,"depth":149,"text":3102},{"id":3123,"depth":149,"text":3124},{"id":3139,"depth":131,"text":3140},"2026-09-18","Filter gigabytes of newline-delimited JSON from files, gzip archives or stdin with flat memory, clear errors on bad lines and an explicit --skip-invalid mode.","intermediate",false,"md",{},"\u002Fadvanced-input-parsing-user-experience\u002Fworking-with-stdin-stdout-and-pipes\u002Fprocessing-large-files-and-ndjson-streams",{"title":5,"description":3194},"advanced-input-parsing-user-experience\u002Fworking-with-stdin-stdout-and-pipes\u002Fprocessing-large-files-and-ndjson-streams\u002Findex",[3203,3204,3205,3206,3207],"ndjson","streaming","stdin","performance","typer","wssL5i1dDv1Yehg6HbqgrmThpGR0W6nCQuW91VlV3nc",[3210,3213,3216,3219,3222,3225,3228,3231,3234,3237,3240,3243,3246,3249,3252,3255,3258,3261,3264,3267,3270,3273,3276,3279,3282,3285,3288,3291,3294,3297,3300,3303,3306,3309,3312,3315,3318,3321,3324,3327,3330,3333,3336,3339,3342,3345,3348,3351,3354,3357,3360,3363,3366,3369,3370,3373,3376,3379,3382,3385,3388,3391,3394,3397,3400,3403,3406,3409,3412,3415,3418,3421,3424,3427,3430,3433,3436,3439,3442,3445,3448,3451,3454,3457,3460,3463,3466,3469,3472,3475,3478,3481,3483,3486,3489,3492,3495,3498,3501,3504,3507,3510,3513,3516,3519,3522,3525,3528,3531,3534,3537,3540,3543,3546,3549,3552,3555,3558,3561,3564,3567,3570,3573,3576,3579,3582,3585,3588,3591,3594,3597,3600,3603,3606,3609,3612,3615,3618,3621,3624,3627,3630,3633,3636,3639,3642,3645,3648,3651,3654,3657,3660,3663,3666,3669,3672,3675,3678,3681,3684,3687,3690,3693,3696,3699,3702,3705,3708,3711,3714,3717,3720,3723,3726,3729,3732,3735,3738,3741,3744,3747,3750,3753],{"path":3211,"title":3212},"\u002Fabout","About Python CLI Toolcraft",{"path":3214,"title":3215},"\u002Fadvanced-input-parsing-user-experience\u002Fadvanced-argument-validation-strategies","Advanced Argument Validation Strategies",{"path":3217,"title":3218},"\u002Fadvanced-input-parsing-user-experience\u002Fadvanced-argument-validation-strategies\u002Fparsing-nested-json-arguments-in-python-clis","Parsing Nested JSON Args in Python CLIs",{"path":3220,"title":3221},"\u002Fadvanced-input-parsing-user-experience\u002Fadvanced-argument-validation-strategies\u002Fvalidating-dependent-and-conflicting-options","Validating Dependent and Conflicting CLI Options",{"path":3223,"title":3224},"\u002Fadvanced-input-parsing-user-experience\u002Fadvanced-argument-validation-strategies\u002Fvalidating-file-and-directory-paths-in-clis","Validating File and Directory Paths in CLIs",{"path":3226,"title":3227},"\u002Fadvanced-input-parsing-user-experience\u002Fadvanced-argument-validation-strategies\u002Fwriting-custom-click-parameter-types","Writing Custom Click Parameter Types",{"path":3229,"title":3230},"\u002Fadvanced-input-parsing-user-experience\u002Fbuilding-terminal-uis-with-textual\u002Fbuilding-your-first-textual-app","Building Your First Textual App for a Python CLI",{"path":3232,"title":3233},"\u002Fadvanced-input-parsing-user-experience\u002Fbuilding-terminal-uis-with-textual\u002Fchoosing-between-a-cli-a-prompt-flow-and-a-tui","Choosing Between a CLI, a Prompt Flow and a TUI",{"path":3235,"title":3236},"\u002Fadvanced-input-parsing-user-experience\u002Fbuilding-terminal-uis-with-textual","Building Terminal UIs with Textual for Python CLIs",{"path":3238,"title":3239},"\u002Fadvanced-input-parsing-user-experience\u002Fbuilding-terminal-uis-with-textual\u002Ftesting-textual-apps-with-pilot","Testing Textual Apps with Pilot",{"path":3241,"title":3242},"\u002Fadvanced-input-parsing-user-experience\u002Fcli-help-output-and-documentation\u002Fadding-examples-and-epilogs-to-help-output","Adding Examples and Epilogs to Help Output",{"path":3244,"title":3245},"\u002Fadvanced-input-parsing-user-experience\u002Fcli-help-output-and-documentation\u002Fgenerating-man-pages-and-docs-from-a-cli","Generating Man Pages and Docs from a CLI",{"path":3247,"title":3248},"\u002Fadvanced-input-parsing-user-experience\u002Fcli-help-output-and-documentation","CLI Help Output and Documentation",{"path":3250,"title":3251},"\u002Fadvanced-input-parsing-user-experience\u002Fcli-help-output-and-documentation\u002Fversioning-and-deprecating-cli-flags","Versioning and Deprecating CLI Flags",{"path":3253,"title":3254},"\u002Fadvanced-input-parsing-user-experience\u002Fcli-help-output-and-documentation\u002Fwriting-help-text-users-actually-read","Writing Help Text Users Actually Read",{"path":3256,"title":3257},"\u002Fadvanced-input-parsing-user-experience\u002Fcross-platform-terminal-compatibility\u002Fadapting-output-to-terminal-width","Adapting Python CLI Output to Terminal Width",{"path":3259,"title":3260},"\u002Fadvanced-input-parsing-user-experience\u002Fcross-platform-terminal-compatibility\u002Fdetecting-ci-environments-and-non-interactive-shells","Detecting CI Environments and Non-Interactive Shells",{"path":3262,"title":3263},"\u002Fadvanced-input-parsing-user-experience\u002Fcross-platform-terminal-compatibility\u002Ffixing-unicode-and-encoding-errors-on-windows","Fixing Unicode and Encoding Errors on Windows in Python CLIs",{"path":3265,"title":3266},"\u002Fadvanced-input-parsing-user-experience\u002Fcross-platform-terminal-compatibility","Cross-Platform Terminal Compatibility for Python CLIs",{"path":3268,"title":3269},"\u002Fadvanced-input-parsing-user-experience\u002Fcross-platform-terminal-compatibility\u002Frespecting-no-color-and-force-color","Respecting NO_COLOR and FORCE_COLOR in Python CLIs",{"path":3271,"title":3272},"\u002Fadvanced-input-parsing-user-experience\u002Ferror-handling-and-exit-codes\u002Fchoosing-exit-codes-for-cli-tools","Choosing Exit Codes for CLI Tools",{"path":3274,"title":3275},"\u002Fadvanced-input-parsing-user-experience\u002Ferror-handling-and-exit-codes\u002Fdesigning-an-exception-hierarchy-for-a-cli","Designing an Exception Hierarchy for a Python CLI",{"path":3277,"title":3278},"\u002Fadvanced-input-parsing-user-experience\u002Ferror-handling-and-exit-codes\u002Ffriendly-error-messages-and-tracebacks","Friendly Error Messages and Tracebacks",{"path":3280,"title":3281},"\u002Fadvanced-input-parsing-user-experience\u002Ferror-handling-and-exit-codes\u002Fhandling-keyboard-interrupt-cleanly","Handling Keyboard Interrupt Cleanly",{"path":3283,"title":3284},"\u002Fadvanced-input-parsing-user-experience\u002Ferror-handling-and-exit-codes","Error Handling and Exit Codes for CLIs",{"path":3286,"title":3287},"\u002Fadvanced-input-parsing-user-experience\u002Ferror-handling-and-exit-codes\u002Freporting-machine-readable-errors-in-json-mode","Reporting Machine-Readable Errors in JSON Mode",{"path":3289,"title":3290},"\u002Fadvanced-input-parsing-user-experience\u002Fhandling-configuration-files-env-vars\u002Fconfig-precedence-flags-env-files-defaults","Config Precedence: Flags, Env, Files, Defaults",{"path":3292,"title":3293},"\u002Fadvanced-input-parsing-user-experience\u002Fhandling-configuration-files-env-vars\u002Fdiscovering-project-config-files-by-walking-up-directories","Discovering Project Config Files by Walking Up Directories",{"path":3295,"title":3296},"\u002Fadvanced-input-parsing-user-experience\u002Fhandling-configuration-files-env-vars","Handling Config Files and Env Vars in CLIs",{"path":3298,"title":3299},"\u002Fadvanced-input-parsing-user-experience\u002Fhandling-configuration-files-env-vars\u002Floading-yaml-configs-safely-in-cli-apps","Loading YAML configs safely in CLI apps",{"path":3301,"title":3302},"\u002Fadvanced-input-parsing-user-experience\u002Fhandling-configuration-files-env-vars\u002Freading-toml-config-with-tomllib","Reading TOML Config with tomllib in Python CLIs",{"path":3304,"title":3305},"\u002Fadvanced-input-parsing-user-experience\u002Fhandling-configuration-files-env-vars\u002Ftyped-settings-with-pydantic-settings","Typed Settings with pydantic-settings in Python CLIs",{"path":3307,"title":3308},"\u002Fadvanced-input-parsing-user-experience","Advanced Input Parsing for Python CLIs",{"path":3310,"title":3311},"\u002Fadvanced-input-parsing-user-experience\u002Finteractive-terminal-ui-with-rich\u002Fadding-progress-bars-and-spinners-to-python-clis","Progress Bars and Spinners for Python CLIs",{"path":3313,"title":3314},"\u002Fadvanced-input-parsing-user-experience\u002Finteractive-terminal-ui-with-rich\u002Fbuilding-interactive-prompts-and-menus","Building Interactive Prompts and Menus in Python CLIs",{"path":3316,"title":3317},"\u002Fadvanced-input-parsing-user-experience\u002Finteractive-terminal-ui-with-rich","Interactive Terminal UI with Rich",{"path":3319,"title":3320},"\u002Fadvanced-input-parsing-user-experience\u002Finteractive-terminal-ui-with-rich\u002Flive-dashboards-with-rich-live","Live Dashboards with Rich Live in Python CLIs",{"path":3322,"title":3323},"\u002Fadvanced-input-parsing-user-experience\u002Finteractive-terminal-ui-with-rich\u002Frendering-tables-and-json-with-rich","Rendering Tables and JSON with Rich",{"path":3325,"title":3326},"\u002Fadvanced-input-parsing-user-experience\u002Finteractive-terminal-ui-with-rich\u002Ftheming-rich-output-consistently","Theming Rich Output Consistently in Python CLIs",{"path":3328,"title":3329},"\u002Fadvanced-input-parsing-user-experience\u002Fshell-completion-for-python-clis\u002Fdynamic-completion-values-from-apis-and-files","Dynamic Completion Values from APIs and Files",{"path":3331,"title":3332},"\u002Fadvanced-input-parsing-user-experience\u002Fshell-completion-for-python-clis\u002Fenabling-tab-completion-in-click-and-typer","Enabling Tab Completion in Click and Typer",{"path":3334,"title":3335},"\u002Fadvanced-input-parsing-user-experience\u002Fshell-completion-for-python-clis","Shell Completion for Python CLIs",{"path":3337,"title":3338},"\u002Fadvanced-input-parsing-user-experience\u002Fshell-completion-for-python-clis\u002Finstalling-shell-completion-for-bash-zsh-fish","Installing Shell Completion for bash, zsh, fish",{"path":3340,"title":3341},"\u002Fadvanced-input-parsing-user-experience\u002Fshell-completion-for-python-clis\u002Ftesting-shell-completion-in-python-clis","Testing Shell Completion in Python CLIs",{"path":3343,"title":3344},"\u002Fadvanced-input-parsing-user-experience\u002Fstructured-logging-for-cli-apps\u002Fadding-trace-ids-and-context-to-cli-logs","Adding Trace IDs and Context to Python CLI Logs",{"path":3346,"title":3347},"\u002Fadvanced-input-parsing-user-experience\u002Fstructured-logging-for-cli-apps\u002Fadding-verbose-and-quiet-logging-flags","Adding Verbose and Quiet Logging Flags",{"path":3349,"title":3350},"\u002Fadvanced-input-parsing-user-experience\u002Fstructured-logging-for-cli-apps","Structured Logging for CLI Apps",{"path":3352,"title":3353},"\u002Fadvanced-input-parsing-user-experience\u002Fstructured-logging-for-cli-apps\u002Fstructured-json-logging-in-python-clis","Structured JSON Logging in Python CLIs",{"path":3355,"title":3356},"\u002Fadvanced-input-parsing-user-experience\u002Fstructured-logging-for-cli-apps\u002Fwriting-rotating-log-files-from-a-cli","Writing Rotating Log Files from a Python CLI",{"path":3358,"title":3359},"\u002Fadvanced-input-parsing-user-experience\u002Fworking-with-stdin-stdout-and-pipes\u002Fdetecting-tty-and-adapting-output","Detecting a TTY and Adapting Output",{"path":3361,"title":3362},"\u002Fadvanced-input-parsing-user-experience\u002Fworking-with-stdin-stdout-and-pipes\u002Femitting-json-output-for-scripting","Emitting JSON Output for Scripting",{"path":3364,"title":3365},"\u002Fadvanced-input-parsing-user-experience\u002Fworking-with-stdin-stdout-and-pipes\u002Fhandling-broken-pipe-and-sigpipe","Handling Broken Pipe and SIGPIPE",{"path":3367,"title":3368},"\u002Fadvanced-input-parsing-user-experience\u002Fworking-with-stdin-stdout-and-pipes","Working with stdin, stdout and Pipes",{"path":3199,"title":5},{"path":3371,"title":3372},"\u002Fadvanced-input-parsing-user-experience\u002Fworking-with-stdin-stdout-and-pipes\u002Freading-piped-input-in-python-clis","Reading Piped Input in Python CLIs",{"path":3374,"title":3375},"\u002Fcli-runtime-systems-integration\u002Fcalling-http-apis-from-python-clis\u002Fbuilding-an-api-client-cli-with-httpx","Building an API Client CLI with httpx",{"path":3377,"title":3378},"\u002Fcli-runtime-systems-integration\u002Fcalling-http-apis-from-python-clis\u002Fdownloading-files-with-progress-in-python","Downloading Files with Progress in Python CLIs",{"path":3380,"title":3381},"\u002Fcli-runtime-systems-integration\u002Fcalling-http-apis-from-python-clis","Calling HTTP APIs from Python CLIs",{"path":3383,"title":3384},"\u002Fcli-runtime-systems-integration\u002Fcalling-http-apis-from-python-clis\u002Foauth-device-flow-login-for-clis","OAuth Device Flow Login for Python CLIs",{"path":3386,"title":3387},"\u002Fcli-runtime-systems-integration\u002Fcalling-http-apis-from-python-clis\u002Fpaginating-api-results-in-a-cli","Paginating API Results in a Python CLI",{"path":3389,"title":3390},"\u002Fcli-runtime-systems-integration\u002Fcalling-http-apis-from-python-clis\u002Fretries-and-backoff-for-cli-http-calls","Retries and Backoff for CLI HTTP Calls",{"path":3392,"title":3393},"\u002Fcli-runtime-systems-integration\u002Fconcurrency-and-async-in-python-clis\u002Fcancelling-async-tasks-on-ctrl-c","Cancelling Async Tasks on Ctrl+C in Python CLIs",{"path":3395,"title":3396},"\u002Fcli-runtime-systems-integration\u002Fconcurrency-and-async-in-python-clis","Concurrency and Async in Python CLIs",{"path":3398,"title":3399},"\u002Fcli-runtime-systems-integration\u002Fconcurrency-and-async-in-python-clis\u002Fmultiprocessing-for-cpu-bound-cli-tasks","Multiprocessing for CPU-Bound CLI Tasks",{"path":3401,"title":3402},"\u002Fcli-runtime-systems-integration\u002Fconcurrency-and-async-in-python-clis\u002Fparallelising-cli-work-with-thread-pools","Parallelising CLI Work with Thread Pools",{"path":3404,"title":3405},"\u002Fcli-runtime-systems-integration\u002Fconcurrency-and-async-in-python-clis\u002Frate-limiting-concurrent-requests-in-clis","Rate-Limiting Concurrent Requests in Python CLIs",{"path":3407,"title":3408},"\u002Fcli-runtime-systems-integration\u002Fconcurrency-and-async-in-python-clis\u002Frunning-async-code-in-typer-and-click","Running Async Code in Typer and Click",{"path":3410,"title":3411},"\u002Fcli-runtime-systems-integration\u002Ffilesystem-paths-and-atomic-writes\u002Fcross-platform-paths-with-pathlib","Cross-Platform Paths with pathlib in CLIs",{"path":3413,"title":3414},"\u002Fcli-runtime-systems-integration\u002Ffilesystem-paths-and-atomic-writes\u002Ffile-locking-for-concurrent-cli-runs","File Locking for Concurrent CLI Runs in Python",{"path":3416,"title":3417},"\u002Fcli-runtime-systems-integration\u002Ffilesystem-paths-and-atomic-writes","Filesystem Paths and Atomic Writes for CLIs",{"path":3419,"title":3420},"\u002Fcli-runtime-systems-integration\u002Ffilesystem-paths-and-atomic-writes\u002Fsafe-temporary-files-and-directories","Safe Temporary Files and Directories in CLIs",{"path":3422,"title":3423},"\u002Fcli-runtime-systems-integration\u002Ffilesystem-paths-and-atomic-writes\u002Fstoring-app-data-with-platformdirs","Storing CLI App Data with platformdirs",{"path":3425,"title":3426},"\u002Fcli-runtime-systems-integration\u002Ffilesystem-paths-and-atomic-writes\u002Fwriting-files-atomically-in-python-clis","Writing Files Atomically in Python CLIs",{"path":3428,"title":3429},"\u002Fcli-runtime-systems-integration","CLI Runtime & Systems Integration for Python",{"path":3431,"title":3432},"\u002Fcli-runtime-systems-integration\u002Flong-running-and-watch-mode-clis\u002Fbuilding-a-watch-mode-with-watchfiles","Building a Watch Mode with watchfiles in Python",{"path":3434,"title":3435},"\u002Fcli-runtime-systems-integration\u002Flong-running-and-watch-mode-clis\u002Fhandling-sigterm-and-graceful-shutdown","Handling SIGTERM and Graceful Shutdown in CLIs",{"path":3437,"title":3438},"\u002Fcli-runtime-systems-integration\u002Flong-running-and-watch-mode-clis\u002Fhealth-checks-and-heartbeats-for-long-running-clis","Health Checks and Heartbeats for Long-Running CLIs",{"path":3440,"title":3441},"\u002Fcli-runtime-systems-integration\u002Flong-running-and-watch-mode-clis","Long-Running and Watch-Mode Python CLIs",{"path":3443,"title":3444},"\u002Fcli-runtime-systems-integration\u002Flong-running-and-watch-mode-clis\u002Frunning-a-cli-on-a-schedule-with-cron-and-systemd","Running a Python CLI on a Schedule with cron and systemd",{"path":3446,"title":3447},"\u002Fcli-runtime-systems-integration\u002Frunning-subprocesses-from-python-clis\u002Favoiding-shell-injection-in-python-clis","Avoiding Shell Injection in Python CLIs",{"path":3449,"title":3450},"\u002Fcli-runtime-systems-integration\u002Frunning-subprocesses-from-python-clis\u002Fcalling-external-commands-safely-with-subprocess","Calling External Commands Safely with subprocess",{"path":3452,"title":3453},"\u002Fcli-runtime-systems-integration\u002Frunning-subprocesses-from-python-clis\u002Fhandling-subprocess-timeouts-and-exit-codes","Handling Subprocess Timeouts and Exit Codes",{"path":3455,"title":3456},"\u002Fcli-runtime-systems-integration\u002Frunning-subprocesses-from-python-clis","Running Subprocesses from Python CLIs",{"path":3458,"title":3459},"\u002Fcli-runtime-systems-integration\u002Frunning-subprocesses-from-python-clis\u002Fstreaming-subprocess-output-in-real-time","Streaming Subprocess Output in Real Time",{"path":3461,"title":3462},"\u002Fcli-runtime-systems-integration\u002Frunning-subprocesses-from-python-clis\u002Fwrapping-git-and-other-tools-from-a-python-cli","Wrapping git and Other Tools from a Python CLI",{"path":3464,"title":3465},"\u002Fcli-runtime-systems-integration\u002Fsecrets-and-credentials-in-python-clis","Secrets and Credentials in Python CLIs",{"path":3467,"title":3468},"\u002Fcli-runtime-systems-integration\u002Fsecrets-and-credentials-in-python-clis\u002Fprompting-for-passwords-securely","Prompting for Passwords Securely in Python CLIs",{"path":3470,"title":3471},"\u002Fcli-runtime-systems-integration\u002Fsecrets-and-credentials-in-python-clis\u002Freading-secrets-from-env-and-files","Reading Secrets from Env Vars and Files in CLIs",{"path":3473,"title":3474},"\u002Fcli-runtime-systems-integration\u002Fsecrets-and-credentials-in-python-clis\u002Fredacting-secrets-from-cli-output-and-logs","Redacting Secrets from CLI Output and Logs",{"path":3476,"title":3477},"\u002Fcli-runtime-systems-integration\u002Fsecrets-and-credentials-in-python-clis\u002Fstoring-tokens-with-keyring","Storing CLI Tokens Securely with keyring",{"path":3479,"title":3480},"\u002Fcli-runtime-systems-integration\u002Fsecrets-and-credentials-in-python-clis\u002Fsupporting-multiple-profiles-and-accounts","Supporting Multiple Profiles and Accounts in CLIs",{"path":2278,"title":3482},"Python CLI Toolcraft",{"path":3484,"title":3485},"\u002Fmodern-python-cli-frameworks-architecture\u002Fcli-startup-performance-and-lazy-loading\u002Fcaching-expensive-work-between-cli-runs","Caching Expensive Work Between Python CLI Runs",{"path":3487,"title":3488},"\u002Fmodern-python-cli-frameworks-architecture\u002Fcli-startup-performance-and-lazy-loading","CLI Startup Performance and Lazy Loading",{"path":3490,"title":3491},"\u002Fmodern-python-cli-frameworks-architecture\u002Fcli-startup-performance-and-lazy-loading\u002Flazy-loading-subcommands-for-faster-startup","Lazy Loading Subcommands for Faster Startup",{"path":3493,"title":3494},"\u002Fmodern-python-cli-frameworks-architecture\u002Fcli-startup-performance-and-lazy-loading\u002Fprofiling-python-cli-startup-time","Profiling Python CLI Startup Time",{"path":3496,"title":3497},"\u002Fmodern-python-cli-frameworks-architecture\u002Fcli-startup-performance-and-lazy-loading\u002Freducing-cli-dependency-weight","Reducing CLI Dependency Weight",{"path":3499,"title":3500},"\u002Fmodern-python-cli-frameworks-architecture\u002Fcommand-line-parsing-with-argparse\u002Fargparse-subparsers-for-subcommands","argparse Subparsers for Subcommands",{"path":3502,"title":3503},"\u002Fmodern-python-cli-frameworks-architecture\u002Fcommand-line-parsing-with-argparse\u002Fargparse-vs-click-vs-typer-comparison","argparse vs Click vs Typer Compared",{"path":3505,"title":3506},"\u002Fmodern-python-cli-frameworks-architecture\u002Fcommand-line-parsing-with-argparse","Command-Line Parsing with argparse",{"path":3508,"title":3509},"\u002Fmodern-python-cli-frameworks-architecture\u002Fcommand-line-parsing-with-argparse\u002Fmigrating-from-argparse-to-typer","Migrating from argparse to Typer",{"path":3511,"title":3512},"\u002Fmodern-python-cli-frameworks-architecture\u002Fcommand-line-parsing-with-argparse\u002Fmutually-exclusive-options-in-argparse","Mutually Exclusive Options in argparse",{"path":3514,"title":3515},"\u002Fmodern-python-cli-frameworks-architecture\u002Fcommand-line-parsing-with-argparse\u002Fwriting-custom-argparse-actions","Writing Custom argparse Actions in Python",{"path":3517,"title":3518},"\u002Fmodern-python-cli-frameworks-architecture\u002Fdesigning-cli-interfaces-and-conventions\u002Fadding-dry-run-and-confirmation-to-destructive-commands","Adding Dry-Run and Confirmation to Destructive Commands",{"path":3520,"title":3521},"\u002Fmodern-python-cli-frameworks-architecture\u002Fdesigning-cli-interfaces-and-conventions\u002Ffollowing-posix-and-gnu-argument-conventions","Following POSIX and GNU Argument Conventions in Python",{"path":3523,"title":3524},"\u002Fmodern-python-cli-frameworks-architecture\u002Fdesigning-cli-interfaces-and-conventions\u002Fglobal-options-vs-per-command-options","Global Options vs Per-Command Options in Python CLIs",{"path":3526,"title":3527},"\u002Fmodern-python-cli-frameworks-architecture\u002Fdesigning-cli-interfaces-and-conventions","Designing CLI Interfaces and Conventions in Python",{"path":3529,"title":3530},"\u002Fmodern-python-cli-frameworks-architecture\u002Fdesigning-cli-interfaces-and-conventions\u002Fnaming-commands-and-flags-consistently","Naming Commands and Flags Consistently in Python CLIs",{"path":3532,"title":3533},"\u002Fmodern-python-cli-frameworks-architecture","Python CLI Frameworks and Architecture",{"path":3535,"title":3536},"\u002Fmodern-python-cli-frameworks-architecture\u002Fplugin-architectures-for-extensible-clis\u002Fdiscovering-plugins-with-entry-points","Discovering Plugins with Entry Points in Python CLIs",{"path":3538,"title":3539},"\u002Fmodern-python-cli-frameworks-architecture\u002Fplugin-architectures-for-extensible-clis\u002Fhook-based-plugins-with-pluggy","Hook-Based Plugins for Python CLIs with pluggy",{"path":3541,"title":3542},"\u002Fmodern-python-cli-frameworks-architecture\u002Fplugin-architectures-for-extensible-clis","Plugin Architectures for Extensible CLIs",{"path":3544,"title":3545},"\u002Fmodern-python-cli-frameworks-architecture\u002Fplugin-architectures-for-extensible-clis\u002Fversioning-a-plugin-api","Versioning a Plugin API for a Python CLI",{"path":3547,"title":3548},"\u002Fmodern-python-cli-frameworks-architecture\u002Fplugin-architectures-for-extensible-clis\u002Fwriting-a-plugin-for-an-existing-cli","Writing a Plugin for an Existing CLI",{"path":3550,"title":3551},"\u002Fmodern-python-cli-frameworks-architecture\u002Fstructuring-multi-command-python-clis\u002Fbest-practices-for-python-cli-entry-points","Best practices for Python CLI entry points",{"path":3553,"title":3554},"\u002Fmodern-python-cli-frameworks-architecture\u002Fstructuring-multi-command-python-clis\u002Fdependency-injection-patterns-for-cli-commands","Dependency Injection Patterns for CLI Commands",{"path":3556,"title":3557},"\u002Fmodern-python-cli-frameworks-architecture\u002Fstructuring-multi-command-python-clis\u002Fhow-to-structure-a-large-python-cli-project","Structuring a Large Python CLI Project",{"path":3559,"title":3560},"\u002Fmodern-python-cli-frameworks-architecture\u002Fstructuring-multi-command-python-clis","Structuring Multi-Command Python CLIs",{"path":3562,"title":3563},"\u002Fmodern-python-cli-frameworks-architecture\u002Fstructuring-multi-command-python-clis\u002Fsharing-common-options-across-commands","Sharing Common Options Across Python CLI Commands",{"path":3565,"title":3566},"\u002Fmodern-python-cli-frameworks-architecture\u002Fstructuring-multi-command-python-clis\u002Fsharing-state-with-click-context-objects","Sharing State with Click Context Objects",{"path":3568,"title":3569},"\u002Fmodern-python-cli-frameworks-architecture\u002Ftesting-python-cli-applications\u002Fend-to-end-testing-an-installed-cli","End-to-End Testing an Installed Python CLI",{"path":3571,"title":3572},"\u002Fmodern-python-cli-frameworks-architecture\u002Ftesting-python-cli-applications","Testing Python CLI Applications",{"path":3574,"title":3575},"\u002Fmodern-python-cli-frameworks-architecture\u002Ftesting-python-cli-applications\u002Fmeasuring-cli-test-coverage","Measuring CLI Test Coverage",{"path":3577,"title":3578},"\u002Fmodern-python-cli-frameworks-architecture\u002Ftesting-python-cli-applications\u002Fmocking-filesystem-and-network-in-cli-tests","Mocking the Filesystem and Network in CLI Tests",{"path":3580,"title":3581},"\u002Fmodern-python-cli-frameworks-architecture\u002Ftesting-python-cli-applications\u002Fproperty-based-testing-cli-arguments-with-hypothesis","Property-Based Testing CLI Arguments with Hypothesis",{"path":3583,"title":3584},"\u002Fmodern-python-cli-frameworks-architecture\u002Ftesting-python-cli-applications\u002Fsnapshot-testing-cli-output","Snapshot Testing CLI Output",{"path":3586,"title":3587},"\u002Fmodern-python-cli-frameworks-architecture\u002Ftesting-python-cli-applications\u002Ftesting-click-commands-with-clirunner","Testing Click Commands with CliRunner",{"path":3589,"title":3590},"\u002Fmodern-python-cli-frameworks-architecture\u002Ftesting-python-cli-applications\u002Ftesting-interactive-prompts-and-stdin","Testing Interactive Prompts and stdin",{"path":3592,"title":3593},"\u002Fmodern-python-cli-frameworks-architecture\u002Ftyper-vs-click-when-to-use-each\u002Fbuilding-a-cli-with-subcommands-in-click","Building a CLI with subcommands in Click",{"path":3595,"title":3596},"\u002Fmodern-python-cli-frameworks-architecture\u002Ftyper-vs-click-when-to-use-each\u002Fbuilding-dynamic-commands-in-click","Building Dynamic Commands in Click",{"path":3598,"title":3599},"\u002Fmodern-python-cli-frameworks-architecture\u002Ftyper-vs-click-when-to-use-each\u002Fconverting-a-click-app-to-typer","Converting a Click App to Typer",{"path":3601,"title":3602},"\u002Fmodern-python-cli-frameworks-architecture\u002Ftyper-vs-click-when-to-use-each","Typer vs Click: When to Use Each",{"path":3604,"title":3605},"\u002Fmodern-python-cli-frameworks-architecture\u002Ftyper-vs-click-when-to-use-each\u002Ftyper-callback-functions-explained","Typer callback functions explained",{"path":3607,"title":3608},"\u002Fmodern-python-cli-frameworks-architecture\u002Ftyper-vs-click-when-to-use-each\u002Fusing-annotated-options-in-typer","Using Annotated Options in Typer",{"path":3610,"title":3611},"\u002Fproject-setup-dependency-management\u002Fci-cd-pipelines-for-python-clis\u002Fautomating-releases-from-git-tags","Automating Python CLI Releases from Git Tags",{"path":3613,"title":3614},"\u002Fproject-setup-dependency-management\u002Fci-cd-pipelines-for-python-clis\u002Fcaching-uv-dependencies-in-ci","Caching uv Dependencies in CI for Python CLIs",{"path":3616,"title":3617},"\u002Fproject-setup-dependency-management\u002Fci-cd-pipelines-for-python-clis","CI\u002FCD Pipelines for Python CLIs",{"path":3619,"title":3620},"\u002Fproject-setup-dependency-management\u002Fci-cd-pipelines-for-python-clis\u002Fpublishing-to-pypi-with-trusted-publishing","Publishing a CLI to PyPI with Trusted Publishing",{"path":3622,"title":3623},"\u002Fproject-setup-dependency-management\u002Fci-cd-pipelines-for-python-clis\u002Fsmoke-testing-the-built-wheel-in-ci","Smoke-Testing the Built Wheel of a Python CLI in CI",{"path":3625,"title":3626},"\u002Fproject-setup-dependency-management\u002Fci-cd-pipelines-for-python-clis\u002Ftesting-a-cli-across-python-versions-with-github-actions","Testing a CLI Across Python Versions in GitHub Actions",{"path":3628,"title":3629},"\u002Fproject-setup-dependency-management\u002Fcli-project-scaffolding-with-cookiecutter\u002Fbuilding-a-cookiecutter-template-for-typer-clis","Building a Cookiecutter Template for Typer CLIs",{"path":3631,"title":3632},"\u002Fproject-setup-dependency-management\u002Fcli-project-scaffolding-with-cookiecutter\u002Fcopier-vs-cookiecutter-for-cli-templates","Copier vs Cookiecutter for CLI Templates",{"path":3634,"title":3635},"\u002Fproject-setup-dependency-management\u002Fcli-project-scaffolding-with-cookiecutter","CLI Project Scaffolding with Cookiecutter",{"path":3637,"title":3638},"\u002Fproject-setup-dependency-management\u002Fcli-project-scaffolding-with-cookiecutter\u002Fpost-generation-hooks-in-cli-templates","Post-Generation Hooks in Python CLI Templates",{"path":3640,"title":3641},"\u002Fproject-setup-dependency-management\u002Fdistributing-clis-as-standalone-binaries\u002Fbuilding-cross-platform-release-binaries-in-ci","Building Cross-Platform Release Binaries in CI",{"path":3643,"title":3644},"\u002Fproject-setup-dependency-management\u002Fdistributing-clis-as-standalone-binaries\u002Fbundling-a-python-cli-with-pyinstaller","Bundling a Python CLI with PyInstaller",{"path":3646,"title":3647},"\u002Fproject-setup-dependency-management\u002Fdistributing-clis-as-standalone-binaries\u002Fhomebrew-and-scoop-packaging-for-python-clis","Homebrew and Scoop Packaging for Python CLIs",{"path":3649,"title":3650},"\u002Fproject-setup-dependency-management\u002Fdistributing-clis-as-standalone-binaries","Distributing CLIs as Standalone Binaries",{"path":3652,"title":3653},"\u002Fproject-setup-dependency-management\u002Fdistributing-clis-as-standalone-binaries\u002Fnuitka-vs-pyinstaller-for-python-clis","Nuitka vs PyInstaller for Python CLI Binaries",{"path":3655,"title":3656},"\u002Fproject-setup-dependency-management\u002Fdistributing-clis-as-standalone-binaries\u002Fshipping-a-cli-as-a-zipapp-with-shiv","Shipping a CLI as a Zipapp with shiv",{"path":3658,"title":3659},"\u002Fproject-setup-dependency-management","Project Setup & Dependency Management",{"path":3661,"title":3662},"\u002Fproject-setup-dependency-management\u002Flinting-and-type-checking-cli-code\u002Fconfiguring-ruff-for-a-cli-project","Configuring Ruff for a Python CLI Project",{"path":3664,"title":3665},"\u002Fproject-setup-dependency-management\u002Flinting-and-type-checking-cli-code\u002Fenforcing-import-boundaries-in-a-cli-codebase","Enforcing Import Boundaries in a Python CLI Codebase",{"path":3667,"title":3668},"\u002Fproject-setup-dependency-management\u002Flinting-and-type-checking-cli-code","Linting and Type-Checking Python CLI Code",{"path":3670,"title":3671},"\u002Fproject-setup-dependency-management\u002Flinting-and-type-checking-cli-code\u002Ftype-checking-click-and-typer-code-with-mypy","Type-Checking Click and Typer Code with mypy",{"path":3673,"title":3674},"\u002Fproject-setup-dependency-management\u002Fmanaging-cli-versioning-changelogs\u002Fautomating-changelogs-with-conventional-commits","Automating Changelogs with Conventional Commits",{"path":3676,"title":3677},"\u002Fproject-setup-dependency-management\u002Fmanaging-cli-versioning-changelogs\u002Fderiving-versions-from-git-tags-with-hatch-vcs","Deriving CLI Versions from Git Tags with hatch-vcs",{"path":3679,"title":3680},"\u002Fproject-setup-dependency-management\u002Fmanaging-cli-versioning-changelogs\u002Fexposing-version-info-and-build-metadata","Exposing Version Info and Build Metadata",{"path":3682,"title":3683},"\u002Fproject-setup-dependency-management\u002Fmanaging-cli-versioning-changelogs","Managing CLI Versioning & Changelogs",{"path":3685,"title":3686},"\u002Fproject-setup-dependency-management\u002Fmanaging-cli-versioning-changelogs\u002Fsemantic-versioning-policy-for-cli-tools","A Semantic Versioning Policy for CLI Tools",{"path":3688,"title":3689},"\u002Fproject-setup-dependency-management\u002Fpackaging-python-clis-for-distribution\u002Fbuilding-wheels-and-sdists-for-python-clis","Building Wheels and sdists for Python CLIs",{"path":3691,"title":3692},"\u002Fproject-setup-dependency-management\u002Fpackaging-python-clis-for-distribution\u002Fbundling-data-files-with-importlib-resources","Bundling Data Files with importlib.resources in CLIs",{"path":3694,"title":3695},"\u002Fproject-setup-dependency-management\u002Fpackaging-python-clis-for-distribution","Packaging Python CLIs for Distribution",{"path":3697,"title":3698},"\u002Fproject-setup-dependency-management\u002Fpackaging-python-clis-for-distribution\u002Finstalling-and-distributing-clis-with-pipx","Installing and Distributing CLIs with pipx",{"path":3700,"title":3701},"\u002Fproject-setup-dependency-management\u002Fpackaging-python-clis-for-distribution\u002Fpublishing-a-python-cli-to-pypi","Publishing a Python CLI to PyPI",{"path":3703,"title":3704},"\u002Fproject-setup-dependency-management\u002Fpackaging-python-clis-for-distribution\u002Fwriting-pyproject-toml-metadata-for-a-cli","Writing pyproject.toml Metadata for a Python CLI",{"path":3706,"title":3707},"\u002Fproject-setup-dependency-management\u002Fpoetry-workflows-for-cli-development","Poetry Workflows for CLI Development",{"path":3709,"title":3710},"\u002Fproject-setup-dependency-management\u002Fpoetry-workflows-for-cli-development\u002Fmigrating-a-cli-from-poetry-to-uv","Migrating a Python CLI from Poetry to uv",{"path":3712,"title":3713},"\u002Fproject-setup-dependency-management\u002Fpoetry-workflows-for-cli-development\u002Fpoetry-dependency-groups-for-cli-tooling","Poetry Dependency Groups for CLI Tooling",{"path":3715,"title":3716},"\u002Fproject-setup-dependency-management\u002Fpoetry-workflows-for-cli-development\u002Fpoetry-entry-points-and-scripts-for-clis","Poetry Entry Points and Scripts for CLIs",{"path":3718,"title":3719},"\u002Fproject-setup-dependency-management\u002Fpre-commit-hooks-for-cli-projects","Pre-commit Hooks for CLI Projects",{"path":3721,"title":3722},"\u002Fproject-setup-dependency-management\u002Fpre-commit-hooks-for-cli-projects\u002Fsetting-up-pre-commit-for-python-cli-repos","Setting up pre-commit for Python CLI repos",{"path":3724,"title":3725},"\u002Fproject-setup-dependency-management\u002Fpre-commit-hooks-for-cli-projects\u002Fshipping-your-cli-as-a-pre-commit-hook","Shipping Your Python CLI as a pre-commit Hook",{"path":3727,"title":3728},"\u002Fproject-setup-dependency-management\u002Fpre-commit-hooks-for-cli-projects\u002Fwriting-local-pre-commit-hooks-in-python","Writing Local pre-commit Hooks in Python",{"path":3730,"title":3731},"\u002Fproject-setup-dependency-management\u002Fuv-for-python-cli-dependency-management","uv for Python CLI Dependency Management",{"path":3733,"title":3734},"\u002Fproject-setup-dependency-management\u002Fuv-for-python-cli-dependency-management\u002Frunning-one-off-cli-scripts-with-uv-run","Running One-Off CLI Scripts with uv run and PEP 723",{"path":3736,"title":3737},"\u002Fproject-setup-dependency-management\u002Fuv-for-python-cli-dependency-management\u002Fuv-init-vs-poetry-init-for-cli-tools","uv init vs poetry init for CLI tools",{"path":3739,"title":3740},"\u002Fproject-setup-dependency-management\u002Fuv-for-python-cli-dependency-management\u002Fuv-tool-install-vs-pipx-for-clis","uv tool install vs pipx for CLIs",{"path":3742,"title":3743},"\u002Fproject-setup-dependency-management\u002Fuv-for-python-cli-dependency-management\u002Fuv-workspaces-for-multi-package-clis","uv Workspaces for Multi-Package Python CLIs",{"path":3745,"title":3746},"\u002Fproject-setup-dependency-management\u002Fvirtual-environments-isolation-best-practices","Python CLI Env Isolation Best Practices",{"path":3748,"title":3749},"\u002Fproject-setup-dependency-management\u002Fvirtual-environments-isolation-best-practices\u002Fmanaging-virtual-environments-for-cross-platform-clis","Managing Python CLI Virtual Environments",{"path":3751,"title":3752},"\u002Fproject-setup-dependency-management\u002Fvirtual-environments-isolation-best-practices\u002Fpinning-the-python-version-for-a-cli","Pinning the Python Version for a CLI",{"path":3754,"title":3755},"\u002Fproject-setup-dependency-management\u002Fvirtual-environments-isolation-best-practices\u002Fsupporting-multiple-python-versions-with-nox","Supporting Multiple Python Versions with nox",1789736905047]