diff --git a/.gitignore b/.gitignore index 6651622..3312be7 100644 --- a/.gitignore +++ b/.gitignore @@ -6,3 +6,6 @@ __pycache__/ booth-data/ uv.lock graphify-out/ + +# scripts/mutation_check.py crash marker — never committed +.mutation-inflight diff --git a/CLAUDE.md b/CLAUDE.md index c7656dd..5ba62d2 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -215,6 +215,23 @@ curl -s localhost:8090/healthz # the live service (systemd --user) systemctl --user restart booth.service # after a code change, to see it live ``` +```sh +.venv/bin/python scripts/mutation_check.py # prove the falsifiers still falsify +``` + +**A green test is not evidence.** A test that has never seen its own defeating +change may pass under it too — forbidding nothing while reading as though it +forbids something. This repo shipped that three times before the tool existed +(twice in one session, once an hour after writing the entry about it). Tables +live in `tests/mutations/*.toml`, one per unit, committed so a unit's proofs are +an artifact rather than scrollback; adding a unit means adding a file, never +editing the script. `tests/test_mutation_check.py` holds the tool's own positive +and negative controls, because an instrument that only ever sees unknowns cannot +tell "nothing wrong" from "I am blind". + +When you add a `*Falsifiable:*` line to a contract, add its row to the table and +run it. A falsifier nobody has run is a claim, not a test. + `tests/test_embed_browser.py` drives a real Chromium against a real uvicorn on an ephemeral port — the only place U3's placement and `form=` binding can be observed at all. Browsers are NOT downloaded per project; they live box-wide in diff --git a/scripts/mutation_check.py b/scripts/mutation_check.py new file mode 100755 index 0000000..f64b485 --- /dev/null +++ b/scripts/mutation_check.py @@ -0,0 +1,131 @@ +#!/usr/bin/env python3 +"""Prove a falsifier falsifies, by running the change it forbids. + +A green test is not evidence. A test that has never seen its own DEFEATING +CHANGE is only evidence that the code and the assertion agree today; it may +agree under the mutation too, in which case it forbids nothing and reads as +though it forbids something. This repo has shipped that three times -- +persistent-memory.d/2026-09-22-vacuous-falsifiers.md, +persistent-memory.d/2026-09-22-seven-of-seven-falsifiers.md, and once more in +U7 an hour after the second was written. + +So: for each declared mutation, apply it to the source, run the one test that +claims to catch it, and require RED. Revert either way. + + .venv/bin/python scripts/mutation_check.py # every table + .venv/bin/python scripts/mutation_check.py u7_navigation # one table + +Tables live in tests/mutations/*.toml and are committed, so a unit's proofs are +an artifact rather than terminal scrollback. Adding a unit means adding a file, +never editing this script. + +⚠ TWO DEFECTS THIS TOOL HAD, both of which made it CERTIFY A FALSIFIER WITHOUT +RUNNING IT. Neither is obvious and both cost real time: + +1. NO GREEN BASELINE. A test that is ALREADY red reports red for every mutation + thrown at it, so a broken assertion reads as a proven falsifier. Every run + now checks the test passes unmutated first; a red baseline is a harness + failure, reported as such, never as a proof. + +2. THE BYTECODE CACHE. `< 2` -> `< 1` is BYTE-IDENTICAL IN SIZE, and CPython + validates a .pyc against the source's (mtime, size) at ONE-SECOND + granularity -- so a mutation landing in the same second as the revert before + it is invisible and the unmutated code runs. The tell was a verdict that + flipped between consecutive runs with nothing changed. Caches are dropped + and PYTHONDONTWRITEBYTECODE is set for every run. This biases toward exactly + the mutations most worth making: comparison flips, off-by-one constants, + and/or swaps. +""" + +from __future__ import annotations + +import os +import shutil +import subprocess +import sys +import tomllib +from pathlib import Path + +REPO = Path(__file__).resolve().parent.parent +TABLES = REPO / "tests" / "mutations" +# Written before a source file is touched and removed after it is restored. Its +# presence at startup means a previous run died between the two -- a `kill -9` +# mid-mutation leaves a mutated tracked file that looks like authored code. +INFLIGHT = REPO / ".mutation-inflight" + + +def run(test: str, repo: Path = REPO) -> int: + """Exit code of one test, with the bytecode cache defeated. See defect 2.""" + for cache in repo.rglob("__pycache__"): + shutil.rmtree(cache, ignore_errors=True) + return subprocess.run( + [sys.executable, "-m", "pytest", test, "-q", "--no-header", "-p", "no:warnings"], + cwd=repo, capture_output=True, text=True, + env=dict(os.environ, PYTHONDONTWRITEBYTECODE="1"), + ).returncode + + +def check(mutation: dict, repo: Path = REPO) -> tuple[bool, str]: + """(proved, note) for one mutation. Never leaves the source mutated. + + `repo` is a parameter so the harness can be pointed at a throwaway tree and + given KNOWN-vacuous and KNOWN-good falsifiers — see + tests/test_mutation_check.py. An instrument that only ever sees unknowns + cannot tell "nothing wrong here" from "I am blind", which is the whole of + CLAUDE.md's positive-control rule applied to the tool that enforces it.""" + test = mutation["test"] + path = repo / mutation["file"] + + if run(test, repo) != 0: + return False, f"BASELINE RED — {test} fails BEFORE the mutation" + + src = path.read_text() + if mutation["old"] not in src: + return False, f"anchor not found in {mutation['file']} — the table has drifted" + + INFLIGHT.write_text(f"{path}\n") + try: + path.write_text(src.replace(mutation["old"], mutation["new"], 1)) + red = run(test, repo) != 0 + finally: + path.write_text(src) + # Verified, not assumed: a restore that silently failed would leave a + # mutation in a tracked file and the next run would measure it. + assert path.read_text() == src, f"RESTORE FAILED for {path} — fix by hand" + INFLIGHT.unlink(missing_ok=True) + + return red, "" if red else "VACUOUS — stayed green under the change it forbids" + + +def main(argv: list[str]) -> int: + if INFLIGHT.exists(): + print(f"refusing to run: {INFLIGHT} exists, so a previous run died mid-mutation.") + print(f"check `git diff {INFLIGHT.read_text().strip()}`, restore it, then delete the marker.") + return 2 + + wanted = argv[1:] or None + tables = sorted(TABLES.glob("*.toml")) + if wanted: + tables = [t for t in tables if t.stem in wanted] + if not tables: + print(f"no table matching {wanted} in {TABLES}") + return 2 + + failed = [] + for table in tables: + doc = tomllib.loads(table.read_text()) + print(f"\n### {table.stem} — {doc.get('unit', '')}") + for m in doc.get("mutation", []): + proved, note = check(m) + print(f"{' proved' if proved else ' NOT PROVED':14s} {m['label']}") + if not proved: + print(f"{'':14s} ^ {note}") + failed.append(m["label"]) + + total = sum(len(tomllib.loads(t.read_text()).get("mutation", [])) for t in tables) + print(f"\n{total - len(failed)}/{total} falsifiers proved by running the change they forbid") + return 1 if failed else 0 + + +if __name__ == "__main__": + raise SystemExit(main(sys.argv)) diff --git a/tests/mutations/u7_navigation.toml b/tests/mutations/u7_navigation.toml new file mode 100644 index 0000000..6923dc3 --- /dev/null +++ b/tests/mutations/u7_navigation.toml @@ -0,0 +1,120 @@ +# U7 — every falsifier the navigation unit claims, and the change it forbids. +# +# Generated from the session harness that proved them, not retyped. Each `old` +# must match the source byte-for-byte; a drifted anchor is REPORTED, never +# silently skipped — a table that stops matching stops proving anything. + +unit = "the rail, the filters, the grid keyboard, and the groups" + +[[mutation]] +label = "INV-2 sort the grid by (group, rel) so groups render contiguously" +file = "booth/app.py" +test = "tests/test_navigation.py::test_grouping_never_reorders_the_grid" +old = ''' + shown = buckets[active]''' +new = ''' + shown = sorted(buckets[active], key=lambda i: (i["group"] or "", i["name"]))''' + +[[mutation]] +label = "INV-3a drop the >=2 groups guard (a rail with one row)" +file = "booth/app.py" +test = "tests/test_navigation.py::test_no_group_rail_when_there_is_only_one_group" +old = ''' + if len(sizes) < 2 or sizes[len(sizes) // 2] <= 1:''' +new = ''' + if len(sizes) < 1 or sizes[len(sizes) // 2] <= 1:''' + +[[mutation]] +label = "INV-3b drop the median guard (a rail that is a second copy of the grid)" +file = "booth/app.py" +test = "tests/test_navigation.py::test_no_group_rail_when_every_item_is_its_own_group" +old = ''' + if len(sizes) < 2 or sizes[len(sizes) // 2] <= 1:''' +new = ''' + if len(sizes) < 2:''' + +[[mutation]] +label = "groups derived from the FULL gallery, not the rendered list" +file = "booth/app.py" +test = "tests/test_navigation.py::test_groups_describe_the_filtered_grid" +old = ''' + "groups": _groups(shown),''' +new = ''' + "groups": _groups(gallery),''' + +[[mutation]] +label = "the anchor names the group key instead of the tile id" +file = "booth/app.py" +test = "tests/test_navigation.py::test_every_group_anchor_lands_on_a_rendered_tile" +old = ''' +{"key": k, "n": len(v), "anchor": f"item-{v[0]['name']}"}''' +new = ''' +{"key": k, "n": len(v), "anchor": f"group-{k}"}''' + +[[mutation]] +label = "the rail orders groups alphabetically instead of by first member" +file = "booth/app.py" +test = "tests/test_navigation.py::test_group_order_is_the_position_of_the_first_member" +old = ''' + for k, v in by_group.items()''' +new = ''' + for k, v in sorted(by_group.items())''' + +[[mutation]] +label = "the rail orders groups by count, which the docstring also claims differs" +file = "booth/app.py" +test = "tests/test_navigation.py::test_group_order_is_the_position_of_the_first_member" +old = ''' + for k, v in by_group.items()''' +new = ''' + for k, v in sorted(by_group.items(), key=lambda kv: -len(kv[1]))''' + +[[mutation]] +label = "_group_of reverts to the contract's original strip-trailing-digits rule" +file = "booth/items.py" +test = "tests/test_items.py::test_group_of_takes_the_first_segment" +old = ''' + segs = _SEG.split(stem) + if len(segs) == 1: + return re.sub(r"\d+$", "", stem) or None + return segs[0] or None''' +new = ''' + m = re.match(r"^(.*?)[-_. ]?\d+$", stem) + return (m.group(1) or None) if m else (stem or None)''' + +[[mutation]] +label = "INV-1 a route body derives the group inline" +file = "booth/app.py" +test = "tests/test_navigation.py::test_no_route_body_derives_a_group" +old = ''' + by_group: dict[str, list[dict]] = {}''' +new = ''' + _ = _group_of # noqa + by_group: dict[str, list[dict]] = {}''' + +[[mutation]] +label = "the rail markup is emitted with |safe" +file = "booth/templates/booth.html" +test = "tests/test_navigation.py::test_a_hostile_filename_cannot_break_out_of_the_rail" +old = ''' +href="#{{ g.anchor }}">{{ g.key }} {{ g.n }}''' +new = ''' +href="#{{ g.anchor }}">{{ g.key|safe }} {{ g.n }}''' + +[[mutation]] +label = "a flat all-digit stem yields the empty string instead of None" +file = "booth/items.py" +test = "tests/test_navigation.py::test_a_group_key_is_never_the_empty_string" +old = ''' + return re.sub(r"\d+$", "", stem) or None''' +new = ''' + return re.sub(r"\d+$", "", stem)''' + +[[mutation]] +label = "the template renders the group row whenever there is any group at all" +file = "booth/templates/booth.html" +test = "tests/test_navigation.py::test_no_group_rail_when_every_item_is_its_own_group" +old = ''' + {% if rail.groups %}''' +new = ''' + {% if rail.groups is not none %}''' diff --git a/tests/test_mutation_check.py b/tests/test_mutation_check.py new file mode 100644 index 0000000..09c262e --- /dev/null +++ b/tests/test_mutation_check.py @@ -0,0 +1,110 @@ +"""Controls for the instrument that certifies every other falsifier. + +`scripts/mutation_check.py` exists because a green test proves nothing until it +has seen the change it forbids. The same sentence applies to the tool: it +shipped two defects in one session, each of which made it report a falsifier +PROVED WITHOUT RUNNING IT (no green baseline; the pyc cache silently reverting +byte-identical mutations). Both were found by accident. + +So the tool gets what CLAUDE.md demands of any measurement: a POSITIVE CONTROL +it must detect, and a NEGATIVE CONTROL it must not fire on. An instrument that +only ever sees unknowns cannot distinguish "absent" from "blind". +""" + +from __future__ import annotations + +import pathlib +import sys + +sys.path.insert(0, str(pathlib.Path(__file__).parent.parent / "scripts")) + +from mutation_check import check # noqa: E402 + + +def _tree(tmp_path, source: str, test_body: str): + """A throwaway repo: one module, one test file, both real on disk.""" + (tmp_path / "mod.py").write_text(source) + (tmp_path / "test_probe.py").write_text( + "import sys, pathlib\n" + "sys.path.insert(0, str(pathlib.Path(__file__).parent))\n" + "from mod import f\n\n" + test_body + ) + return tmp_path + + +def test_a_real_falsifier_is_reported_proved(tmp_path): + """NEGATIVE CONTROL — the tool must not cry wolf on a sound test. + + `f` returns 2; the test asserts it. Flipping the constant must go red, and + the tool must say so.""" + repo = _tree(tmp_path, "def f():\n return 2\n", + "def test_f():\n assert f() == 2\n") + proved, note = check( + {"label": "flip the constant", "file": "mod.py", "test": "test_probe.py::test_f", + "old": "return 2", "new": "return 3"}, repo=repo) + assert proved, note + + +def test_a_vacuous_falsifier_is_caught(tmp_path): + """POSITIVE CONTROL — the one that matters, and the one usually skipped. + + The test asserts only that `f()` is an int, so flipping the constant does + NOT break it. The test cites the behaviour without forbidding it. The tool + must report NOT PROVED; if it cannot detect a known-vacuous falsifier, its + twelve `proved` lines are worth nothing.""" + repo = _tree(tmp_path, "def f():\n return 2\n", + "def test_f():\n assert isinstance(f(), int)\n") + proved, note = check( + {"label": "flip the constant", "file": "mod.py", "test": "test_probe.py::test_f", + "old": "return 2", "new": "return 3"}, repo=repo) + assert not proved + assert "VACUOUS" in note + + +def test_an_already_red_test_is_a_harness_failure_not_a_proof(tmp_path): + """DEFECT 1, as a control. Before the baseline check this returned PROVED — + a broken assertion reading as a certified falsifier.""" + repo = _tree(tmp_path, "def f():\n return 2\n", + "def test_f():\n assert f() == 99\n") + proved, note = check( + {"label": "flip the constant", "file": "mod.py", "test": "test_probe.py::test_f", + "old": "return 2", "new": "return 3"}, repo=repo) + assert not proved + assert "BASELINE RED" in note + + +def test_a_same_size_mutation_is_not_swallowed_by_the_bytecode_cache(tmp_path): + """DEFECT 2, as a control. `< 2` -> `< 1` is byte-identical in size, so a + mutation landing in the same mtime second as the revert before it used to + run against cached bytecode and report PROVED having tested nothing. + + Run twice: the verdict must be stable. The original defect's tell was + exactly a verdict that flipped between consecutive identical runs.""" + repo = _tree(tmp_path, "def f(n):\n return n < 2\n", + "def test_f():\n assert f(1) is True and f(2) is False\n") + m = {"label": "off by one", "file": "mod.py", "test": "test_probe.py::test_f", + "old": "return n < 2", "new": "return n < 1"} + assert [check(m, repo=repo)[0] for _ in range(2)] == [True, True] + + +def test_a_drifted_anchor_is_reported_not_skipped(tmp_path): + """A table whose `old` no longer matches the source stops proving anything. + Silently skipping it would shrink the denominator and keep the run green.""" + repo = _tree(tmp_path, "def f():\n return 2\n", + "def test_f():\n assert f() == 2\n") + proved, note = check( + {"label": "stale", "file": "mod.py", "test": "test_probe.py::test_f", + "old": "return 2222", "new": "return 3"}, repo=repo) + assert not proved + assert "anchor not found" in note + + +def test_the_source_is_restored_even_when_the_mutation_proves(tmp_path): + """The tool writes to tracked source files. Leaving one mutated would put a + defect in the tree that looks like authored code.""" + repo = _tree(tmp_path, "def f():\n return 2\n", + "def test_f():\n assert f() == 2\n") + before = (repo / "mod.py").read_text() + check({"label": "flip", "file": "mod.py", "test": "test_probe.py::test_f", + "old": "return 2", "new": "return 3"}, repo=repo) + assert (repo / "mod.py").read_text() == before