test: keep the mutation harness — scripts/mutation_check.py, with its own controls

Promotes the session-scratchpad harness that proved U7's twelve falsifiers into
a repo tool, on the operator's call. No version bump: test tooling and docs, no
production-code change, per the SemVer SKIP list.

A green test is not evidence. A test that has never seen its own defeating
change may pass under it too, forbidding nothing while reading as though it
forbids something. This repo shipped that three times — twice in one session,
and once an hour after writing the persistent-memory entry about it. Prose in a
memory file is not an instrument.

Tables live in tests/mutations/*.toml, one per unit, committed so a unit's
proofs are an artifact rather than terminal scrollback. Adding a unit means
adding a file, never editing the script. u7_navigation.toml was generated from
the harness that proved those twelve, not retyped, and every anchor was verified
against the source before it landed.

THE TOOL GETS ITS OWN POSITIVE AND NEGATIVE CONTROLS, which is the point. It
shipped two defects in one session, each of which made it report a falsifier
PROVED WITHOUT RUNNING IT, and both were found by accident rather than by
anything checking:

  no green baseline — a test that is ALREADY red reports red for every mutation
  thrown at it, so a broken assertion reads as a certified falsifier

  the bytecode cache — `< 2` -> `< 1` is byte-identical in size, and CPython
  validates a .pyc against the source's (mtime, size) at one-second granularity,
  so a mutation landing in the same second as the revert before it runs against
  cached bytecode; the tell was a verdict flipping between consecutive identical
  runs

tests/test_mutation_check.py now carries a control for each, plus the one
usually skipped: a KNOWN-VACUOUS falsifier the tool must catch. An instrument
that only ever sees unknowns cannot tell "nothing wrong here" from "I am blind",
and twelve `proved` lines from a blind instrument are worth nothing.

Also hardens the tool against itself: it writes to tracked source files, so the
restore is verified rather than assumed, and a .mutation-inflight marker makes a
run killed mid-mutation refuse the next start instead of silently measuring a
mutated tree.

648 tests green; 12/12 U7 falsifiers still proved.

.gitignore

@@ -6,3 +6,6 @@ __pycache__/
 booth-data/
 uv.lock
 graphify-out/
+
+# scripts/mutation_check.py crash marker — never committed
+.mutation-inflight

CLAUDE.md

@@ -215,6 +215,23 @@ curl -s localhost:8090/healthz         # the live service (systemd --user)
 systemctl --user restart booth.service # after a code change, to see it live
 ```
 
+```sh
+.venv/bin/python scripts/mutation_check.py          # prove the falsifiers still falsify
+```
+
+**A green test is not evidence.** A test that has never seen its own defeating
+change may pass under it too — forbidding nothing while reading as though it
+forbids something. This repo shipped that three times before the tool existed
+(twice in one session, once an hour after writing the entry about it). Tables
+live in `tests/mutations/*.toml`, one per unit, committed so a unit's proofs are
+an artifact rather than scrollback; adding a unit means adding a file, never
+editing the script. `tests/test_mutation_check.py` holds the tool's own positive
+and negative controls, because an instrument that only ever sees unknowns cannot
+tell "nothing wrong" from "I am blind".
+
+When you add a `*Falsifiable:*` line to a contract, add its row to the table and
+run it. A falsifier nobody has run is a claim, not a test.
+
 `tests/test_embed_browser.py` drives a real Chromium against a real uvicorn on
 an ephemeral port — the only place U3's placement and `form=` binding can be
 observed at all. Browsers are NOT downloaded per project; they live box-wide in

scripts/mutation_check.py

@@ -0,0 +1,131 @@
+#!/usr/bin/env python3
+"""Prove a falsifier falsifies, by running the change it forbids.
+
+A green test is not evidence. A test that has never seen its own DEFEATING
+CHANGE is only evidence that the code and the assertion agree today; it may
+agree under the mutation too, in which case it forbids nothing and reads as
+though it forbids something. This repo has shipped that three times --
+persistent-memory.d/2026-09-22-vacuous-falsifiers.md,
+persistent-memory.d/2026-09-22-seven-of-seven-falsifiers.md, and once more in
+U7 an hour after the second was written.
+
+So: for each declared mutation, apply it to the source, run the one test that
+claims to catch it, and require RED. Revert either way.
+
+    .venv/bin/python scripts/mutation_check.py                # every table
+    .venv/bin/python scripts/mutation_check.py u7_navigation  # one table
+
+Tables live in tests/mutations/*.toml and are committed, so a unit's proofs are
+an artifact rather than terminal scrollback. Adding a unit means adding a file,
+never editing this script.
+
+⚠ TWO DEFECTS THIS TOOL HAD, both of which made it CERTIFY A FALSIFIER WITHOUT
+RUNNING IT. Neither is obvious and both cost real time:
+
+1. NO GREEN BASELINE. A test that is ALREADY red reports red for every mutation
+   thrown at it, so a broken assertion reads as a proven falsifier. Every run
+   now checks the test passes unmutated first; a red baseline is a harness
+   failure, reported as such, never as a proof.
+
+2. THE BYTECODE CACHE. `< 2` -> `< 1` is BYTE-IDENTICAL IN SIZE, and CPython
+   validates a .pyc against the source's (mtime, size) at ONE-SECOND
+   granularity -- so a mutation landing in the same second as the revert before
+   it is invisible and the unmutated code runs. The tell was a verdict that
+   flipped between consecutive runs with nothing changed. Caches are dropped
+   and PYTHONDONTWRITEBYTECODE is set for every run. This biases toward exactly
+   the mutations most worth making: comparison flips, off-by-one constants,
+   and/or swaps.
+"""
+
+from __future__ import annotations
+
+import os
+import shutil
+import subprocess
+import sys
+import tomllib
+from pathlib import Path
+
+REPO = Path(__file__).resolve().parent.parent
+TABLES = REPO / "tests" / "mutations"
+# Written before a source file is touched and removed after it is restored. Its
+# presence at startup means a previous run died between the two -- a `kill -9`
+# mid-mutation leaves a mutated tracked file that looks like authored code.
+INFLIGHT = REPO / ".mutation-inflight"
+
+
+def run(test: str, repo: Path = REPO) -> int:
+    """Exit code of one test, with the bytecode cache defeated. See defect 2."""
+    for cache in repo.rglob("__pycache__"):
+        shutil.rmtree(cache, ignore_errors=True)
+    return subprocess.run(
+        [sys.executable, "-m", "pytest", test, "-q", "--no-header", "-p", "no:warnings"],
+        cwd=repo, capture_output=True, text=True,
+        env=dict(os.environ, PYTHONDONTWRITEBYTECODE="1"),
+    ).returncode
+
+
+def check(mutation: dict, repo: Path = REPO) -> tuple[bool, str]:
+    """(proved, note) for one mutation. Never leaves the source mutated.
+
+    `repo` is a parameter so the harness can be pointed at a throwaway tree and
+    given KNOWN-vacuous and KNOWN-good falsifiers — see
+    tests/test_mutation_check.py. An instrument that only ever sees unknowns
+    cannot tell "nothing wrong here" from "I am blind", which is the whole of
+    CLAUDE.md's positive-control rule applied to the tool that enforces it."""
+    test = mutation["test"]
+    path = repo / mutation["file"]
+
+    if run(test, repo) != 0:
+        return False, f"BASELINE RED — {test} fails BEFORE the mutation"
+
+    src = path.read_text()
+    if mutation["old"] not in src:
+        return False, f"anchor not found in {mutation['file']} — the table has drifted"
+
+    INFLIGHT.write_text(f"{path}\n")
+    try:
+        path.write_text(src.replace(mutation["old"], mutation["new"], 1))
+        red = run(test, repo) != 0
+    finally:
+        path.write_text(src)
+        # Verified, not assumed: a restore that silently failed would leave a
+        # mutation in a tracked file and the next run would measure it.
+        assert path.read_text() == src, f"RESTORE FAILED for {path} — fix by hand"
+        INFLIGHT.unlink(missing_ok=True)
+
+    return red, "" if red else "VACUOUS — stayed green under the change it forbids"
+
+
+def main(argv: list[str]) -> int:
+    if INFLIGHT.exists():
+        print(f"refusing to run: {INFLIGHT} exists, so a previous run died mid-mutation.")
+        print(f"check `git diff {INFLIGHT.read_text().strip()}`, restore it, then delete the marker.")
+        return 2
+
+    wanted = argv[1:] or None
+    tables = sorted(TABLES.glob("*.toml"))
+    if wanted:
+        tables = [t for t in tables if t.stem in wanted]
+        if not tables:
+            print(f"no table matching {wanted} in {TABLES}")
+            return 2
+
+    failed = []
+    for table in tables:
+        doc = tomllib.loads(table.read_text())
+        print(f"\n### {table.stem} — {doc.get('unit', '')}")
+        for m in doc.get("mutation", []):
+            proved, note = check(m)
+            print(f"{'  proved' if proved else '  NOT PROVED':14s}  {m['label']}")
+            if not proved:
+                print(f"{'':14s}  ^ {note}")
+                failed.append(m["label"])
+
+    total = sum(len(tomllib.loads(t.read_text()).get("mutation", [])) for t in tables)
+    print(f"\n{total - len(failed)}/{total} falsifiers proved by running the change they forbid")
+    return 1 if failed else 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main(sys.argv))

tests/mutations/u7_navigation.toml

@@ -0,0 +1,120 @@
+# U7 — every falsifier the navigation unit claims, and the change it forbids.
+#
+# Generated from the session harness that proved them, not retyped. Each `old`
+# must match the source byte-for-byte; a drifted anchor is REPORTED, never
+# silently skipped — a table that stops matching stops proving anything.
+
+unit = "the rail, the filters, the grid keyboard, and the groups"
+
+[[mutation]]
+label = "INV-2 sort the grid by (group, rel) so groups render contiguously"
+file = "booth/app.py"
+test = "tests/test_navigation.py::test_grouping_never_reorders_the_grid"
+old = '''
+        shown = buckets[active]'''
+new = '''
+        shown = sorted(buckets[active], key=lambda i: (i["group"] or "", i["name"]))'''
+
+[[mutation]]
+label = "INV-3a drop the >=2 groups guard (a rail with one row)"
+file = "booth/app.py"
+test = "tests/test_navigation.py::test_no_group_rail_when_there_is_only_one_group"
+old = '''
+        if len(sizes) < 2 or sizes[len(sizes) // 2] <= 1:'''
+new = '''
+        if len(sizes) < 1 or sizes[len(sizes) // 2] <= 1:'''
+
+[[mutation]]
+label = "INV-3b drop the median guard (a rail that is a second copy of the grid)"
+file = "booth/app.py"
+test = "tests/test_navigation.py::test_no_group_rail_when_every_item_is_its_own_group"
+old = '''
+        if len(sizes) < 2 or sizes[len(sizes) // 2] <= 1:'''
+new = '''
+        if len(sizes) < 2:'''
+
+[[mutation]]
+label = "groups derived from the FULL gallery, not the rendered list"
+file = "booth/app.py"
+test = "tests/test_navigation.py::test_groups_describe_the_filtered_grid"
+old = '''
+            "groups": _groups(shown),'''
+new = '''
+            "groups": _groups(gallery),'''
+
+[[mutation]]
+label = "the anchor names the group key instead of the tile id"
+file = "booth/app.py"
+test = "tests/test_navigation.py::test_every_group_anchor_lands_on_a_rendered_tile"
+old = '''
+{"key": k, "n": len(v), "anchor": f"item-{v[0]['name']}"}'''
+new = '''
+{"key": k, "n": len(v), "anchor": f"group-{k}"}'''
+
+[[mutation]]
+label = "the rail orders groups alphabetically instead of by first member"
+file = "booth/app.py"
+test = "tests/test_navigation.py::test_group_order_is_the_position_of_the_first_member"
+old = '''
+            for k, v in by_group.items()'''
+new = '''
+            for k, v in sorted(by_group.items())'''
+
+[[mutation]]
+label = "the rail orders groups by count, which the docstring also claims differs"
+file = "booth/app.py"
+test = "tests/test_navigation.py::test_group_order_is_the_position_of_the_first_member"
+old = '''
+            for k, v in by_group.items()'''
+new = '''
+            for k, v in sorted(by_group.items(), key=lambda kv: -len(kv[1]))'''
+
+[[mutation]]
+label = "_group_of reverts to the contract's original strip-trailing-digits rule"
+file = "booth/items.py"
+test = "tests/test_items.py::test_group_of_takes_the_first_segment"
+old = '''
+    segs = _SEG.split(stem)
+    if len(segs) == 1:
+        return re.sub(r"\d+$", "", stem) or None
+    return segs[0] or None'''
+new = '''
+    m = re.match(r"^(.*?)[-_. ]?\d+$", stem)
+    return (m.group(1) or None) if m else (stem or None)'''
+
+[[mutation]]
+label = "INV-1 a route body derives the group inline"
+file = "booth/app.py"
+test = "tests/test_navigation.py::test_no_route_body_derives_a_group"
+old = '''
+        by_group: dict[str, list[dict]] = {}'''
+new = '''
+        _ = _group_of  # noqa
+        by_group: dict[str, list[dict]] = {}'''
+
+[[mutation]]
+label = "the rail markup is emitted with |safe"
+file = "booth/templates/booth.html"
+test = "tests/test_navigation.py::test_a_hostile_filename_cannot_break_out_of_the_rail"
+old = '''
+href="#{{ g.anchor }}">{{ g.key }} <b>{{ g.n }}</b></a>'''
+new = '''
+href="#{{ g.anchor }}">{{ g.key|safe }} <b>{{ g.n }}</b></a>'''
+
+[[mutation]]
+label = "a flat all-digit stem yields the empty string instead of None"
+file = "booth/items.py"
+test = "tests/test_navigation.py::test_a_group_key_is_never_the_empty_string"
+old = '''
+        return re.sub(r"\d+$", "", stem) or None'''
+new = '''
+        return re.sub(r"\d+$", "", stem)'''
+
+[[mutation]]
+label = "the template renders the group row whenever there is any group at all"
+file = "booth/templates/booth.html"
+test = "tests/test_navigation.py::test_no_group_rail_when_every_item_is_its_own_group"
+old = '''
+    {% if rail.groups %}'''
+new = '''
+    {% if rail.groups is not none %}'''

tests/test_mutation_check.py

@@ -0,0 +1,110 @@
+"""Controls for the instrument that certifies every other falsifier.
+
+`scripts/mutation_check.py` exists because a green test proves nothing until it
+has seen the change it forbids. The same sentence applies to the tool: it
+shipped two defects in one session, each of which made it report a falsifier
+PROVED WITHOUT RUNNING IT (no green baseline; the pyc cache silently reverting
+byte-identical mutations). Both were found by accident.
+
+So the tool gets what CLAUDE.md demands of any measurement: a POSITIVE CONTROL
+it must detect, and a NEGATIVE CONTROL it must not fire on. An instrument that
+only ever sees unknowns cannot distinguish "absent" from "blind".
+"""
+
+from __future__ import annotations
+
+import pathlib
+import sys
+
+sys.path.insert(0, str(pathlib.Path(__file__).parent.parent / "scripts"))
+
+from mutation_check import check  # noqa: E402
+
+
+def _tree(tmp_path, source: str, test_body: str):
+    """A throwaway repo: one module, one test file, both real on disk."""
+    (tmp_path / "mod.py").write_text(source)
+    (tmp_path / "test_probe.py").write_text(
+        "import sys, pathlib\n"
+        "sys.path.insert(0, str(pathlib.Path(__file__).parent))\n"
+        "from mod import f\n\n" + test_body
+    )
+    return tmp_path
+
+
+def test_a_real_falsifier_is_reported_proved(tmp_path):
+    """NEGATIVE CONTROL — the tool must not cry wolf on a sound test.
+
+    `f` returns 2; the test asserts it. Flipping the constant must go red, and
+    the tool must say so."""
+    repo = _tree(tmp_path, "def f():\n    return 2\n",
+                 "def test_f():\n    assert f() == 2\n")
+    proved, note = check(
+        {"label": "flip the constant", "file": "mod.py", "test": "test_probe.py::test_f",
+         "old": "return 2", "new": "return 3"}, repo=repo)
+    assert proved, note
+
+
+def test_a_vacuous_falsifier_is_caught(tmp_path):
+    """POSITIVE CONTROL — the one that matters, and the one usually skipped.
+
+    The test asserts only that `f()` is an int, so flipping the constant does
+    NOT break it. The test cites the behaviour without forbidding it. The tool
+    must report NOT PROVED; if it cannot detect a known-vacuous falsifier, its
+    twelve `proved` lines are worth nothing."""
+    repo = _tree(tmp_path, "def f():\n    return 2\n",
+                 "def test_f():\n    assert isinstance(f(), int)\n")
+    proved, note = check(
+        {"label": "flip the constant", "file": "mod.py", "test": "test_probe.py::test_f",
+         "old": "return 2", "new": "return 3"}, repo=repo)
+    assert not proved
+    assert "VACUOUS" in note
+
+
+def test_an_already_red_test_is_a_harness_failure_not_a_proof(tmp_path):
+    """DEFECT 1, as a control. Before the baseline check this returned PROVED —
+    a broken assertion reading as a certified falsifier."""
+    repo = _tree(tmp_path, "def f():\n    return 2\n",
+                 "def test_f():\n    assert f() == 99\n")
+    proved, note = check(
+        {"label": "flip the constant", "file": "mod.py", "test": "test_probe.py::test_f",
+         "old": "return 2", "new": "return 3"}, repo=repo)
+    assert not proved
+    assert "BASELINE RED" in note
+
+
+def test_a_same_size_mutation_is_not_swallowed_by_the_bytecode_cache(tmp_path):
+    """DEFECT 2, as a control. `< 2` -> `< 1` is byte-identical in size, so a
+    mutation landing in the same mtime second as the revert before it used to
+    run against cached bytecode and report PROVED having tested nothing.
+
+    Run twice: the verdict must be stable. The original defect's tell was
+    exactly a verdict that flipped between consecutive identical runs."""
+    repo = _tree(tmp_path, "def f(n):\n    return n < 2\n",
+                 "def test_f():\n    assert f(1) is True and f(2) is False\n")
+    m = {"label": "off by one", "file": "mod.py", "test": "test_probe.py::test_f",
+         "old": "return n < 2", "new": "return n < 1"}
+    assert [check(m, repo=repo)[0] for _ in range(2)] == [True, True]
+
+
+def test_a_drifted_anchor_is_reported_not_skipped(tmp_path):
+    """A table whose `old` no longer matches the source stops proving anything.
+    Silently skipping it would shrink the denominator and keep the run green."""
+    repo = _tree(tmp_path, "def f():\n    return 2\n",
+                 "def test_f():\n    assert f() == 2\n")
+    proved, note = check(
+        {"label": "stale", "file": "mod.py", "test": "test_probe.py::test_f",
+         "old": "return 2222", "new": "return 3"}, repo=repo)
+    assert not proved
+    assert "anchor not found" in note
+
+
+def test_the_source_is_restored_even_when_the_mutation_proves(tmp_path):
+    """The tool writes to tracked source files. Leaving one mutated would put a
+    defect in the tree that looks like authored code."""
+    repo = _tree(tmp_path, "def f():\n    return 2\n",
+                 "def test_f():\n    assert f() == 2\n")
+    before = (repo / "mod.py").read_text()
+    check({"label": "flip", "file": "mod.py", "test": "test_probe.py::test_f",
+           "old": "return 2", "new": "return 3"}, repo=repo)
+    assert (repo / "mod.py").read_text() == before