test: keep the mutation harness — scripts/mutation_check.py, with its own controls
Promotes the session-scratchpad harness that proved U7's twelve falsifiers into a repo tool, on the operator's call. No version bump: test tooling and docs, no production-code change, per the SemVer SKIP list. A green test is not evidence. A test that has never seen its own defeating change may pass under it too, forbidding nothing while reading as though it forbids something. This repo shipped that three times — twice in one session, and once an hour after writing the persistent-memory entry about it. Prose in a memory file is not an instrument. Tables live in tests/mutations/*.toml, one per unit, committed so a unit's proofs are an artifact rather than terminal scrollback. Adding a unit means adding a file, never editing the script. u7_navigation.toml was generated from the harness that proved those twelve, not retyped, and every anchor was verified against the source before it landed. THE TOOL GETS ITS OWN POSITIVE AND NEGATIVE CONTROLS, which is the point. It shipped two defects in one session, each of which made it report a falsifier PROVED WITHOUT RUNNING IT, and both were found by accident rather than by anything checking: no green baseline — a test that is ALREADY red reports red for every mutation thrown at it, so a broken assertion reads as a certified falsifier the bytecode cache — `< 2` -> `< 1` is byte-identical in size, and CPython validates a .pyc against the source's (mtime, size) at one-second granularity, so a mutation landing in the same second as the revert before it runs against cached bytecode; the tell was a verdict flipping between consecutive identical runs tests/test_mutation_check.py now carries a control for each, plus the one usually skipped: a KNOWN-VACUOUS falsifier the tool must catch. An instrument that only ever sees unknowns cannot tell "nothing wrong here" from "I am blind", and twelve `proved` lines from a blind instrument are worth nothing. Also hardens the tool against itself: it writes to tracked source files, so the restore is verified rather than assumed, and a .mutation-inflight marker makes a run killed mid-mutation refuse the next start instead of silently measuring a mutated tree. 648 tests green; 12/12 U7 falsifiers still proved.
This commit is contained in:
@@ -215,6 +215,23 @@ curl -s localhost:8090/healthz # the live service (systemd --user)
|
||||
systemctl --user restart booth.service # after a code change, to see it live
|
||||
```
|
||||
|
||||
```sh
|
||||
.venv/bin/python scripts/mutation_check.py # prove the falsifiers still falsify
|
||||
```
|
||||
|
||||
**A green test is not evidence.** A test that has never seen its own defeating
|
||||
change may pass under it too — forbidding nothing while reading as though it
|
||||
forbids something. This repo shipped that three times before the tool existed
|
||||
(twice in one session, once an hour after writing the entry about it). Tables
|
||||
live in `tests/mutations/*.toml`, one per unit, committed so a unit's proofs are
|
||||
an artifact rather than scrollback; adding a unit means adding a file, never
|
||||
editing the script. `tests/test_mutation_check.py` holds the tool's own positive
|
||||
and negative controls, because an instrument that only ever sees unknowns cannot
|
||||
tell "nothing wrong" from "I am blind".
|
||||
|
||||
When you add a `*Falsifiable:*` line to a contract, add its row to the table and
|
||||
run it. A falsifier nobody has run is a claim, not a test.
|
||||
|
||||
`tests/test_embed_browser.py` drives a real Chromium against a real uvicorn on
|
||||
an ephemeral port — the only place U3's placement and `form=` binding can be
|
||||
observed at all. Browsers are NOT downloaded per project; they live box-wide in
|
||||
|
||||
Reference in New Issue
Block a user