Files
booth/persistent-memory.d/2026-09-22-a-mutation-harness-that-certified-a-broken-test.md
T
vh bf351a26d1 feat(u7): filename groups — the last v1 unit, and a table that did not reproduce
Completes U7 with its fourth component: a jump-to-group rail derived from
filename prefixes, replacing the subfolder sections ROADMAP named. The scope
departure was ratified by the operator 2026-09-22; this commit deletes
test_no_group_rail_is_shipped_yet, the guard that held it back, in the same
change that builds what it guarded against.

All seven v1 capabilities are now landed. The 1.0 cut is a decision, not a
dependency, and it is the operator's — no version bump here, because a commit
is not a release.

THE RULE CHANGED AT IMPLEMENTATION, ON MEASURED GROUNDS. The contract specified
`strip ONE trailing run of digits`; run against the live set that yields 24
groups for sindra-bakeoff's 40 images and 27 for sindra's 30 — a rail with a row
per tile — because it keys on the END of the stem, where the instance number
lives. The contract's own table claimed 5 and 1 for those two booths and neither
reproduces; the numbers are reachable only by two OTHER heuristics, so the table
that justified the design was assembled from more than one rule. Its own worked
example contradicts it in plain sight.

The shipped rule keys on the first separator-delimited segment, where the family
lives, destemming only when the stem has no separator at all — so `ac01` -> `ac`
while `v30-seed8302` and `v35-seed8302` stay apart. Re-measured across all 17
live booths; the table is in the contract.

INV-3 GAINED ITS SECOND DEGENERACY. The contract guarded one group for
everything (sc-iso-spread: DSC0001-DSC0006). The live set's actual failure is
the opposite — pewpew-ui-brief yields 23 groups for 34 items, dfa-concepts 13
for 20 — and the contract as written would have shipped a rail that is a second
copy of the grid. The rail now renders only when grouping is informative: two or
more groups, and the middle group holding more than one item. That predicate
gets all 17 booths right.

Grouping is a VIEW. The grid stays sorted(rel) and the zoom ring stays that
order filtered to images; the group fixture interleaves across subdirectories
precisely so a (group, rel) re-sort goes red. Groups are derived from the
RENDERED list, not the full gallery, so no anchor points at a filtered-out tile.

booth/items.py       _group_of + Item.group, derived in the resolver (INV-1)
booth/app.py         _groups() builds the rail rows; build_gallery carries it
booth/templates/     the rail-groups nav and its CSS
tests/               +16 tests; 639 green

Every new falsifier was proved by running its defeating change (12/12). Three
were vacuous first time out: one fixture's positional order happened to be
alphabetical, one assertion miscounted elements, and the harness itself
certified a broken test twice — no green baseline, and byte-identical mutations
silently defeated by the pyc cache's one-second mtime granularity.
2026-09-22 21:33:54 -07:00

2.6 KiB

A mutation harness that certified a broken test, twice, for two reasons

2026-09-22 · booth

This repo already knows that an assertion which has never seen its own defeating change is not known to falsify anything — two prior entries say so (2026-09-22-vacuous-falsifiers, 2026-09-22-seven-of-seven-falsifiers). So U7's groups were built with a harness that applies each defeating change and asserts the named test goes red. The harness itself had two defects, and both produce the same lie: a falsifier certified without being run.

Defect 1 — no green baseline

A test that is already red reports RED for every mutation thrown at it. The escaping test had an arithmetic slip (counted < against <a/<nav/</ and forgot the two <b> elements), so it was failing for a reason unrelated to escaping — and the harness cheerfully reported RED ✓ the rail markup is emitted with |safe. Run the test unmutated first; a non-zero baseline is a harness failure, not a proven falsifier.

Defect 2 — the bytecode cache, which is the subtle one

if len(sizes) < 2 → if len(sizes) < 1 is byte-identical in size. CPython validates a .pyc against the source's (mtime, size) at one-second granularity — so a mutation that lands in the same second as the revert before it is invisible, the cached bytecode is reused, and the harness runs the unmutated code and reports the falsifier proven.

The tell was non-determinism with no cause: INV-3a certified RED on one run and GREEN on the next with neither the test nor the code changing, and reproduced by hand every time. Fix: delete __pycache__ and set PYTHONDONTWRITEBYTECODE=1 in the subprocess environment before every run.

⚠ This bites any same-size source mutation, which is most interesting ones: comparison flips, off-by-one constants, and↔or, <↔>. A mutation harness without cache defeat is biased toward exactly the mutations most worth running.

Result

12 falsifiers, 12 proved, stable across consecutive runs. Two of them only after these fixes — and one of the twelve (test_group_order_is_the_position_of _the_first_member) was genuinely vacuous on the first pass: its w, x, y fixture's positional order happened to be alphabetical, so it stayed green under the alphabetical-sort mutation it forbade. Rebuilt so all three plausible rules (position, alphabetical, count) disagree.

The harness lives in the session scratchpad and dies with the session. Whether it becomes scripts/ is an open question for the operator — this repo has now been bitten by vacuous falsifiers three times, and prose in a memory file is not an instrument.