# A mutation harness that certified a broken test, twice, for two reasons

_2026-09-22 · booth_

This repo already knows that **an assertion which has never seen its own
defeating change is not known to falsify anything** — two prior entries say so
([[2026-09-22-vacuous-falsifiers]], [[2026-09-22-seven-of-seven-falsifiers]]).
So U7's groups were built with a harness that applies each defeating change and
asserts the named test goes red. **The harness itself had two defects, and both
produce the same lie: a falsifier certified without being run.**

## Defect 1 — no green baseline

A test that is **already red** reports RED for every mutation thrown at it. The
escaping test had an arithmetic slip (counted `<` against `<a`/`<nav`/`</` and
forgot the two `<b>` elements), so it was failing for a reason unrelated to
escaping — and the harness cheerfully reported `RED ✓ the rail markup is emitted
with |safe`. **Run the test unmutated first; a non-zero baseline is a harness
failure, not a proven falsifier.**

## Defect 2 — the bytecode cache, which is the subtle one

`if len(sizes) < 2` → `if len(sizes) < 1` is **byte-identical in size**. CPython
validates a `.pyc` against the source's `(mtime, size)` at **one-second
granularity** — so a mutation that lands in the same second as the revert before
it is invisible, the cached bytecode is reused, and **the harness runs the
unmutated code and reports the falsifier proven.**

The tell was non-determinism with no cause: INV-3a certified RED on one run and
GREEN on the next with neither the test nor the code changing, and reproduced by
hand every time. Fix: delete `__pycache__` and set `PYTHONDONTWRITEBYTECODE=1`
in the subprocess environment before every run.

⚠ **This bites any same-size source mutation**, which is most interesting ones:
comparison flips, off-by-one constants, `and`↔`or`, `<`↔`>`. A mutation harness
without cache defeat is biased toward exactly the mutations most worth running.

## Result

12 falsifiers, 12 proved, stable across consecutive runs. Two of them only
after these fixes — and one of the twelve (`test_group_order_is_the_position_of
_the_first_member`) was genuinely vacuous on the first pass: its `w, x, y`
fixture's positional order **happened to be alphabetical**, so it stayed green
under the alphabetical-sort mutation it forbade. Rebuilt so all three plausible
rules (position, alphabetical, count) disagree.

**The harness lives in the session scratchpad and dies with the session.**
Whether it becomes `scripts/` is an open question for the operator — this repo
has now been bitten by vacuous falsifiers three times, and prose in a memory
file is not an instrument.

## A third way an instrument goes blind: `nth-child` vs `nth-of-type`

_Added 2026-09-23, credited to design-dev, who hit it in his R2 order check._

His layout check has a positive control — one tile given `order:-1` that the
check must catch. **The control went blind when group headers became grid
children**: `nth-child(5)` started landing on a header instead of the fifth
tile, so the control stopped controlling and the check kept reporting clean.

Same class as this file's other two, and the reason it belongs here: **a control
that no longer controls reads exactly like a passing test.** Nothing in the
output distinguishes "detected nothing because there was nothing" from
"detected nothing because I am aimed at the wrong element".

**The rule worth having written down:** use `nth-of-type` over `nth-child` for
any assertion that means *the Nth TILE* rather than *the Nth child element*.
The two agree right up until somebody adds a sibling of a different kind — and
adding a sibling is what a redesign is.