vh
2511aab3d6
Credited to design-dev. His R2 order check has a positive control — one tile given order:-1 that the check must catch — and the control went blind when group headers became grid children: nth-child(5) started landing on a header rather than the fifth tile. Same class as the two defects already in this file. A control that no longer controls reads exactly like a passing test; nothing in the output distinguishes 'detected nothing because there was nothing' from 'detected nothing because I am aimed at the wrong element'. The rule: nth-of-type over nth-child wherever the assertion means the Nth TILE rather than the Nth child element. They agree until somebody adds a sibling of a different kind, and adding siblings is what a redesign is.
3.6 KiB
A mutation harness that certified a broken test, twice, for two reasons
2026-09-22 · booth
This repo already knows that an assertion which has never seen its own defeating change is not known to falsify anything — two prior entries say so (2026-09-22-vacuous-falsifiers, 2026-09-22-seven-of-seven-falsifiers). So U7's groups were built with a harness that applies each defeating change and asserts the named test goes red. The harness itself had two defects, and both produce the same lie: a falsifier certified without being run.
Defect 1 — no green baseline
A test that is already red reports RED for every mutation thrown at it. The
escaping test had an arithmetic slip (counted < against <a/<nav/</ and
forgot the two <b> elements), so it was failing for a reason unrelated to
escaping — and the harness cheerfully reported RED ✓ the rail markup is emitted with |safe. Run the test unmutated first; a non-zero baseline is a harness
failure, not a proven falsifier.
Defect 2 — the bytecode cache, which is the subtle one
if len(sizes) < 2 → if len(sizes) < 1 is byte-identical in size. CPython
validates a .pyc against the source's (mtime, size) at one-second
granularity — so a mutation that lands in the same second as the revert before
it is invisible, the cached bytecode is reused, and the harness runs the
unmutated code and reports the falsifier proven.
The tell was non-determinism with no cause: INV-3a certified RED on one run and
GREEN on the next with neither the test nor the code changing, and reproduced by
hand every time. Fix: delete __pycache__ and set PYTHONDONTWRITEBYTECODE=1
in the subprocess environment before every run.
⚠ This bites any same-size source mutation, which is most interesting ones:
comparison flips, off-by-one constants, and↔or, <↔>. A mutation harness
without cache defeat is biased toward exactly the mutations most worth running.
Result
12 falsifiers, 12 proved, stable across consecutive runs. Two of them only
after these fixes — and one of the twelve (test_group_order_is_the_position_of _the_first_member) was genuinely vacuous on the first pass: its w, x, y
fixture's positional order happened to be alphabetical, so it stayed green
under the alphabetical-sort mutation it forbade. Rebuilt so all three plausible
rules (position, alphabetical, count) disagree.
The harness lives in the session scratchpad and dies with the session.
Whether it becomes scripts/ is an open question for the operator — this repo
has now been bitten by vacuous falsifiers three times, and prose in a memory
file is not an instrument.
A third way an instrument goes blind: nth-child vs nth-of-type
Added 2026-09-23, credited to design-dev, who hit it in his R2 order check.
His layout check has a positive control — one tile given order:-1 that the
check must catch. The control went blind when group headers became grid
children: nth-child(5) started landing on a header instead of the fifth
tile, so the control stopped controlling and the check kept reporting clean.
Same class as this file's other two, and the reason it belongs here: a control that no longer controls reads exactly like a passing test. Nothing in the output distinguishes "detected nothing because there was nothing" from "detected nothing because I am aimed at the wrong element".
The rule worth having written down: use nth-of-type over nth-child for
any assertion that means the Nth TILE rather than the Nth child element.
The two agree right up until somebody adds a sibling of a different kind — and
adding a sibling is what a redesign is.