vh
2511aab3d6
Credited to design-dev. His R2 order check has a positive control — one tile given order:-1 that the check must catch — and the control went blind when group headers became grid children: nth-child(5) started landing on a header rather than the fifth tile. Same class as the two defects already in this file. A control that no longer controls reads exactly like a passing test; nothing in the output distinguishes 'detected nothing because there was nothing' from 'detected nothing because I am aimed at the wrong element'. The rule: nth-of-type over nth-child wherever the assertion means the Nth TILE rather than the Nth child element. They agree until somebody adds a sibling of a different kind, and adding siblings is what a redesign is.
70 lines
3.6 KiB
Markdown
70 lines
3.6 KiB
Markdown
# A mutation harness that certified a broken test, twice, for two reasons
|
|
|
|
_2026-09-22 · booth_
|
|
|
|
This repo already knows that **an assertion which has never seen its own
|
|
defeating change is not known to falsify anything** — two prior entries say so
|
|
([[2026-09-22-vacuous-falsifiers]], [[2026-09-22-seven-of-seven-falsifiers]]).
|
|
So U7's groups were built with a harness that applies each defeating change and
|
|
asserts the named test goes red. **The harness itself had two defects, and both
|
|
produce the same lie: a falsifier certified without being run.**
|
|
|
|
## Defect 1 — no green baseline
|
|
|
|
A test that is **already red** reports RED for every mutation thrown at it. The
|
|
escaping test had an arithmetic slip (counted `<` against `<a`/`<nav`/`</` and
|
|
forgot the two `<b>` elements), so it was failing for a reason unrelated to
|
|
escaping — and the harness cheerfully reported `RED ✓ the rail markup is emitted
|
|
with |safe`. **Run the test unmutated first; a non-zero baseline is a harness
|
|
failure, not a proven falsifier.**
|
|
|
|
## Defect 2 — the bytecode cache, which is the subtle one
|
|
|
|
`if len(sizes) < 2` → `if len(sizes) < 1` is **byte-identical in size**. CPython
|
|
validates a `.pyc` against the source's `(mtime, size)` at **one-second
|
|
granularity** — so a mutation that lands in the same second as the revert before
|
|
it is invisible, the cached bytecode is reused, and **the harness runs the
|
|
unmutated code and reports the falsifier proven.**
|
|
|
|
The tell was non-determinism with no cause: INV-3a certified RED on one run and
|
|
GREEN on the next with neither the test nor the code changing, and reproduced by
|
|
hand every time. Fix: delete `__pycache__` and set `PYTHONDONTWRITEBYTECODE=1`
|
|
in the subprocess environment before every run.
|
|
|
|
⚠ **This bites any same-size source mutation**, which is most interesting ones:
|
|
comparison flips, off-by-one constants, `and`↔`or`, `<`↔`>`. A mutation harness
|
|
without cache defeat is biased toward exactly the mutations most worth running.
|
|
|
|
## Result
|
|
|
|
12 falsifiers, 12 proved, stable across consecutive runs. Two of them only
|
|
after these fixes — and one of the twelve (`test_group_order_is_the_position_of
|
|
_the_first_member`) was genuinely vacuous on the first pass: its `w, x, y`
|
|
fixture's positional order **happened to be alphabetical**, so it stayed green
|
|
under the alphabetical-sort mutation it forbade. Rebuilt so all three plausible
|
|
rules (position, alphabetical, count) disagree.
|
|
|
|
**The harness lives in the session scratchpad and dies with the session.**
|
|
Whether it becomes `scripts/` is an open question for the operator — this repo
|
|
has now been bitten by vacuous falsifiers three times, and prose in a memory
|
|
file is not an instrument.
|
|
|
|
## A third way an instrument goes blind: `nth-child` vs `nth-of-type`
|
|
|
|
_Added 2026-09-23, credited to design-dev, who hit it in his R2 order check._
|
|
|
|
His layout check has a positive control — one tile given `order:-1` that the
|
|
check must catch. **The control went blind when group headers became grid
|
|
children**: `nth-child(5)` started landing on a header instead of the fifth
|
|
tile, so the control stopped controlling and the check kept reporting clean.
|
|
|
|
Same class as this file's other two, and the reason it belongs here: **a control
|
|
that no longer controls reads exactly like a passing test.** Nothing in the
|
|
output distinguishes "detected nothing because there was nothing" from
|
|
"detected nothing because I am aimed at the wrong element".
|
|
|
|
**The rule worth having written down:** use `nth-of-type` over `nth-child` for
|
|
any assertion that means *the Nth TILE* rather than *the Nth child element*.
|
|
The two agree right up until somebody adds a sibling of a different kind — and
|
|
adding a sibling is what a redesign is.
|