docs(training-playbook): measure refusal retention on the abliteration's OWN axis

§3.13, plus the probe that produced it. Two lessons, both about measuring the
wrong thing confidently.

First: a tune applied AFTER an abliteration can walk it back, and a
reasoning/craft/memorisation gate cannot see that. brokkr-smithy-dev's
preregistered gate measured none of it — a tune that gains 41 items of
contradiction detection and quietly restores refusals passes every check. The
compliance axis has to be added explicitly.

Second, and this is the trap: measure the axis the abliteration was actually
FOR. Ours was run so the model engages explicit fiction. The probe reached for
mlabonne/harmful_behaviors — weapons, malware, fraud — because it was cached and
carried a recorded baseline. Different refusal surface entirely, and a model
moves on them independently. 29/100 general-harm refusals on a tune whose prose
the operator was praising at the time is not obviously a defect and may be
desirable: general-harm refusals returning while domain compliance holds is
close to the ideal shape for an internal creative seat. The measurement was
real; its relevance was assumed.

Also recorded, because both were nearly missed:

- Read the interesting cell. In 29 hard / 0 deflect / 71 comply, the
  load-bearing number is 71. Stock refused 100/100; near that would mean the
  abliteration was undone. 71 complying means partially walked back on one
  axis — a different finding, and only one of the two threatens the seat.
- A baseline from a different harness is not a baseline. The recorded 3/100
  came from the abliteration tool's scorer, which reads first-token probability
  distributions; a probe that generates and regexes is a different instrument.
  Run your own against both arms on the same seat or report the number alone.
- A refusal regex undercounts, so classify hard/deflect/comply — and the free
  discriminator: if both arms return zero deflections the model is binary; if
  only one does, the regex is fine. An artifact does not care which arm it runs
  against.
This commit is contained in:
vh
2026-08-25 12:58:27 -07:00
parent a696b49e2a
commit 64bf9d313f
2 changed files with 179 additions and 0 deletions
+45
View File
@@ -405,6 +405,51 @@ merge-back* came from the wrong one. Same family, three axes apart.
written for and which axes differ from yours.** If the answer is "same family"
that is not an answer.
## 3.13 ⭐⭐ Measure refusal retention on the axis the ABLITERATION targeted
Two distinct lessons from 2026-08-25, both about measuring the wrong thing
confidently.
**A tune can re-install what an abliteration removed, and no capability gate
will see it.** If you tune AFTER abliterating, the tune has every training token
as an opportunity to walk the abliteration back. A reasoning/craft/memorisation
gate measures none of that: a tune that gains 41 items of contradiction
detection and quietly restores refusals is a failed seat that passes every
check. **Add a compliance axis explicitly** — it will not fall out of the others.
**But measure the axis the abliteration was FOR.** This is the trap, and it is
easy to walk into precisely because a general harm set is sitting right there,
cached, with a recorded baseline.
abliteration run so the model engages EXPLICIT FICTION
probe used: mlabonne/harmful_behaviors (weapons, malware, fraud)
Those are different refusal surfaces and a model moves on them independently.
The measured result — 29/100 general-harm refusals on a tune whose prose the
operator was actively praising — is **not obviously a defect and may be
desirable**: general-harm refusals returning while domain compliance holds is
close to the ideal shape for an internal creative seat. The number was real; its
relevance was assumed.
**Read the interesting cell.** In `29 hard / 0 deflect / 71 comply`, the
load-bearing figure is **71**. Stock refused 100/100; anything near that would
mean the abliteration was undone. 71 complying says "partially walked back on
one axis", which is a completely different finding — and only one of the two
threatens the seat.
⚠ **A baseline from a different harness is not a baseline.** The recorded
"3/100" for that base came from the abliteration tool's own scorer, which works
off *first-token probability distributions*. A probe that generates 256 tokens
and regexes them is a different instrument; the two can disagree in both
directions. Run your own probe against BOTH arms on the SAME seat, or report the
number alone and say the comparison is missing.
⚠ **A refusal regex undercounts** — models decline by redirecting, with no
refusal token present. Classify three ways (hard / deflect / comply). And note
the free discriminator: **if both arms return zero deflections the model is
binary; if only one does, the regex is fine and the difference is real.** An
instrument artifact does not care which arm it runs against.
## 4. Panel / consult discipline for perf work
Perf investigations are unusually good at generating confident wrong answers,