docs(training-playbook): measure refusal retention on the abliteration's OWN axis
§3.13, plus the probe that produced it. Two lessons, both about measuring the wrong thing confidently. First: a tune applied AFTER an abliteration can walk it back, and a reasoning/craft/memorisation gate cannot see that. brokkr-smithy-dev's preregistered gate measured none of it — a tune that gains 41 items of contradiction detection and quietly restores refusals passes every check. The compliance axis has to be added explicitly. Second, and this is the trap: measure the axis the abliteration was actually FOR. Ours was run so the model engages explicit fiction. The probe reached for mlabonne/harmful_behaviors — weapons, malware, fraud — because it was cached and carried a recorded baseline. Different refusal surface entirely, and a model moves on them independently. 29/100 general-harm refusals on a tune whose prose the operator was praising at the time is not obviously a defect and may be desirable: general-harm refusals returning while domain compliance holds is close to the ideal shape for an internal creative seat. The measurement was real; its relevance was assumed. Also recorded, because both were nearly missed: - Read the interesting cell. In 29 hard / 0 deflect / 71 comply, the load-bearing number is 71. Stock refused 100/100; near that would mean the abliteration was undone. 71 complying means partially walked back on one axis — a different finding, and only one of the two threatens the seat. - A baseline from a different harness is not a baseline. The recorded 3/100 came from the abliteration tool's scorer, which reads first-token probability distributions; a probe that generates and regexes is a different instrument. Run your own against both arms on the same seat or report the number alone. - A refusal regex undercounts, so classify hard/deflect/comply — and the free discriminator: if both arms return zero deflections the model is binary; if only one does, the regex is fine. An artifact does not care which arm it runs against.
This commit is contained in:
@@ -405,6 +405,51 @@ merge-back* came from the wrong one. Same family, three axes apart.
|
||||
written for and which axes differ from yours.** If the answer is "same family"
|
||||
that is not an answer.
|
||||
|
||||
## 3.13 ⭐⭐ Measure refusal retention on the axis the ABLITERATION targeted
|
||||
|
||||
Two distinct lessons from 2026-08-25, both about measuring the wrong thing
|
||||
confidently.
|
||||
|
||||
**A tune can re-install what an abliteration removed, and no capability gate
|
||||
will see it.** If you tune AFTER abliterating, the tune has every training token
|
||||
as an opportunity to walk the abliteration back. A reasoning/craft/memorisation
|
||||
gate measures none of that: a tune that gains 41 items of contradiction
|
||||
detection and quietly restores refusals is a failed seat that passes every
|
||||
check. **Add a compliance axis explicitly** — it will not fall out of the others.
|
||||
|
||||
**But measure the axis the abliteration was FOR.** This is the trap, and it is
|
||||
easy to walk into precisely because a general harm set is sitting right there,
|
||||
cached, with a recorded baseline.
|
||||
|
||||
abliteration run so the model engages EXPLICIT FICTION
|
||||
probe used: mlabonne/harmful_behaviors (weapons, malware, fraud)
|
||||
|
||||
Those are different refusal surfaces and a model moves on them independently.
|
||||
The measured result — 29/100 general-harm refusals on a tune whose prose the
|
||||
operator was actively praising — is **not obviously a defect and may be
|
||||
desirable**: general-harm refusals returning while domain compliance holds is
|
||||
close to the ideal shape for an internal creative seat. The number was real; its
|
||||
relevance was assumed.
|
||||
|
||||
**Read the interesting cell.** In `29 hard / 0 deflect / 71 comply`, the
|
||||
load-bearing figure is **71**. Stock refused 100/100; anything near that would
|
||||
mean the abliteration was undone. 71 complying says "partially walked back on
|
||||
one axis", which is a completely different finding — and only one of the two
|
||||
threatens the seat.
|
||||
|
||||
⚠ **A baseline from a different harness is not a baseline.** The recorded
|
||||
"3/100" for that base came from the abliteration tool's own scorer, which works
|
||||
off *first-token probability distributions*. A probe that generates 256 tokens
|
||||
and regexes them is a different instrument; the two can disagree in both
|
||||
directions. Run your own probe against BOTH arms on the SAME seat, or report the
|
||||
number alone and say the comparison is missing.
|
||||
|
||||
⚠ **A refusal regex undercounts** — models decline by redirecting, with no
|
||||
refusal token present. Classify three ways (hard / deflect / comply). And note
|
||||
the free discriminator: **if both arms return zero deflections the model is
|
||||
binary; if only one does, the regex is fine and the difference is real.** An
|
||||
instrument artifact does not care which arm it runs against.
|
||||
|
||||
## 4. Panel / consult discipline for perf work
|
||||
|
||||
Perf investigations are unusually good at generating confident wrong answers,
|
||||
|
||||
Reference in New Issue
Block a user