feat(coldfusion-abliteration): first-token KL measured — 28.4x selectivity, harmless median 0.0211
Adds `kl_divergence.py`: first-token KL(stock || abliterated) over the full 248,320-token vocabulary, bf16 vs bf16, scored separately for held-out harmless and reserved-harmful prompts. Result (L35, 256 harmless / 104 harmful, answer mode): harmless median 0.0211 mean 0.0364 top-1 agreement 89.8% harmful median 0.5996 mean 0.6992 top-1 agreement 55.8% selectivity 28.4x (72.8x in think mode) Self-KL noise floor is exactly 0.0, and all 720 per-prompt values are bit-identical between a single-process and a two-process run, so the figures are signal rather than bf16 jitter. Reverse KL on harmful/answer is 1.43 vs forward 0.70 — the mass-where-stock-had-none asymmetry expected of a refusal-direction removal. Against the Heretic reference figures (0.1191 prior seat, 0.0759 the live absolute-heresy seat) this is materially gentler, but those are the other tool's optimizer output on a different base with its own harmless set and template — order-of-magnitude, not head-to-head. KL remains a fidelity number; the viability gate is still MTP acceptance (59.1%). Method notes: - Prompt classes are reported separately by design. A single averaged KL over a mixed corpus is close to meaningless, since the metric is meant to be large on harmful prompts and small on benign ones; the ratio carries the information. - The harmless evaluation set is drawn from the alpaca pool minus calibration's own draw, reconstructed by replaying that draw rather than remembered, and asserted disjoint on text. The harmful set is the reserved test split. - `render` is imported from abliterate.py rather than copied, so the measurement cannot drift from the rendering the direction was captured against. - Batch size 1 with logits_to_keep=1: no padding semantics, ~0.6 MB of logits. Three corrections to the runbook, each of which cost time: - "bf16 is 50 GB, only gen must go" was 50.10 GiB mislabelled. Text-only weights are 51,300 MiB; freeing either GPU0 seat alone leaves ~50,933 MiB. Both must stop. VRAM is now sized from the safetensors headers at run time. - A 27B model cannot be released in-process: `del` + gc + empty_cache left free VRAM at 45,287 MiB, and so did confining the model to an inner frame that exits. Only process exit returned the card (96,689 MiB). The first run completed only because the allocator hit OOM, collected, and retried. Each model now gets its own process, handing log-probs to disk between stages. - The residency gate read hf_device_map, which transformers leaves empty when the model fits on one device — it reported "(unsharded)" whether or not anything was wrong, so it could never fail. It now reads parameter devices directly. Model-agnostic lessons promoted to the quant playbook (new 3.12).
This commit is contained in:
@@ -66,6 +66,63 @@ vision byte-identical (Δ0.0), 735/735 others untouched.**
|
||||
- Hidden states captured via **forward pre-hook**, not `output_hidden_states` off
|
||||
the returned object (buffers get recycled → Inf that moves run-to-run).
|
||||
|
||||
## ✅ KL divergence measured (2026-08-20, third session)
|
||||
|
||||
`services/coldfusion-abliteration/kl_divergence.py` — first-token KL(stock ‖ L35)
|
||||
over the full 248,320-token vocabulary, bf16 vs bf16, on prompts the direction was
|
||||
never fitted on (256 harmless held out of the alpaca pool by replaying and
|
||||
subtracting calibration's own draw; 104 harmful from the reserved test split).
|
||||
|
||||
| mode | class | median | mean | p95 | top-1 agreement |
|
||||
|---|---|---|---|---|---|
|
||||
| answer | harmless | **0.0211** | 0.0364 | 0.1219 | 89.8% |
|
||||
| answer | harmful | **0.5996** | 0.6992 | 1.6937 | 55.8% |
|
||||
| think | harmless | 0.0042 | 0.0066 | 0.0205 | 94.5% |
|
||||
| think | harmful | 0.3068 | 0.3186 | 0.4689 | 57.7% |
|
||||
|
||||
Run twice — single-process, then through the two-process design — and **all 720
|
||||
per-prompt KL values came back bit-identical**, so these figures are stable across
|
||||
processes, not just within one.
|
||||
|
||||
**Selectivity 28.4× (answer) / 72.8× (think).** The surgery moves the model hard on
|
||||
refusal-triggering prompts and barely at all on benign ones — on held-out harmless
|
||||
prompts the abliterated model still picks the same first token 89.8% of the time.
|
||||
**Self-KL noise floor: exactly 0.0**, so none of this is bf16 jitter, and the
|
||||
scoring path is validated end to end. Reverse KL on harmful/answer is 1.43 vs
|
||||
forward 0.70 — the mass-where-stock-had-none asymmetry that is abliteration's
|
||||
signature.
|
||||
|
||||
Against the Heretic reference figures (0.1191 prior seat, **0.0759 the current
|
||||
`absolute-heresy` seat**) ours is materially gentler — but ⚠️ **that is not a
|
||||
head-to-head**: those are Heretic's own optimizer output on a different base with
|
||||
its own harmless set and template. Order-of-magnitude only. A real comparison
|
||||
means re-measuring the incumbent through this script (one more GPU window).
|
||||
|
||||
Consistent with [[reference_abliteration_mtp_lessons]]: KL is a **fidelity**
|
||||
number here, not the viability gate — that remains MTP acceptance (59.1%).
|
||||
|
||||
### Three durable process lessons from the measurement
|
||||
|
||||
1. **★ Report abliteration KL SPLIT BY PROMPT CLASS.** A single averaged KL over a
|
||||
mixed corpus is close to meaningless, because the metric is *supposed* to be
|
||||
large on harmful prompts and small on benign ones — averaging them together
|
||||
lets a blunt abliteration and a surgical one produce the same number. The
|
||||
selectivity ratio is the quantity with information in it.
|
||||
2. **★ "50 GB" was 50.10 GiB mislabelled — and the 3.7 GB gap changed the runbook.**
|
||||
Text-only weights are **51,300 MiB**; GPU0's tenants are meromero 50,072 and gen
|
||||
46,304, so freeing *either alone* leaves ~50,933 MiB — ~400 MiB short. The
|
||||
runbook's "only gen must go" was wrong. **Both seats must stop.** Size VRAM from
|
||||
the safetensors headers, never from a remembered gigabyte figure.
|
||||
3. **★ You cannot release a 27B model in-process; give each model its own process.**
|
||||
Measured twice: `del model` + `gc.collect()` + `empty_cache()` left free VRAM at
|
||||
45,287 MiB, and so did confining the model to an inner frame that exits. The
|
||||
first run only worked because PyTorch's allocator hit OOM on the second load,
|
||||
collected, and retried — *rescue, not design*. On this architecture a silent
|
||||
CPU offload does not error; it zeroes the residual stream past the boundary and
|
||||
returns confident garbage. Also: the old residency gate read `hf_device_map`,
|
||||
which is **empty when the model fits on one device** — so it printed
|
||||
"(unsharded)" and could never fail. It now reads parameter devices directly.
|
||||
|
||||
## Still owed before this is a gen-seat candidate
|
||||
|
||||
- Canonical refusal re-profile via `services/refusal-probe/` (not the ad-hoc
|
||||
|
||||
Reference in New Issue
Block a user