feat(coldfusion-abliteration): first-token KL measured — 28.4x selectivity, harmless median 0.0211
Adds `kl_divergence.py`: first-token KL(stock || abliterated) over the full 248,320-token vocabulary, bf16 vs bf16, scored separately for held-out harmless and reserved-harmful prompts. Result (L35, 256 harmless / 104 harmful, answer mode): harmless median 0.0211 mean 0.0364 top-1 agreement 89.8% harmful median 0.5996 mean 0.6992 top-1 agreement 55.8% selectivity 28.4x (72.8x in think mode) Self-KL noise floor is exactly 0.0, and all 720 per-prompt values are bit-identical between a single-process and a two-process run, so the figures are signal rather than bf16 jitter. Reverse KL on harmful/answer is 1.43 vs forward 0.70 — the mass-where-stock-had-none asymmetry expected of a refusal-direction removal. Against the Heretic reference figures (0.1191 prior seat, 0.0759 the live absolute-heresy seat) this is materially gentler, but those are the other tool's optimizer output on a different base with its own harmless set and template — order-of-magnitude, not head-to-head. KL remains a fidelity number; the viability gate is still MTP acceptance (59.1%). Method notes: - Prompt classes are reported separately by design. A single averaged KL over a mixed corpus is close to meaningless, since the metric is meant to be large on harmful prompts and small on benign ones; the ratio carries the information. - The harmless evaluation set is drawn from the alpaca pool minus calibration's own draw, reconstructed by replaying that draw rather than remembered, and asserted disjoint on text. The harmful set is the reserved test split. - `render` is imported from abliterate.py rather than copied, so the measurement cannot drift from the rendering the direction was captured against. - Batch size 1 with logits_to_keep=1: no padding semantics, ~0.6 MB of logits. Three corrections to the runbook, each of which cost time: - "bf16 is 50 GB, only gen must go" was 50.10 GiB mislabelled. Text-only weights are 51,300 MiB; freeing either GPU0 seat alone leaves ~50,933 MiB. Both must stop. VRAM is now sized from the safetensors headers at run time. - A 27B model cannot be released in-process: `del` + gc + empty_cache left free VRAM at 45,287 MiB, and so did confining the model to an inner frame that exits. Only process exit returned the card (96,689 MiB). The first run completed only because the allocator hit OOM, collected, and retried. Each model now gets its own process, handing log-probs to disk between stages. - The residency gate read hf_device_map, which transformers leaves empty when the model fits on one device — it reported "(unsharded)" whether or not anything was wrong, so it could never fail. It now reads parameter devices directly. Model-agnostic lessons promoted to the quant playbook (new 3.12).
This commit is contained in:
@@ -154,6 +154,14 @@ The recipe is a drop-in for the front half of the House quant pipeline:
|
||||
3994; gate the result on **MTP acceptance ≳40%, not KL** (KL misled us once —
|
||||
`reference_abliteration_mtp_lessons`).
|
||||
4. Verify vision byte-identical, refusals down, PPL not blown, no catatonia.
|
||||
**Measure first-token KL as a *fidelity* number** (`kl_divergence.py`,
|
||||
bf16-vs-bf16, held-out prompts) — it does not replace the acceptance gate in
|
||||
step 3, and it is not a pass/fail on its own. Report it **split by prompt
|
||||
class**: a single averaged KL over a mixed corpus is close to meaningless,
|
||||
because the metric is supposed to be large on harmful prompts and small on
|
||||
benign ones. The ratio is the interesting quantity. Cold-Fusion L35 measured
|
||||
**0.0211 median harmless / 0.5996 median harmful = 28.4× selectivity**, on a
|
||||
stack whose self-KL noise floor is exactly 0.0.
|
||||
5. NVFP4-quantize in-house (mixed W4A4 + FP8-attn/lm_head —
|
||||
`model-quantization-playbook.md`). **Foot-gun the GGUF card itself flags:
|
||||
the imatrix does not cover the MTP block** — so a GGUF requant path leaves
|
||||
|
||||
@@ -310,6 +310,45 @@ determinism check with a **magnitude** check (residual norms should grow smoothl
|
||||
exact 0.0 mid-stack is impossible) and, where you can, a **coherence** check (generate 40 tokens and
|
||||
read them).
|
||||
|
||||
### 3.12 ⭐⭐ You cannot free a 27B model in-process — give each model its own process
|
||||
|
||||
Any A/B that loads two large checkpoints in sequence (KL, logit diffing, teacher-vs-student)
|
||||
will try to release the first before loading the second. **On this stack, it does not work.**
|
||||
Measured 2026-08-20 on Qwen3.8-27B bf16, free VRAM after each attempt:
|
||||
|
||||
| teardown | free VRAM |
|
||||
|---|---|
|
||||
| `del model` + `gc.collect()` + `torch.cuda.empty_cache()` | 45,287 MiB |
|
||||
| same, with the model confined to an inner frame that exits | 45,287 MiB |
|
||||
| **the process exits** | **97,247 MiB** |
|
||||
|
||||
The ~51,300 MiB of weights stayed resident through both in-process teardowns. The first
|
||||
run survived only because **PyTorch's allocator hit OOM on the second load, ran a collection
|
||||
itself, and retried** — the second model landed by rescue, not by design. That is not a
|
||||
release strategy: on an architecture where a silent CPU offload does not raise (§3.9), the
|
||||
day the retry does not fire you get confident garbage instead of an error.
|
||||
|
||||
**Do this instead:** one process per model, hand results to disk between them
|
||||
(first-token log-probs for a 250k vocab are ~715 MiB per model — nothing), and gate each
|
||||
stage on free VRAM *before* the load. Reference implementation:
|
||||
`services/coldfusion-abliteration/kl_divergence.py` (`--stage ref|cand|score`).
|
||||
|
||||
Two gate corollaries learned in the same session:
|
||||
|
||||
- **⭐ A residency gate that reads `hf_device_map` cannot fail.** The map is **empty**
|
||||
whenever transformers puts the whole model on one device, so the check reports
|
||||
"unsharded" both when everything is fine and when there is nothing to inspect. Read
|
||||
`{p.device for p in model.parameters()}` — ground truth in every case. (Generalises
|
||||
[[feedback_assert_effective_value_not_substring]]: presence of a passing check is not
|
||||
evidence of a check that can fail.)
|
||||
- **⭐ Size VRAM from the checkpoint's own headers, never from a remembered figure.** A
|
||||
runbook carried "bf16 is 50 GB"; the real number was 50.10 **GiB** = 51,300 MiB of
|
||||
text-only weights. That 3.7 GB unit error is exactly the difference between "stop one
|
||||
co-tenant" and "stop both", and it cost an aborted window. Sum the safetensors header
|
||||
offsets (excluding tensors the loader class won't instantiate — vision, MTP); read only
|
||||
the 8-byte length prefix + JSON header, never `safe_open`, which mmaps the whole shard
|
||||
and ENOMEMs on ZFS (§ *Avoid mmap on `/tank`*).
|
||||
|
||||
---
|
||||
|
||||
## 4. Pipeline shape
|
||||
|
||||
Reference in New Issue
Block a user