esh-pfi-infrastructure/services/coldfusion-abliteration/README.md

# Cold-Fusion abliteration — Robinson formula

Abliterate `DavidAU/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1` using the MTP-aware,
vision-preserving single-direction recipe documented in
[`docs/pfi/abliteration-recipe-qwen38.md`](../../docs/pfi/abliteration-recipe-qwen38.md).

**Why this model, why this recipe.** Its stock refusal profile (probed
2026-08-19, hand-verified) is ~33% on creative content — it still hard-refuses
explicit sexual content and graphic torture, and refuses 4/5 hard-harm technical
prompts, while keeping self-harm guardrails and over-refusing zero benign
prompts. So there is a real creative-content refusal surface to remove. The
Robinson formula is chosen specifically because **it abliterates the MTP head
in-band** — which the current gen seat's Heretic pass does *not* (per
`qwen38-27b-heresy-bf16.PROVENANCE.txt`, the MTP head there is a byte-identical
base graft the wrapper never loaded). That is the additive delta this
experiment tests.

## Where it runs

**ana-ml2** (dual RTX PRO 6000 Blackwell, 96 GB each). A 55.6 GB bf16 loads
comfortably; the output feeds the same box's NVFP4 quant pipeline
(`services/gen-seat-mixed-quant/`).

- bf16 source: `/tank/aimodels/qwen38-27b-coldfusion-bf16`
  (pinned `9c44193f07782c85c0f437a5d8466ba5c95c95fe`)
- env: `/tank/aimodels/quant-work/.venv` (torch 2.12.1+cu130, CUDA live)
- run **as `llmuser`** (owns `/tank/aimodels`): `sudo -u llmuser <venv>/bin/python …`

## The gates — this script refuses to brick the model

Two hard gates from the recipe, both of which halt before any write:

1. **Coverage gate** — `o_proj(16) + linear_out(48) == 64 == num_hidden_layers`,
   plus `down_proj==64`, MTP writers `==2`, exactly one `embed_tokens`. Catches a
   tensor-name mismatch that would otherwise ship a half-abliterated model. **131
   tensors** edited when it passes; vision (333) never touched.
2. **Attention-sink screen** — Qwen3.8-27B's massive-activation dimension is
   **3994**. Orthogonalizing a direction that lives in dim 3994 produces a model
   that loads, runs, and emits garbage. The script aborts if the chosen layer's
   direction carries >1% of its energy in dim 3994 (recipe's layer-26 reference:
   0.06%).

The refusal direction is captured from **two chat-template renderings**
(`enable_thinking=false` and thinking at `xhigh`); the layer is auto-picked by
peak two-template `|cos|` agreement in the recipe's [18,45] window (anchor: 26).
Verified 2026-08-20 against `chat_template.jinja`: `enable_thinking=True` resolves
`reasoning_effort` to `'xhigh'` by default, so these really are the recipe's two
renderings — the low agreement is not a template-selection bug.

Two more gates were added 2026-08-20, both protecting numbers rather than
tensors:

3. **Batch-equivalence gate** — capture batches prompts, so before the real run
   it proves a padded batch reproduces one-at-a-time forwards and aborts
   otherwise. Tolerance is dtype-aware (bf16 5e-2, fp32 1e-3): the gate hunts
   *contamination*, not bit-exactness, and changing batch shape changes kernel
   tiling and therefore accumulation order, so a few ULP is expected. Real
   contamination is not subtle — the sharding defect read rel 1.00. Padding is on
   the **right**, and that is load-bearing: in a causal stack nothing after
   position *t* reaches position *t*, so trailing pads cannot touch the token we
   read, whereas left padding would feed pad tokens *into* the DeltaNet
   recurrence ahead of the prompt.
4. **Residency gate** (exit 8) and **allocator gate** (exit 9) — capture-only.
   See the gotchas; both encode defects that silently produce wrong numbers
   (multi-GPU sharding zeroes the upper residual stream; `expandable_segments`
   corrupts retained tensors).
5. **Write completeness check** — the write path is shard surgery with no model
   object, so the offload/meta silent-no-op failure class is gone; it instead
   verifies all 131 target tensors were found across the shards before declaring
   success (exit 7 otherwise) and refuses to overwrite an existing checkpoint
   (exit 10).

## Calibration corpus

`calibration.py`. The first capture used 8 harmful / 8 harmless and produced
`|cos|` agreement of **0.594** — valid but far off the recipe's 0.9925, and a
difference-in-means is only as clean as the number of prompts in each mean.

The recipe's line about a "held-out train/test split of 416/104 with overlap 0"
turns out to name the corpus exactly: **`mlabonne/harmful_behaviors` is 416 train
/ 104 test** (the AdvBench-derived pair used by the standard abliteration
notebooks), and both it and `mlabonne/harmless_alpaca` were **already staged** in
ana-ml2's HF dataset cache. So `--calib mlabonne` reproduces Robinson's
calibration set rather than approximating it. Read via pyarrow — no `datasets`
dependency, no hub access.

- **harmful** = `harmful_behaviors[train]`, file order, truncated to `n`. No
  seed dependence, so a re-capture is bit-reproducible from the flags alone.
- **harmless** = `harmless_alpaca[train]`, seeded sample (pool is 25058).
- **`harmful_behaviors[test]` (104) is reserved, not calibration.** It is the
  held-out generalization probe — the set Robinson reported 8% post-abliteration
  refusal on, and therefore our one directly comparable number. `load_calibration`
  will not draw from it and asserts overlap 0 against it, so a later edit cannot
  quietly turn the evaluation in-distribution.
- `--calib builtin` reproduces the legacy 8/8 run exactly.

Note the axis mismatch, and that it is deliberate: this corpus is **operational**
harm (hacking, fraud, weapons) while Cold-Fusion's measured refusal surface is
**creative** (explicit-sexual, graphic-torture). Robinson calibrated on exactly
this set and still drove creative refusal to 8% with self-harm guardrails intact,
which is the single-direction result holding across refusal types. Reproduce
first; a creative-axis supplement is the *second* experiment, not a variable to
change in the same run — and if one is added it must stay disjoint from
`services/refusal-probe/battery*.yaml`, or the post-write re-profile stops being
a held-out measurement.

## Sequence

Run from `/tank/aimodels/coldfusion-abliteration` on ana-ml2 (the deployed copy
of this directory), as `llmuser`, with `pylibs` on `PYTHONPATH`:

```bash
P=/tank/aimodels/coldfusion-abliteration
V=/tank/aimodels/quant-work/.venv/bin/python
M=/tank/aimodels/qwen38-27b-coldfusion-bf16
A=/tank/aimodels/qwen38-27b-coldfusion-abliterated-bf16
# CUDA_VISIBLE_DEVICES=0 is REQUIRED for capture (gate exit 8) and
# PYTORCH_CUDA_ALLOC_CONF must stay unset (gate exit 9) — see the gotchas below.
RUN="sudo -u llmuser env HF_HUB_OFFLINE=1 CUDA_VISIBLE_DEVICES=0 \
     PYTHONPATH=$P/pylibs $V $P/abliterate.py --model $M"

# 1. DRY RUN FIRST — verify the tensor map + coverage gate on the static
#    surface, no forward, no write. Safe with the seats up. Do not skip: this
#    confirms the recipe maps onto THIS checkpoint's names.
$RUN --dry-run

#    --- capture needs GPU0 to itself. The text weights are 51,300 MiB, and
#        freeing either seat alone leaves ~50,900 MiB -- BOTH must go. See
#        gotcha 3; the "only gen" line that used to be here was a unit error. ---
sudo docker stop -t 60 vllm-gen vllm-meromero-rp
cp $M/refusal-direction.pt $M/refusal-direction.pt.bak   # capture overwrites it

# 2. CONTROL RUN — the legacy 8/8 set. Reproduces layer 22, |cos| 0.5944, sink
#    0.001% exactly. Keep it as the regression test: ~30s of forwards that
#    validate the whole path against a known number before the real run.
$RUN --capture --calib builtin

# 3. THE REAL CAPTURE — Robinson's 416-prompt corpus. ~35s of forwards.
$RUN --capture --calib mlabonne
#    Measured 2026-08-20: layer 18, |cos| 0.6238, sink 0.360%.

# 4. Restore. If meromero was stopped too, start it FIRST — gen takes a fraction
#    of FREE VRAM at startup and will starve it otherwise.
sudo docker start vllm-gen

# 5. Abliterate (writes the new bf16). Only after 1-3 pass, and only on the
#    operator's go — this is the destructive step. Needs the VRAM window again
#    Shard-level surgery: reads/writes the 18 safetensors shards directly, NO
#    model object, NO GPU. That is a correctness requirement, not just thrift —
#    see "Why the write is shard surgery" below. --direction is REQUIRED.
$RUN --out $A --direction $M/refusal-direction.pt
```

### Flags added 2026-08-20

| flag | default | why |
|---|---|---|
| `--calib {mlabonne,builtin}` | `mlabonne` | corpus selection; `builtin` = legacy 8/8 |
| `--calib-n-harmful` | 416 | the full train split, as the recipe used |
| `--calib-n-harmless` | 416 | matched n from alpaca |
| `--calib-seed` | 0 | harmless sample only; harmful is order-deterministic |
| `--batch-size` | 8 | 832 prompts x 2 templates = 1664 forwards; batching is what makes that affordable |
| `--capture-dtype {bfloat16,float32}` | `bfloat16` | bf16 (50 GB, full 64 layers, one GPU) is validated deterministic + coherent; fp32 (111 GB, needs `--max-layer`) is a misdiagnosis-era escape hatch that agrees to 5e-4 |
| `--max-layer` | off | capture-only. Truncates the decoder. **Exact, not an approximation** — a causal stack's layer-N state cannot depend on layers above N. Only needed with `--capture-dtype float32`; bf16 fits whole. Refused on the write path. |

## ✅ RESULT — layer 35, and why the recipe's layer-selection metric had to be replaced

The write lands and works. Verified bitwise: **131/131 target tensors changed,
333/333 vision byte-identical (delta 0.0), 735/735 other tensors untouched.**
A/B against stock on a matched battery (greedy, held-out prompts):

| probe | stock | abliterated (L35) |
|---|---|---|
| explicit sexual (target axis) | refuses | **complies** |
| graphic torture (target axis) | refuses | **engages** (softened) |
| spam-bot / malware (held-out AdvBench) | refuses | **complies / engages** |
| self-harm method (guardrail) | redirects | **still redirects** |
| coherence ×2 | fine | **fine** |

That is the Robinson design point exactly: creative refusals fall, the self-harm
guardrail survives, coherence intact. Output at
`/tank/aimodels/qwen38-27b-coldfusion-abliterated-L35-bf16`.

**It took THREE captures, and the lesson is the metric.** The recipe selects the
abliteration layer by peak two-template `|cos|` agreement. On this checkpoint that
metric is not just weak, it is *anti-correlated* with what matters:

| capture | selector | layer picked | Cohen's d | result |
|---|---|---|---|---|
| 1 (8/8) | agreement | 22 | 5.70 | (sharding-corrupted, void) |
| 2 (416/416) | agreement | **18** | **5.51 — worst in window** | write was a **behavioral no-op** |
| 3 (416/416) | **separation, sink-gated** | **35** | **9.35** | **works** |

The tell that cracked it: after capture 2's write changed *nothing*, a per-layer
separation diagnostic (does the direction split harmful from harmless
activations?) showed the direction is **excellent** — AUC 0.9996+ across the whole
window — and that agreement had steered us to layer 18, the single **weakest**
separator (d 5.51 vs 9.89 at the peak). Agreement was measuring answer-vs-reason
*mode* (the two templates end `</think>\n\n` vs `<think>\n`), not refusal, and on
a heavily-merged base that mode term dominates.

**So selection is now by separation (Cohen's d), gated on the sink screen.**
Separation and sink-energy both rise with depth, so the raw peak (L39, d 9.89)
is sink-dominated (1.97% > 1%) and would brick the model; the script filters to
layers that pass the screen and takes the best separator among them — **L35, d
9.35 (within 5% of peak), sink 0.094% (10× under the limit).** One pass, no
guess-and-retry. Agreement is still computed and printed, as a diagnostic.

> The corpus-size hypothesis this session started on was **falsified**: 52× more
> calibration data (8→416) moved agreement 0.594→0.624, essentially nothing. The
> problem was never the calibration set. See the calibration section above; kept
> as the record of a dead-end worth not re-running.

## ✅ THESIS RESULT — the in-band-abliterated MTP head accepts BETTER than a graft (2026-08-20)

The whole reason to abliterate Cold-Fusion ourselves rather than run the incumbent
Heretic seat: Heretic leaves the MTP head a **byte-identical base graft** (its
wrapper never loads it), while the Robinson formula abliterates the MTP head
**in-band** (its 2 residual writers). The open question was whether that in-band
edit *survives* — an abliterated MTP head that no longer predicts well would kill
speculative decoding. Measured, end to end:

| metric | L35 quant | incumbent (heresy) | gate | verdict |
|---|---|---|---|---|
| **MTP acceptance** (median, 8 cache-busted topics) | **59.1%** (51–65%) | ~47% | ≳40% | **PASS — beats incumbent** |
| decode tok/s (median) | 118.7 | ~95–103 | ≥ incumbent | faster (⚠ image-confounded, read as "not worse") |
| abliteration survives quant | yes | — | creative↓, self-harm intact | **PASS** |
| coherence / no catatonia | clean | — | eyeball | **PASS** |

So the in-band MTP abliteration doesn't merely preserve speculative decoding — the
abliterated head **accepts 59.1% vs the untouched graft's ~47%.** That is the
additive delta the experiment set out to test, and it's positive.

**Pipeline** (`services/gen-seat-mixed-quant/`): mixed NVFP4 (W4A4 L0–55 MLP) +
FP8 (attn/linear_attn/lm_head/L56–63 MLP) + FP8 KV → 22.5 GB. Post-quant grafts
the **abliterated** MTP (15 tensors, 849 MB) from the L35 bf16 source and
re-injects `re:^mtp.*` into `quantization_config.ignore` (llm-compressor pruned it
again — the two-rounds-lost bug, fired and repaired as designed). Output:
`/tank/aimodels/qwen38-27b-coldfusion-L35-nvfp4-mixed`. Result JSON:
`services/gen-seat-mixed-quant/bench/mtp_coldfusion_L35.json`.

> ⚠️ Env foot-gun banked: the quant venv's `transformers` moved to 5.10 /
> `llmcompressor` 0.12 since the Aug-15 heresy quant, and the top-level config no
> longer delegates `num_attention_heads` to `text_config` → oneshot raised
> "Cannot determine num_attention_heads". `quant_mixed_nvfp4.py` now promotes those
> fields from `text_config` for the duration of quant, then restores. Also: a
> small (<~23 GB) quant saves as a **single** `model.safetensors` with no index,
> so `post_quant`'s MTP graft needs an index built first (from the safetensors
> header — never `safe_open`, which mmaps the whole shard and ENOMEMs on ZFS).

**NOT cut over.** The incumbent gen seat is untouched. Making L35 the `gen` seat is
a separate operator decision needing the full Stage-3 gate (PPL, prefill, surface
6/6, refusal-probe battery) + the real multi-turn-use hold (the 2026-08-14
delete-too-early / multi-day-degeneration lesson). The thesis is proven; the
cutover is a distinct call.

## ✅ KL RESULT — the surgery is highly selective (2026-08-20)

`kl_divergence.py` measures **first-token KL(stock ‖ abliterated)** over the full
248,320-entry vocabulary, bf16 vs bf16, on prompts the direction was never fitted
on. Both classes are scored separately because a single mixed average would hide
the only thing worth knowing: the divergence is supposed to be *large* on harmful
prompts (that is the effect) and *small* on benign ones (that is the damage).

| mode | class | n | median | mean | p95 | max | top-1 agreement |
|---|---|---|---|---|---|---|---|
| **answer** | harmless (held out) | 256 | **0.0211** | **0.0364** | 0.1219 | 0.2654 | 89.8% |
| **answer** | harmful (reserved test) | 104 | **0.5996** | 0.6992 | 1.6937 | 1.9920 | 55.8% |
| think | harmless (held out) | 256 | 0.0042 | 0.0066 | 0.0205 | 0.0392 | 94.5% |
| think | harmful (reserved test) | 104 | 0.3068 | 0.3186 | 0.4689 | 0.5298 | 57.7% |

**Selectivity — harmful/harmless median KL — is 28.4× in answer mode and 72.8× in
think mode.** The direction moves the model hard exactly where it is meant to and
leaves benign behaviour close to untouched: on held-out harmless prompts the
abliterated model still picks the *same first token* 89.8% of the time.

**Noise floor: exactly 0.0** in both modes (32 prompts re-run through the same
model, self-KL). This stack is bit-deterministic here, so every digit above is
signal — none of it is bf16 jitter. It also validates the scoring path end to end:
a bug in the KL code would almost certainly have shown up as a non-zero floor.

**The reverse-KL asymmetry is the abliteration's signature.** On harmful prompts
in answer mode, KL(stock‖abl) is 0.70 but KL(abl‖stock) is **1.43** — the
abliterated model puts substantial mass where the stock model put almost none.
That is precisely what removing a refusal direction does, and it is a sanity check
that the surgery did the intended thing rather than merely adding noise.

### Against the Heretic reference figures — favourable, with a caveat

| model | first-token KL, harmless | abliteration method |
|---|---|---|
| `JonathanColetti/Qwen3.8-27B-Uncensored` (prior gen seat) | 0.1191 | Heretic, out-of-band MTP |
| `absolute-heresy` (**current** gen seat) | 0.0759 | Heretic v1.4.0 + SOMPOA |
| **Cold-Fusion L35 (ours)** | **0.0211 median / 0.0364 mean** | Robinson, in-band MTP |

⚠️ **Not a head-to-head.** The two reference numbers are Heretic's own optimizer
output on a *different base model*, with *its own* harmless prompt set and
template. Same metric, different measurement conditions — read this as
order-of-magnitude ("ours is not worse, and looks materially gentler"), not as a
ranking. A true head-to-head would mean re-measuring the incumbent through this
same script, which is one more GPU window if the cutover decision ever needs it.

Also note what this does **not** cover: the MTP head (`AutoModelForCausalLM` is
text-only, so this is the main head only — MTP is gated on acceptance, measured at
**59.1%**), quantization damage (both sides are bf16), and anything past the first
token. Consistent with `reference_abliteration_mtp_lessons`, KL is reported here
as a *fidelity* number, not as the viability gate.

**Reproducibility: exact.** The measurement was run twice — once single-process,
once through the two-process design below — and **all 720 per-prompt KL values are
bit-identical** between them. Combined with the 0.0 self-KL floor, the numbers
above are stable across processes, not just within one.

Artifacts: `kl-L35.json` (+ `kl-L35-rerun.json`, the reproducibility check) and the
two `.ref.pt` / `.cand.pt` log-prob caches, beside the harness on ana-ml2. Run
cost: **2m40s** single-process, **3m26s** two-process, both seats down.

```bash
# free, no GPU, safe with the seats up — run this first
$V $P/kl_divergence.py --ref $M --cand $A --out $P/kl-L35.json --dry-run
# the real thing: needs BOTH GPU0 seats stopped (see gotcha 3)
$V $P/kl_divergence.py --ref $M --cand $A --out $P/kl-L35.json
```

**Why it runs one process per model.** The default `--stage all` re-execs itself
once per checkpoint (`--stage ref`, then `--stage cand`), each writing its
first-token log-probs to a ~682 MiB `.pt` cache, then scores from the caches.
This is not tidiness — **it is the only teardown that works.** Measured, free VRAM
after the reference model:

| teardown | free VRAM |
|---|---|
| `del model` + `gc.collect()` + `empty_cache()` | 45,287 MiB |
| the same, model confined to an inner frame that exits | 45,287 MiB |
| **the process exits** | **96,689 MiB** |

The weights survive both in-process teardowns. The very first run only completed
because PyTorch's allocator hit OOM on the second load, collected, and retried —
the second model landed on the card *by rescue, not by design*, and on this
architecture a silent CPU offload does not raise, it returns confident garbage
(gotcha 1). The headroom gate (`exit 10`) is what turned that from an invisible
near-miss into a loud failure. Side benefit: the `ref` cache is reusable, so
measuring a different candidate against the same stock model skips a stage
entirely (`--stage cand` then `--stage score`).

⚠️ **The old residency gate could not fail.** It read `hf_device_map`, which
transformers leaves **empty** when the whole model fits on one device — so it
printed "(unsharded)" both when everything was fine and when there was nothing to
inspect. It now reads `{p.device for p in model.parameters()}` and prints the real
placement (`all parameters on cuda:0`).

## Why the write is shard surgery, not `model.save_pretrained`

The `--out` path edits the 18 safetensors shards directly and never instantiates
a model for the write. This is correctness, not thrift. `AutoModelForCausalLM`
resolves to `Qwen3_5ForCausalLM` — the **text** model — so saving from it would
(a) **drop all 333 vision tensors**, silently breaking the byte-identical-vision
guarantee, and (b) **skip the MTP head**, which the `ForConditionalGeneration`
wrapper does not load (the same reason the incumbent gen seat's Heretic pass left
its MTP head an untouched base graft) — and the in-band MTP edit is the entire
point of the Robinson formula. Neither failure raises. Shard surgery re-serializes
every non-target tensor from the exact bytes read, so vision and the other 1068
tensors are byte-identical *by construction*, the two MTP writers are just two
more keys, and the whole offload/meta-tensor silent-no-op class disappears with
the model object. Math is done in fp32, stored back at the original bf16.

## Verify after (do not trust the write blind)

1. **Vision byte-identical + target count** — `services/coldfusion-abliteration`
   verify: `targets changed=131/131  vision identical=333/333 delta=0.0  other
   differ=0/735`. Done 2026-08-20, clean.
2. **Refusal re-profile** — the ad-hoc battery above is a smoke test. The full
   canonical re-profile still owed: run `services/refusal-probe/` (the gen-seat
   harness, NOT the GGUF one) once L35 is served, and confirm creative refusals
   near the RobinsonLabs 8% floor with self-harm guardrails intact.
3. **MTP acceptance** — the whole point of the in-band MTP edit; measure on the
   quantized build per `services/gen-seat-mixed-quant/RUNBOOK-heresy-swap.md`.
   Gate ≳40% (`reference_abliteration_mtp_lessons` — gate on acceptance, not KL).
4. **PPL / coherence / no catatonia** — DavidAU fine-tunes are idiosyncratic;
   eyeball the outputs, don't trust the metric alone. (Smoke: coherent, no
   catatonia observed.)

Then, if it holds, NVFP4-quantize via `services/gen-seat-mixed-quant/` and it
becomes a gen-seat candidate — **do not delete the incumbent weights** until it
survives real multi-turn use (the 2026-08-14 delete-too-early lesson).

## ⚠️ Environment gotchas (2026-08-20 — cost real time, read before re-running)

> ⚠️ **RETRACTED 2026-08-20 — the "bf16 NaNs, use fp32" rule that lived here was
> a misdiagnosis, and it sent the next session down a 111 GB dead end.** The NaN
> was never precision. It was the two defects below. fp32 only made it *rarer*,
> which is worse than failing outright, because it let a broken forward produce a
> plausible-looking direction. **bf16, full 64 layers, one GPU: 50 GB, exactly
> deterministic through layer 63, coherent prose, 4.3× the throughput.**

**1. ⭐ Never let the capture shard across both GPUs.** With `device_map="auto"`
across the two Blackwells, this model loads clean, raises nothing, and computes
garbage: the residual stream collapses to **exactly zero** two layers past the
GPU0→GPU1 boundary and the logits decode to rubbish. Layers *below* the boundary
are healthy and bit-identical to a single-GPU run — which is exactly why the
first capture looked fine. It picked layer 22, which sat on GPU0 in the healthy
region; the upper half of its window was zeros and their agreement scores were
meaningless.

   → **Run `CUDA_VISIBLE_DEVICES=0`.** The `--capture` path enforces this with a
   residency gate (exit 8) that refuses a sharded or offloaded model.

**2. ⭐ Never set `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`.** On this
stack it corrupts tensors that outlive their allocation — captured states came
back with Inf/NaN/zeros that **moved between bit-identical forwards**. Unset,
the same forwards are exactly reproducible. The old runbook recommended this flag
for headroom; it buys corruption. Gated (exit 9).

   The tell worth remembering: a real numerical blowup **propagates** to later
   layers and is **deterministic**. This did neither. *If a NaN doesn't
   propagate, debug memory, not math.*

**3. bf16 fits on one GPU — but it needs BOTH GPU0 seats stopped, not one.**

> ⚠️ **CORRECTED 2026-08-20.** This section used to read "50.1 GB … a capture
> needs only `vllm-gen` stopped." **The unit was wrong and the conclusion that
> rode on it was wrong.** The real figure is **50.10 GiB = 51,300 MiB = 53.8 GB**
> of text-only weights, measured from the safetensors headers rather than read off
> a `/1e9` print:
>
> | | GB | GiB | MiB |
> |---|---|---|---|
> | checkpoint total | 55.56 | 51.75 | 52,989 |
> | vision (not loaded by `AutoModelForCausalLM`) | 0.92 | 0.86 | 879 |
> | MTP (not loaded either) | 0.85 | 0.79 | 810 |
> | **text-only — what actually lands on the card** | **53.79** | **50.10** | **51,300** |
>
> GPU0's two tenants are meromero (50,072 MiB) and gen (46,304 MiB), and
> **freeing either one alone leaves at most 50,933 MiB — about 400 MiB short.**
> A run that assumes one seat is enough will stop a service, sit at the edge, and
> then OOM. Stop **both**. Recompute this table if the checkpoint changes; do not
> trust a remembered gigabyte figure.

Both seats down leaves ~97,200 MiB, so the fit is comfortable rather than
marginal. Gate the run on **observing** the free VRAM (`nvidia-smi
--query-gpu=memory.free`) rather than sleeping after `docker stop`, and put the
restore in a `trap ... EXIT` so an abort hands the seats back — the 2026-08-20
aborted window did exactly that and cost nothing but two minutes.

(`--capture-dtype float32` remains as an escape hatch; it needs 111 GB, so it
also needs `--max-layer 46` to fit on one card. The two agree to 0.0005, so there
is no reason to reach for it.)

**⚠️ Restore after: start `vllm-meromero-rp` first, and WAIT FOR IT TO GO HEALTHY
before starting `vllm-gen`.** This ordering is load-bearing, and "first" means
*fully up*, not *ten seconds earlier* — a `docker start meromero; sleep 10; docker
start gen` put meromero into a **7-restart crash-loop** on 2026-08-20, because gen
finished claiming the card while meromero was still loading weights:

```
ValueError: Free memory on device cuda:0 (35.3/94.97 GiB) on startup is less than
desired GPU memory utilization (0.52, 49.38 GiB).
```

The mechanism, stated precisely because a half-right version of it is what caused
the mistake: `--gpu-memory-utilization` is a fraction of **total** VRAM (0.52 ×
94.97 = 49.38 GiB is meromero's *target*), but vLLM gates startup on **free** VRAM
— it refuses to start unless the card currently has the whole target available.
So the two seats coexist only in the order they were originally brought up. GPU0
runs at ~96.4/97.9 GB with about 0.4 GiB of slack; whichever seat starts second
gets whatever the first one left, and meromero is the one that does not fit in the
remainder. Recovery when it does happen: `docker stop vllm-gen`, wait for meromero
to report `healthy`, then `docker start vllm-gen`.

**4. fla is irrelevant here — but harmless.** `fla` + `einops` are `--target`
-installed to `/tank/aimodels/coldfusion-abliteration/pylibs` and reached via
`PYTHONPATH` (the shared `quant-work/.venv` is not llmuser-writable). Tested
2026-08-20: the nondeterminism reproduces **identically with `fla` absent**, so
the linear-attention kernel was never the culprit. Keep passing `PYTHONPATH`;
just don't blame it.

## Status

Harness written 2026-08-19; bf16 fully staged. Dry-run PASSED (recipe maps 1:1,
131 tensors). **`--capture` PASSED 2026-08-20** (fp32, after the gotchas above):
refusal direction is **finite, unit-normed, layer 22**, sink energy **0.0008%**
in dim 3994 (recipe L26 ref 0.06%, threshold 1%) — clean, not sink-dominated.
Saved to `qwen38-27b-coldfusion-bf16/refusal-direction.pt`.

### 2026-08-20, second session — the corpus hypothesis is FALSIFIED

**Measured, on a forward that is trustworthy for the first time:**

| calibration | layer | `\|cos\|` agreement | sink energy |
|---|---|---|---|
| 8 / 8 (legacy) | 22 | **0.5944** | 0.001% |
| 416 / 416 (Robinson's corpus) | 18 | **0.6238** | 0.360% |

**52× more calibration data bought +0.03.** The small calibration set was *not*
why agreement sat at 0.59, and Robinson's 0.9925 is not reachable on this
checkpoint by adding prompts. Agreement is uniformly ~0.54–0.62 across the whole
healthy window (L18 0.6238, L22 0.6158, L21 0.6101, L19 0.5944, L28 0.5841), not
peaked-and-noisy — which is the signature of a genuinely diffuse direction rather
than an under-sampled one.

Cross-validated two ways: the 8/8 run **reproduces the previous session's 0.5944
at layer 22 exactly**, and fp32-truncated vs bf16-full-64-layer agree to 0.0005.
So the number is real and the pipeline is sound.

> ⚠️ The first capture's log reported 0.594 as `|cos|=0.8538`. Reporting bug,
> fixed: the line printed the **global** `agree.max()` next to the **window's**
> argmax layer. The global peak sits in the early layers where the dim-3994
> massive activation dominates both templates and inflates agreement for reasons
> unrelated to refusal. `0.5944` was always the real number.

**The leading explanation is the metric, not the model.** The two renderings do
not just differ in formatting — they leave the model in **different generative
modes** at the token we read:

- `enable_thinking=false` ends `…<think>\n\n</think>\n\n` → about to write **the answer**
- `xhigh` ends `…<think>\n` → about to write **chain-of-thought**

So `|cos|` here measures *refusal semantics **plus** answer-vs-reason mode*.
Robinson's stock Qwen3.8-27B scored 0.99 across that same split, so on their base
the refusal component dominated; on this DavidAU GAIN merge the mode difference
apparently does not let it. **Note what this does and does not impugn:** the
direction actually used is `dirs[False]` — the no-think one. Cross-template
agreement is only a *quality check*, and a check that conflates two factors is a
weak gate to block on.

**The check that would actually settle it is a split-half.** Split the 416
harmful in two, derive a direction from each half *through the same template*,
and take `|cos|`. That isolates sampling noise — the thing calibration size
governs — with no mode term at all. If split-half is ~0.99, the direction is
well-estimated, the 0.62 is a mode artifact, and the write is justified on a
direction we can defend. If split-half is also ~0.6, the refusal representation
in this checkpoint is genuinely diffuse and single-direction abliteration is the
wrong instrument for it. Cheap: no extra forwards, just two accumulators.

**Status: the destructive `--out` write has NOT been executed.** It gates on the
operator's go. The saved direction
(`refusal-direction.pt`, layer 18, 416/416, sink 0.360%) is usable but its
quality is unresolved pending the split-half. The legacy 8/8 direction is
preserved at `refusal-direction.pt.bak-8x8`.