vh
530f1452e8
Captures the session's real work as the in-flight focus: abliterating DavidAU Cold-Fusion with the Robinson formula. fp32 capture succeeded (finite direction, layer 22, sink-clean) but two-template agreement is 0.59 vs Robinson's 0.99 — calibration-set expansion is the next step. New detail file records the full saga including the transformers/DeltaNet bf16-NaN fight (fp32 fix, the causal-conv1d kernel gap, the seat-restart VRAM-greed gotcha). Supersedes the earlier "watch for DavidAU's heretic build" posture — we abliterate it ourselves. Auto-archived 4 closed entries (Recent decisions: Booth-3-features 08-05, worldtree-sdk 07-31; Tried and abandoned: containerd-race 08-03, mv-rename 08-02) to archival-memory.md; the rest of the over-cap entries are held back by the <14-day and open-deferred guards. Index 331 -> 327.
96 lines
5.5 KiB
Markdown
96 lines
5.5 KiB
Markdown
# `[2026-08-20]` Cold-Fusion abliteration — Robinson recipe captured, and the transformers/DeltaNet bf16-NaN fight
|
|
|
|
The real work of the session: abliterate `DavidAU/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1`
|
|
using the MTP-aware, vision-preserving **Robinson formula** (documented in
|
|
`docs/pfi/abliteration-recipe-qwen38.md` from `RobinsonLabs/Qwen3.8-27B-abliterated`).
|
|
Harness: `services/coldfusion-abliteration/`. Runs on ana-ml2.
|
|
|
|
## Why this model, why abliterate it ourselves
|
|
|
|
Stock Cold-Fusion's refusal profile was **probed 2026-08-19** (Q6_K GGUF on
|
|
llama.cpp, 24-prompt battery, hand-verified after a keyword-classifier bug):
|
|
**~33% creative refusal**, concentrated on **explicit-sexual + graphic-torture**;
|
|
4/5 hard-harm technical refused; self-harm guardrails intact 3/3; benign
|
|
over-refusal 0. So there is a real creative-content refusal surface to remove.
|
|
This **supersedes** the earlier "watch for DavidAU's own heretic build" posture —
|
|
we abliterate it ourselves.
|
|
|
|
**It is additive over the current gen seat.** The live Heretic seat
|
|
(`qwen38-27b-heresy-bf16`) left its MTP head a **byte-identical base graft** —
|
|
the `Qwen3_5ForConditionalGeneration` wrapper never loads it, so Heretic could
|
|
not touch it. The Robinson formula abliterates the MTP head **in-band** (its 2
|
|
residual-write matrices), and the MTP head is what gates speculative acceptance.
|
|
That in-band MTP edit is the delta this experiment tests.
|
|
|
|
## Recipe maps 1:1 — dry-run PASSED
|
|
|
|
Against the staged bf16: 1199 tensors, 333 vision preserved,
|
|
`down_proj=64 o_proj=16 linear_out=48 mtp=2 embed=1`, coverage gate 6/6, exactly
|
|
**131** tensors to orthogonalize. Same architecture as RobinsonLabs' base, no
|
|
name drift. Two hard gates in the harness halt before any write: the coverage
|
|
identity `o_proj(16)+linear_out(48)==64`, and the attention-sink screen on
|
|
**dim 3994** (orthogonalizing a direction living there bricks the model).
|
|
|
|
## Capture SUCCEEDED — but only after a real environment fight (the durable lessons)
|
|
|
|
**The transformers Qwen3.5 DeltaNet linear-attention NaNs in bf16 on ana-ml2.**
|
|
The fast-path needs BOTH `flash-linear-attention` (`fla`, triton, installs fine)
|
|
AND `causal-conv1d` (**needs nvcc to build — absent, no prebuilt wheel**).
|
|
Without causal-conv1d the DeltaNet short-conv runs the torch fallback, which
|
|
produces **nondeterministic all-NaN** hidden states in bf16 (same 11-token input:
|
|
finite on one forward, NaN at layer 4 on the next). bf16 and fp32 share exponent
|
|
range, so this is **precision-driven catastrophic cancellation, not overflow** —
|
|
**fp32 resolves it.** Diagnosed via `diag_nan.py` / `diag2.py`: `sdpa` + plain
|
|
prompt = 65 layers all finite; chat-template input = NaN; the trigger is the
|
|
input path through the unstable recurrence.
|
|
|
|
Fixes, all in the committed harness (`7abd301`):
|
|
- **`--capture` loads fp32**; the write/surgery path stays bf16 (no forward, no NaN).
|
|
- **A finite-gate aborts on a non-finite direction** — the sink screen alone
|
|
can't catch it (`nan > threshold` is False, so a NaN direction "passed" it and
|
|
saved silently on the first run).
|
|
- `attn_implementation="sdpa"` pinned.
|
|
|
|
**fp32 (110 GB) needs the whole GPU.** device_map=auto packed it tight and the
|
|
forward OOM'd against the resident seats. Had to **stop three seats** for VRAM:
|
|
`vllm-meromero-rp`, `vllm-fablefusion-probe`, and production `vllm-gen`.
|
|
⚠ **Restart order matters:** gen restarted into an empty GPU0 and greedily
|
|
grabbed 64 GB (vLLM takes a fraction of *free* memory at startup), starving
|
|
meromero into a crash-loop. Fixed by bringing **meromero up first**, then gen
|
|
into the remainder. All three restored to healthy.
|
|
|
|
⚠ **fla lives in a side dir, not the venv.** The shared
|
|
`/tank/aimodels/quant-work/.venv` is not llmuser-writable; `fla` + `einops` are
|
|
`--target`-installed to `/tank/aimodels/coldfusion-abliteration/pylibs` and
|
|
reached via `PYTHONPATH`. Prune deps that shadow the venv's torch/transformers.
|
|
|
|
## Result
|
|
|
|
Refusal direction: **finite, unit-normed, layer 22**, sink energy **0.0008%**
|
|
in dim 3994 (recipe L26 ref 0.06%, threshold 1%) — clean, not sink-dominated.
|
|
Saved to `/tank/aimodels/qwen38-27b-coldfusion-bf16/refusal-direction.pt`.
|
|
|
|
⚠ **QUALITY CAVEAT — the reason the next step is calibration-set expansion.**
|
|
Two-template `|cos|` agreement at layer 22 is **0.59**, well below Robinson's
|
|
0.99. Almost certainly the small calibration set: **8 harmful / 8 harmless**
|
|
(HARMFUL/HARMLESS in `abliterate.py`) vs Robinson's **416 / 104**. The direction
|
|
is valid and sink-clean but noisier than ideal; abliterating on it risks
|
|
under-removing refusals or nicking capability. **Expand the sets to a few
|
|
hundred each and re-capture** before the `--out` write.
|
|
|
|
## Sequence from here
|
|
|
|
1. **Expand HARMFUL/HARMLESS calibration sets** → re-capture (fp32, seats down).
|
|
2. `--out` write (bf16 surgery, no forward) → `qwen38-27b-coldfusion-abliterated-bf16`.
|
|
3. Verify: vision byte-identical, refusal re-profile via `services/refusal-probe/`
|
|
(the canonical harness, NOT the ad-hoc GGUF one), MTP acceptance on the quant
|
|
(gate ≳40%, not KL — `reference_abliteration_mtp_lessons`), PPL/coherence.
|
|
4. NVFP4-quantize via `services/gen-seat-mixed-quant/` → gen-seat candidate.
|
|
**Do NOT delete the incumbent** (`qwen38-27b-heresy-nvfp4-mixed`) until it
|
|
holds through real multi-turn use.
|
|
|
|
bf16 staged at `/tank/aimodels/qwen38-27b-coldfusion-bf16` (pinned `9c44193`,
|
|
provenance recorded). All write paths re-stop the seats for fp32 VRAM — batch
|
|
re-capture + write in one window. Commits `ccb56a0`, `1857a8e`, `b56cb0d`,
|
|
`7abd301`.
|