vh
c8f128bdff
Pulled orcarouter/Qwen3.8-27B-Uncensored at rev 9878936b (55.5 GB, gated, our token has access) and built /tank/aimodels/qwen38-27b-orcarouter-nvfp4-mixed (23.4 GB, mixed NVFP4+FP8). Verified, not yet cut over. The operator asked whether we could apply the Robinson path to the MTP head. We cannot, because the author already did. compare_mtp_head.py against the verbatim base graft: 13 of 15 tensors byte-identical, exactly 2 differ -- mtp.layers.0.self_attn.o_proj.weight and mtp.layers.0.mlp.down_proj.weight, which are precisely the two residual writers our own abliterate.py targets (EXPECT_MTP_WRITERS = 2). Reverse-engineered the edit from the weights alone (mtp_delta.py, added here): sigma2/sigma1 = 0.0164 on BOTH tensors rank-1, a single-direction projection |cos| between the two recovered dirs = 1.0000 ONE shared direction ||delta||/||W|| = 1.42% and 1.41% a gentle, consistent projection sink energy dim 3994 = 0.0000% sink-clean; Heretic's was 6.18% That is the Robinson in-band MTP abliteration, already applied, with a direction that passes our sink screen outright. Nothing to do but preserve it, and the quant carries it byte-identically. This is the configuration the entire Cold-Fusion experiment was designed to test and never cleanly delivered. The new format screen paid for itself on its first real use: think_prior.py on the bf16 BEFORE any GPU time gave P(<think>) = 1.23e-06 at rank 52, against Cold-Fusion stock 0.1850 and h300 0.2216. Roughly 150,000x cleaner. Two durable findings about the pipeline itself: The quant needs ~17 GB, not a whole card. It ran entirely in GPU1's spare 16 GB with ZERO production seats stopped -- the h300 run's "stop BOTH GPU0 seats" was never necessary, it simply had a free card by coincidence. The first attempt OOM'd by 2.37 GiB at layer 64 of 65 with 3.57 GiB reserved-but-unallocated, which is fragmentation, and PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True closed it. post_quant.py now builds a missing output index from the safetensors headers. A sub-23 GB quant saves one bare shard with no index, and post_quant needs one; this has broken three separate rounds and been hand-fixed every time. The header is read by struct-unpacking the u64 length and parsing the JSON -- never safe_open, which mmaps the whole 22 GB shard and ENOMEMs on ZFS. Artifact verified: mixed-precision, 1968 tensors, 15 mtp, 333 visual, re:^mtp.* present in the ignore list (llm-compressor pruned it as always), preproc restored. Imatrix deferred per operator; the log confirms the usual uniform-MSE fallback, so this build stays apples-to-apples with heresy's PPL 6.910.
45 lines
2.1 KiB
Python
45 lines
2.1 KiB
Python
#!/usr/bin/env python3
|
|
"""Is orcarouter's MTP-head edit a single-direction (Robinson-style) projection?
|
|
|
|
Robinson orthogonalizes a residual WRITER as W' = (I - d d^T) W, so
|
|
delta = W' - W = -d d^T W is EXACTLY RANK 1 with left singular vector d.
|
|
Two independent predictions follow, both checkable from the weights alone:
|
|
1. sigma_2/sigma_1 ~ 0 for each edited tensor
|
|
2. the d recovered from o_proj and from down_proj must AGREE (|cos| ~ 1),
|
|
because Robinson uses ONE shared direction
|
|
Then sink-screen the recovered d: Heretic's was 6.18% concentrated in dim 3994,
|
|
which is what made an in-band graft unsafe on that trunk. Ours want < 1%.
|
|
"""
|
|
import json, torch
|
|
from pathlib import Path
|
|
from safetensors import safe_open
|
|
|
|
CAND = Path("/tank/aimodels/qwen38-27b-orcarouter-bf16")
|
|
REF = Path("/tank/aimodels/qwen38-27b-uncensored-bf16")
|
|
KEYS = ["mtp.layers.0.self_attn.o_proj.weight", "mtp.layers.0.mlp.down_proj.weight"]
|
|
SINK_DIM = 3994
|
|
|
|
def get(d: Path, key: str):
|
|
idx = json.loads((d / "model.safetensors.index.json").read_text())["weight_map"]
|
|
with safe_open(d / idx[key], framework="pt") as f:
|
|
return f.get_tensor(key)
|
|
|
|
dirs = {}
|
|
for k in KEYS:
|
|
delta = (get(CAND, k).float() - get(REF, k).float())
|
|
U, S, Vh = torch.linalg.svd(delta, full_matrices=False)
|
|
ratio = (S[1] / S[0]).item()
|
|
d = U[:, 0] # residual-space direction (dim 5120)
|
|
dirs[k] = d
|
|
sink = (d[SINK_DIM] ** 2).item() / (d @ d).item()
|
|
print(f" {k.split('.',2)[2]:<28} shape={tuple(delta.shape)}")
|
|
print(f" sigma2/sigma1 = {ratio:.6f} <- rank-1 if ~0")
|
|
print(f" ||delta||/||W|| = {(delta.norm()/get(REF,k).float().norm()).item():.5f}")
|
|
print(f" sink energy dim {SINK_DIM} = {sink*100:.4f}% <- want <1%, Heretic's was 6.18%")
|
|
|
|
a, b = dirs[KEYS[0]], dirs[KEYS[1]]
|
|
cos = torch.abs(a @ b / (a.norm() * b.norm())).item()
|
|
print(f"\n |cos| between the two recovered directions = {cos:.4f} <- ~1 means ONE shared direction")
|
|
top = torch.topk(a.abs(), 5)
|
|
print(f" top-5 |d| coords (o_proj): {[(int(i), round(float(v),4)) for i, v in zip(top.indices, top.values)]}")
|