feat(gen-seat): quant orcarouter — its MTP head is already Robinson-abliterated in-band
Pulled orcarouter/Qwen3.8-27B-Uncensored at rev 9878936b (55.5 GB, gated, our token has access) and built /tank/aimodels/qwen38-27b-orcarouter-nvfp4-mixed (23.4 GB, mixed NVFP4+FP8). Verified, not yet cut over. The operator asked whether we could apply the Robinson path to the MTP head. We cannot, because the author already did. compare_mtp_head.py against the verbatim base graft: 13 of 15 tensors byte-identical, exactly 2 differ -- mtp.layers.0.self_attn.o_proj.weight and mtp.layers.0.mlp.down_proj.weight, which are precisely the two residual writers our own abliterate.py targets (EXPECT_MTP_WRITERS = 2). Reverse-engineered the edit from the weights alone (mtp_delta.py, added here): sigma2/sigma1 = 0.0164 on BOTH tensors rank-1, a single-direction projection |cos| between the two recovered dirs = 1.0000 ONE shared direction ||delta||/||W|| = 1.42% and 1.41% a gentle, consistent projection sink energy dim 3994 = 0.0000% sink-clean; Heretic's was 6.18% That is the Robinson in-band MTP abliteration, already applied, with a direction that passes our sink screen outright. Nothing to do but preserve it, and the quant carries it byte-identically. This is the configuration the entire Cold-Fusion experiment was designed to test and never cleanly delivered. The new format screen paid for itself on its first real use: think_prior.py on the bf16 BEFORE any GPU time gave P(<think>) = 1.23e-06 at rank 52, against Cold-Fusion stock 0.1850 and h300 0.2216. Roughly 150,000x cleaner. Two durable findings about the pipeline itself: The quant needs ~17 GB, not a whole card. It ran entirely in GPU1's spare 16 GB with ZERO production seats stopped -- the h300 run's "stop BOTH GPU0 seats" was never necessary, it simply had a free card by coincidence. The first attempt OOM'd by 2.37 GiB at layer 64 of 65 with 3.57 GiB reserved-but-unallocated, which is fragmentation, and PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True closed it. post_quant.py now builds a missing output index from the safetensors headers. A sub-23 GB quant saves one bare shard with no index, and post_quant needs one; this has broken three separate rounds and been hand-fixed every time. The header is read by struct-unpacking the u64 length and parsing the JSON -- never safe_open, which mmaps the whole 22 GB shard and ENOMEMs on ZFS. Artifact verified: mixed-precision, 1968 tensors, 15 mtp, 333 visual, re:^mtp.* present in the ignore list (llm-compressor pruned it as always), preproc restored. Imatrix deferred per operator; the log confirms the usual uniform-MSE fallback, so this build stays apples-to-apples with heresy's PPL 6.910.
This commit is contained in:
@@ -122,6 +122,15 @@ _As of 2026-08-21 00:35 — **the Heretic-300 session, and its reversal** (see t
|
|||||||
|
|
||||||
- **🟢 HOMEPAGE — cleaned + themed (Australis Skyfall).** Three real defects fixed (UltraSeedbox on all tabs, Uptime Kuma double-rendered, fiction column counts), AI tab reordered by clickability, then themed from the operator's Skyfall handoff bundle with a background generated by **Arbo** (`t2i-ui-background`, job `13f0891f4e42`). ⚠ **After any recreate the tab bar/wallpaper/i18n vanish for up to ~an hour and then return on their own — do not chase it.** ⚠ CSS is served per-request: a theme change needs a **reload**, not a recreate, and candidate CSS can be injected live via Playwright for seconds-long iteration. → `persistent-memory.d/2026-08-19-homepage-skyfall-theme.md`
|
- **🟢 HOMEPAGE — cleaned + themed (Australis Skyfall).** Three real defects fixed (UltraSeedbox on all tabs, Uptime Kuma double-rendered, fiction column counts), AI tab reordered by clickability, then themed from the operator's Skyfall handoff bundle with a background generated by **Arbo** (`t2i-ui-background`, job `13f0891f4e42`). ⚠ **After any recreate the tab bar/wallpaper/i18n vanish for up to ~an hour and then return on their own — do not chase it.** ⚠ CSS is served per-request: a theme change needs a **reload**, not a recreate, and candidate CSS can be injected live via Playwright for seconds-long iteration. → `persistent-memory.d/2026-08-19-homepage-skyfall-theme.md`
|
||||||
|
|
||||||
|
- **🟡 ORCAROUTER QUANT BUILT AND VERIFIED 2026-08-21 01:24 — READY TO SERVE, NOT YET CUT OVER.** Replacement for the abandoned Cold-Fusion line. Source `orcarouter/Qwen3.8-27B-Uncensored` (rev `9878936b…`, 55.5 GB, gated but our token has access) → `/tank/aimodels/qwen38-27b-orcarouter-nvfp4-mixed` (23.4 GB). Chosen over `preetpatel/…-NVFP4` — full evaluation in `docs/pfi/gen-seat-candidate-eval-2026-08-21.md`.
|
||||||
|
- **★★ THE MTP HEAD IS ALREADY ROBINSON-ABLITERATED IN-BAND BY THE AUTHOR — the first third-party build we have seen get this right.** Operator asked whether we could apply the Robinson path to the head; the answer is that it is already applied. `compare_mtp_head.py` vs the verbatim base graft: **13 of 15 tensors byte-identical, exactly 2 differ** — `mtp.layers.0.{self_attn.o_proj,mlp.down_proj}.weight`, which are **precisely the two residual writers our own `abliterate.py` targets** (`EXPECT_MTP_WRITERS = 2`). Reverse-engineered from the weights alone (`bench/think-leak/mtp_delta.py`): **σ₂/σ₁ = 0.0164 on both** (rank-1 → a single-direction projection), **|cos| between the two recovered directions = 1.0000** (ONE shared direction, Robinson's `direction_scope=0` shape), ‖Δ‖/‖W‖ = 1.42%/1.41%, and **sink energy in dim 3994 = 0.0000%** — sink-clean outright, where Heretic's was 6.18% and our L35 0.094%. **Nothing for us to do on the head; only preserve it.** Verified preserved: the quant's MTP head is byte-identical to the bf16 source. This is the configuration the entire Cold-Fusion experiment was designed to test and never cleanly delivered.
|
||||||
|
- **★ FORMAT SCREEN PASSED — the new gate paid off immediately.** `think_prior.py` on the bf16 *before* any GPU time: **P(`<think>`) = 1.23e-06, rank 52.** Same probe: Cold-Fusion stock 0.1850, h300 0.2216, heresy <0.002. **~150,000× cleaner than the base we just abandoned.** Base is stock `Qwen/Qwen3.8-27B` + Arditi-et-al. abliteration — not a reasoning-compression finetune.
|
||||||
|
- **Artifact verified:** mixed-precision (group_0 FP8 W8A8 / group_1 NVFP4 W4A4), 1968 tensors, **15 mtp**, **333 visual**, `re:^mtp.*` present in the 303-entry ignore list, preproc restored, index built.
|
||||||
|
- **★ THE QUANT NEEDS ~17 GB, NOT A WHOLE CARD — and `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` is what makes it fit.** Ran entirely on **GPU1's spare 16 GB with ZERO production seats stopped**; the h300 run's "stop BOTH GPU0 seats" was never necessary (it had a free card by coincidence). First attempt OOM'd by 2.37 GiB at layer 64/65 with **3.57 GiB reserved-but-unallocated** — pure fragmentation; `expandable_segments` closed it. ~7 min per attempt.
|
||||||
|
- **★ FIXED IN THE SCRIPT, after biting THREE rounds:** `post_quant.py` crashed with `FileNotFoundError: model.safetensors.index.json` because a sub-23 GB quant saves a single bare shard with no index. It now **builds the index from the safetensors headers** (`struct`-unpack the u64 length + parse the JSON — never `safe_open`, which mmaps 22 GB and ENOMEMs on ZFS). Previously hand-fixed each time and never in the script.
|
||||||
|
- ⚠ Quant dir written root/0600 **again** — normalized to `llmuser 0664`. Fourth time this session; worth fixing in `quant_mixed_nvfp4.py`. ⚠ `compare_mtp_head.py` prints a hardcoded "already measured at 47.7% acceptance" line on an IDENTICAL verdict — **stale text, not a measurement of this head.** ⚠ Imatrix deferred by operator (park id 42); the log confirms the usual `imatrix_mse … Falling back to uniform MSE`, so this build stays apples-to-apples with heresy's PPL 6.910.
|
||||||
|
- **⏭ NEXT: the acceptance/PPL/surface gate, then the cutover.** Rollback stays `.env.bak-coldfusion-L35-20260820` → heresy.
|
||||||
|
|
||||||
- **⛔ COLD-FUSION ABANDONED — GEN SEAT ROLLED BACK TO `heresy` 2026-08-21 00:28 (operator directive).** The operator's call, made in advance of the result: *"If it's the base, we abandon h300 AND the base and chalk it up to a very powerful and useful learning experience. Our heretic methodology will definitely translate in the future."* The measurement came back **base**, so the condition fired.
|
- **⛔ COLD-FUSION ABANDONED — GEN SEAT ROLLED BACK TO `heresy` 2026-08-21 00:28 (operator directive).** The operator's call, made in advance of the result: *"If it's the base, we abandon h300 AND the base and chalk it up to a very powerful and useful learning experience. Our heretic methodology will definitely translate in the future."* The measurement came back **base**, so the condition fired.
|
||||||
- **LIVE GEN SEAT = `/tank/aimodels/qwen38-27b-heresy-nvfp4-mixed`** (`MuXodious/Qwen3.8-27B-absolute-heresy` through our mixed NVFP4+FP8 recipe). Restored from `.env.bak-coldfusion-L35-20260820`; the h300 env is preserved at `.env.bak-h300-abandoned-20260821`. Verified: healthy, **KV 403,065 tok / 1.54× — its exact documented baseline**, 7/7 aliases 200, vision intact (red circle / blue square / green rectangle).
|
- **LIVE GEN SEAT = `/tank/aimodels/qwen38-27b-heresy-nvfp4-mixed`** (`MuXodious/Qwen3.8-27B-absolute-heresy` through our mixed NVFP4+FP8 recipe). Restored from `.env.bak-coldfusion-L35-20260820`; the h300 env is preserved at `.env.bak-h300-abandoned-20260821`. Verified: healthy, **KV 403,065 tok / 1.54× — its exact documented baseline**, 7/7 aliases 200, vision intact (red circle / blue square / green rectangle).
|
||||||
- **★ The clincher: `<think>` is not even in heresy's top-20 first tokens (<0.002), against Cold-Fusion's 0.185.** That is a >100× gap — the two families are categorically different on this axis, and it is why no rollback *inside* Cold-Fusion (L35 or stock) would have helped.
|
- **★ The clincher: `<think>` is not even in heresy's top-20 first tokens (<0.002), against Cold-Fusion's 0.185.** That is a >100× gap — the two families are categorically different on this axis, and it is why no rollback *inside* Cold-Fusion (L35 or stock) would have helped.
|
||||||
|
|||||||
@@ -0,0 +1,44 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Is orcarouter's MTP-head edit a single-direction (Robinson-style) projection?
|
||||||
|
|
||||||
|
Robinson orthogonalizes a residual WRITER as W' = (I - d d^T) W, so
|
||||||
|
delta = W' - W = -d d^T W is EXACTLY RANK 1 with left singular vector d.
|
||||||
|
Two independent predictions follow, both checkable from the weights alone:
|
||||||
|
1. sigma_2/sigma_1 ~ 0 for each edited tensor
|
||||||
|
2. the d recovered from o_proj and from down_proj must AGREE (|cos| ~ 1),
|
||||||
|
because Robinson uses ONE shared direction
|
||||||
|
Then sink-screen the recovered d: Heretic's was 6.18% concentrated in dim 3994,
|
||||||
|
which is what made an in-band graft unsafe on that trunk. Ours want < 1%.
|
||||||
|
"""
|
||||||
|
import json, torch
|
||||||
|
from pathlib import Path
|
||||||
|
from safetensors import safe_open
|
||||||
|
|
||||||
|
CAND = Path("/tank/aimodels/qwen38-27b-orcarouter-bf16")
|
||||||
|
REF = Path("/tank/aimodels/qwen38-27b-uncensored-bf16")
|
||||||
|
KEYS = ["mtp.layers.0.self_attn.o_proj.weight", "mtp.layers.0.mlp.down_proj.weight"]
|
||||||
|
SINK_DIM = 3994
|
||||||
|
|
||||||
|
def get(d: Path, key: str):
|
||||||
|
idx = json.loads((d / "model.safetensors.index.json").read_text())["weight_map"]
|
||||||
|
with safe_open(d / idx[key], framework="pt") as f:
|
||||||
|
return f.get_tensor(key)
|
||||||
|
|
||||||
|
dirs = {}
|
||||||
|
for k in KEYS:
|
||||||
|
delta = (get(CAND, k).float() - get(REF, k).float())
|
||||||
|
U, S, Vh = torch.linalg.svd(delta, full_matrices=False)
|
||||||
|
ratio = (S[1] / S[0]).item()
|
||||||
|
d = U[:, 0] # residual-space direction (dim 5120)
|
||||||
|
dirs[k] = d
|
||||||
|
sink = (d[SINK_DIM] ** 2).item() / (d @ d).item()
|
||||||
|
print(f" {k.split('.',2)[2]:<28} shape={tuple(delta.shape)}")
|
||||||
|
print(f" sigma2/sigma1 = {ratio:.6f} <- rank-1 if ~0")
|
||||||
|
print(f" ||delta||/||W|| = {(delta.norm()/get(REF,k).float().norm()).item():.5f}")
|
||||||
|
print(f" sink energy dim {SINK_DIM} = {sink*100:.4f}% <- want <1%, Heretic's was 6.18%")
|
||||||
|
|
||||||
|
a, b = dirs[KEYS[0]], dirs[KEYS[1]]
|
||||||
|
cos = torch.abs(a @ b / (a.norm() * b.norm())).item()
|
||||||
|
print(f"\n |cos| between the two recovered directions = {cos:.4f} <- ~1 means ONE shared direction")
|
||||||
|
top = torch.topk(a.abs(), 5)
|
||||||
|
print(f" top-5 |d| coords (o_proj): {[(int(i), round(float(v),4)) for i, v in zip(top.indices, top.values)]}")
|
||||||
@@ -60,6 +60,36 @@ def main():
|
|||||||
src_idx = json.load(open(os.path.join(src, "model.safetensors.index.json")))
|
src_idx = json.load(open(os.path.join(src, "model.safetensors.index.json")))
|
||||||
mtp_keys = [k for k in src_idx["weight_map"] if k.startswith("mtp")]
|
mtp_keys = [k for k in src_idx["weight_map"] if k.startswith("mtp")]
|
||||||
out_idx_p = os.path.join(out, "model.safetensors.index.json")
|
out_idx_p = os.path.join(out, "model.safetensors.index.json")
|
||||||
|
|
||||||
|
# A quant that lands under ~23 GB fits in ONE shard, and llm-compressor then
|
||||||
|
# writes a bare `model.safetensors` with NO index at all. Every step below
|
||||||
|
# needs one, so build it here rather than failing.
|
||||||
|
#
|
||||||
|
# Read the safetensors HEADER directly -- the first 8 bytes are a
|
||||||
|
# little-endian u64 header length, followed by that many bytes of JSON
|
||||||
|
# keyed by tensor name. Do NOT use safe_open() for this: it mmaps the whole
|
||||||
|
# shard and ENOMEMs on ZFS against a 22 GB file.
|
||||||
|
#
|
||||||
|
# This has now bitten THREE separate rounds (2026-08-15, -08-20, -08-21),
|
||||||
|
# each time fixed by hand and never in the script. Fixed in the script.
|
||||||
|
if not os.path.exists(out_idx_p):
|
||||||
|
import struct
|
||||||
|
weight_map, total = {}, 0
|
||||||
|
for fn in sorted(f for f in os.listdir(out) if f.endswith(".safetensors")):
|
||||||
|
path = os.path.join(out, fn)
|
||||||
|
total += os.path.getsize(path)
|
||||||
|
with open(path, "rb") as fh:
|
||||||
|
n = struct.unpack("<Q", fh.read(8))[0]
|
||||||
|
header = json.loads(fh.read(n))
|
||||||
|
for key in header:
|
||||||
|
if key != "__metadata__":
|
||||||
|
weight_map[key] = fn
|
||||||
|
json.dump({"metadata": {"total_size": total}, "weight_map": weight_map},
|
||||||
|
open(out_idx_p, "w"), indent=2)
|
||||||
|
print(f"BUILT missing output index from safetensors headers: "
|
||||||
|
f"{len(weight_map)} tensors across "
|
||||||
|
f"{len(set(weight_map.values()))} shard(s), {total/1e9:.1f} GB")
|
||||||
|
|
||||||
out_idx = json.load(open(out_idx_p))
|
out_idx = json.load(open(out_idx_p))
|
||||||
added = 0
|
added = 0
|
||||||
for k in mtp_keys:
|
for k in mtp_keys:
|
||||||
|
|||||||
Reference in New Issue
Block a user