test(gen-seat): orcarouter passes every gate — in-band MTP head delivers +11 points

Gates run against the live seat while the operator tested in parallel.

  <think> leak (n=30, 4 prompt types + multi-turn)   0/30, 0 empty
  MTP acceptance                                     58.4% @ 117.11 tok/s median
  surface                                            6/6
  abliteration survival                              4/4 compliance
  deterministic quality gens                         coherent and correct
  PPL                                                still blocked

For scale on the leak gate, the abandoned h300 build scored 8/30 on this exact
instrument, and its abliteration-survival samples had 2 of 4 open with
"<think>Ok, let's figure this out:". Orcarouter has none.

The headline is MTP acceptance. 58.4% against heresy's byte-identical base head
at 47.2% is +11 points, and it sits level with our own in-band L35 at 59.1%. That
is the additive in-band-vs-graft delta the entire Cold-Fusion experiment was built
to measure and never cleanly delivered -- orcarouter handed it over for free
because the author had already done the Robinson edit on the head.

Surface 6/6 covers plain chat, vision, tool calling, the thinking split, a
36,042-token long-context retrieval, and streaming.

PPL remains blocked on a spec-decode-free probe seat: it needs ~22 GB and GPU1 has
~16 GB free. Comparison target is heresy at 6.910.
This commit is contained in:
vh
2026-08-21 01:48:09 -07:00
parent ba53c30192
commit f85d102813
3 changed files with 69 additions and 1 deletions
+1 -1
View File
@@ -122,7 +122,7 @@ _As of 2026-08-21 00:35 — **the Heretic-300 session, and its reversal** (see t
- **🟢 HOMEPAGE — cleaned + themed (Australis Skyfall).** Three real defects fixed (UltraSeedbox on all tabs, Uptime Kuma double-rendered, fiction column counts), AI tab reordered by clickability, then themed from the operator's Skyfall handoff bundle with a background generated by **Arbo** (`t2i-ui-background`, job `13f0891f4e42`). ⚠ **After any recreate the tab bar/wallpaper/i18n vanish for up to ~an hour and then return on their own — do not chase it.** ⚠ CSS is served per-request: a theme change needs a **reload**, not a recreate, and candidate CSS can be injected live via Playwright for seconds-long iteration. → `persistent-memory.d/2026-08-19-homepage-skyfall-theme.md`
- **🟢 GEN SEAT — LIVE = ORCAROUTER (cut over 2026-08-21 01:35, operator directive: seat first so he can test while gates run).** `GEN_MODEL=/tank/aimodels/qwen38-27b-orcarouter-nvfp4-mixed`; healthy in ~4 min; **KV 401,550 tok / 1.53×**; MTP drafter detected and wired (embedding + lm_head shared); **7/7 aliases 200**; **vision correct**; **`<think>` not in the top-20 first tokens** — the Cold-Fusion failure mode is absent on the live seat. Live decode observed at **102-133 tok/s** under load. **ROLLBACK:** `.env.bak-heresy-restored-20260821` → heresy (one line). ⏳ Full gates (leak n=30, MTP acceptance, surface 6/6, abliteration survival) were still running at the time of writing; PPL remains blocked on a spec-decode-free probe seat (GPU1 has ~16 GB free, a probe needs ~22 GB). Replacement for the abandoned Cold-Fusion line. Source `orcarouter/Qwen3.8-27B-Uncensored` (rev `9878936b…`, 55.5 GB, gated but our token has access) → `/tank/aimodels/qwen38-27b-orcarouter-nvfp4-mixed` (23.4 GB). Chosen over `preetpatel/…-NVFP4` — full evaluation in `docs/pfi/gen-seat-candidate-eval-2026-08-21.md`.
- **🟢 GEN SEAT — LIVE = ORCAROUTER (cut over 2026-08-21 01:35, operator directive: seat first so he can test while gates run).** `GEN_MODEL=/tank/aimodels/qwen38-27b-orcarouter-nvfp4-mixed`; healthy in ~4 min; **KV 401,550 tok / 1.53×**; MTP drafter detected and wired (embedding + lm_head shared); **7/7 aliases 200**; **vision correct**; **`<think>` not in the top-20 first tokens** — the Cold-Fusion failure mode is absent on the live seat. Live decode observed at **102-133 tok/s** under load. **ROLLBACK:** `.env.bak-heresy-restored-20260821` → heresy (one line). **✅ ALL GATES PASS** (run against the live seat while the operator tested in parallel): **`<think>` leak 0/30, 0 empty** (h300 scored 8/30 on this instrument); **MTP acceptance 58.4% median @ 117.11 tok/s**; **surface 6/6** (chat, vision, tool-calling, thinking split, 36k-token long-context retrieval, streaming); **abliteration survival 4/4** with no `<think>` leak in any sample (h300's equivalent eval had 2 of 4 open with `<think>Ok, let's figure this out:`); deterministic quality gens coherent. **★ THE IN-BAND MTP HEAD PAID OFF: 58.4% vs heresy's byte-identical base head at 47.2% — +11 points**, and level with our own in-band L35 at 59.1%. That is the additive delta the entire Cold-Fusion experiment was built to measure and never cleanly delivered; orcarouter handed it over for free. ⏳ PPL still blocked on a spec-decode-free probe seat (needs ~22 GB, GPU1 has ~16 GB free) — comparison target heresy 6.910. Replacement for the abandoned Cold-Fusion line. Source `orcarouter/Qwen3.8-27B-Uncensored` (rev `9878936b…`, 55.5 GB, gated but our token has access) → `/tank/aimodels/qwen38-27b-orcarouter-nvfp4-mixed` (23.4 GB). Chosen over `preetpatel/…-NVFP4` — full evaluation in `docs/pfi/gen-seat-candidate-eval-2026-08-21.md`.
- **★★ THE MTP HEAD IS ALREADY ROBINSON-ABLITERATED IN-BAND BY THE AUTHOR — the first third-party build we have seen get this right.** Operator asked whether we could apply the Robinson path to the head; the answer is that it is already applied. `compare_mtp_head.py` vs the verbatim base graft: **13 of 15 tensors byte-identical, exactly 2 differ** — `mtp.layers.0.{self_attn.o_proj,mlp.down_proj}.weight`, which are **precisely the two residual writers our own `abliterate.py` targets** (`EXPECT_MTP_WRITERS = 2`). Reverse-engineered from the weights alone (`bench/think-leak/mtp_delta.py`): **σ₂/σ₁ = 0.0164 on both** (rank-1 → a single-direction projection), **|cos| between the two recovered directions = 1.0000** (ONE shared direction, Robinson's `direction_scope=0` shape), ‖Δ‖/‖W‖ = 1.42%/1.41%, and **sink energy in dim 3994 = 0.0000%** — sink-clean outright, where Heretic's was 6.18% and our L35 0.094%. **Nothing for us to do on the head; only preserve it.** Verified preserved: the quant's MTP head is byte-identical to the bf16 source. This is the configuration the entire Cold-Fusion experiment was designed to test and never cleanly delivered.
- **★ FORMAT SCREEN PASSED — the new gate paid off immediately.** `think_prior.py` on the bf16 *before* any GPU time: **P(`<think>`) = 1.23e-06, rank 52.** Same probe: Cold-Fusion stock 0.1850, h300 0.2216, heresy <0.002. **~150,000× cleaner than the base we just abandoned.** Base is stock `Qwen/Qwen3.8-27B` + Arditi-et-al. abliteration — not a reasoning-compression finetune.
- **Artifact verified:** mixed-precision (group_0 FP8 W8A8 / group_1 NVFP4 W4A4), 1968 tensors, **15 mtp**, **333 visual**, `re:^mtp.*` present in the 303-entry ignore list, preproc restored, index built.