From 4b6daadb16309257a8b3ef8e0a1b96b467ec9421 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Mon, 17 Aug 2026 20:19:18 -0700 Subject: [PATCH] memory: operator confirms the heresy gen seat working well in real use Records the one signal the synthetic gates cannot provide -- multi-turn degeneration is stochastic and invisible to probes, and four synthetic tests once validated three non-fixes on this exact seat. Not yet the 60k-token bar the prior seat cleared, so the rollback weights stay in place. --- persistent-memory.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/persistent-memory.md b/persistent-memory.md index d5c2d02..371e4f0 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -111,7 +111,7 @@ no longer deployed sidecars here. See Recent decisions.) _As of 2026-08-17 โ€” **quiet; gen-seat degeneration saga CLOSED.** Gen seat resolved and coherent through 60k tokens. Lobe Chat stood up, LiteLLM upgraded + spend-log capped. Two peer research loops (dvalin/bil) closed. No blocking work in flight._ -- **๐ŸŸข GEN SEAT โ€” SWAPPED to `absolute-heresy` 2026-08-17 (validated, promoted).** Live gen = `/tank/aimodels/qwen38-27b-heresy-nvfp4-mixed` โ€” **MuXodious/Qwen3.8-27B-absolute-heresy** (Heretic v1.4.0 + **SOMPOA**, trial T377, pin `c2374593`) put through our own mixed NVFP4+FP8 recipe. Chosen because it beats the incumbent on **both** axes at once: author refusals 2/101 vs 12/100, first-token KL 0.0759 vs 0.1191. **Gate (probe :8017, pinned nightly, seat-matched flags): MTP 47.2% (inc. 48.2%), decode 103.5 tok/s (96.4), prefill 6618/5403 @6.7k/27k (6334/5085), PPL 6.910 (7.059 โ€” 2.1% BETTER), surface 6/6, abliteration 4/4, and 0/55 refusals on our battery-instruct arm with ZERO EMPTY (no catatonia).** โš  speed deltas are **image-confounded** (probe on the pinned nightly, incumbent numbers from an earlier image) โ€” read as "not worse", not a clean win. All 7 LiteLLM aliases verified end-to-end; GPU0 at 91.3/97.9 GB with meromero healthy (more headroom than the old build's 96.8). โš  **RC1, 2 days old, ~348 downloads** โ€” watch real multi-turn use. **ROLLBACK:** `sudo cp /opt/docker/compose/gen-seat/.env.bak-heresy-20260817 /opt/docker/compose/gen-seat/.env && cd /opt/docker/compose/gen-seat && sudo docker compose up -d vllm-gen`; incumbent weights UNTOUCHED at `qwen38-27b-uncensored-nvfp4-mixed` โ€” **do NOT delete** until this holds. Runbook `services/gen-seat-mixed-quant/RUNBOOK-heresy-swap.md`. +- **๐ŸŸข GEN SEAT โ€” SWAPPED to `absolute-heresy` 2026-08-17 (validated, promoted).** Live gen = `/tank/aimodels/qwen38-27b-heresy-nvfp4-mixed` โ€” **MuXodious/Qwen3.8-27B-absolute-heresy** (Heretic v1.4.0 + **SOMPOA**, trial T377, pin `c2374593`) put through our own mixed NVFP4+FP8 recipe. Chosen because it beats the incumbent on **both** axes at once: author refusals 2/101 vs 12/100, first-token KL 0.0759 vs 0.1191. **Gate (probe :8017, pinned nightly, seat-matched flags): MTP 47.2% (inc. 48.2%), decode 103.5 tok/s (96.4), prefill 6618/5403 @6.7k/27k (6334/5085), PPL 6.910 (7.059 โ€” 2.1% BETTER), surface 6/6, abliteration 4/4, and 0/55 refusals on our battery-instruct arm with ZERO EMPTY (no catatonia).** โš  speed deltas are **image-confounded** (probe on the pinned nightly, incumbent numbers from an earlier image) โ€” read as "not worse", not a clean win. All 7 LiteLLM aliases verified end-to-end; GPU0 at 91.3/97.9 GB with meromero healthy (more headroom than the old build's 96.8). โš  **RC1, 2 days old, ~348 downloads.** **Operator-confirmed "working very well" in real use 2026-08-17**, same evening as the cutover โ€” the signal the synthetic gates structurally cannot give (multi-turn degeneration is stochastic; four synthetic tests once validated three non-fixes). Not yet the 60k-token bar the prior seat cleared, so **keep watching and do NOT delete the rollback weights yet**. **ROLLBACK:** `sudo cp /opt/docker/compose/gen-seat/.env.bak-heresy-20260817 /opt/docker/compose/gen-seat/.env && cd /opt/docker/compose/gen-seat && sudo docker compose up -d vllm-gen`; incumbent weights UNTOUCHED at `qwen38-27b-uncensored-nvfp4-mixed` โ€” **do NOT delete** until this holds. Runbook `services/gen-seat-mixed-quant/RUNBOOK-heresy-swap.md`. - **๐ŸŸข PRIOR GEN SEAT โ€” RESOLVED 2026-08-17 (the multi-day degeneration saga); now the ROLLBACK target.** Was the in-house **JonathanColetti/Heretic mixed NVFP4+FP8 build** (`/tank/aimodels/qwen38-27b-uncensored-nvfp4-mixed`, FP8 attention) on **vLLM nightly PINNED** `vllm/vllm-openai:nightly-311b3513โ€ฆ` (`v0.27.2rc1.dev150`, carries #51113 mamba fix), **MTP ON, prefix-caching ON**. Operator-confirmed **coherent through 60k tokens** real multi-turn. Root cause = TWO compounding real causes: (1) genuine vLLM `qwen3_5_mtp`ร—GDN partial-accept bug (#51113, architectural across vLLM/SGLang/llama.cpp, fixed by nightly), and (2) **AEON's full W4A4** being lowest-fidelity on the known activation gradient (W4A4 < W4+FP8 < W4+bf16) โ†’ ~15-20% stochastic degeneration on top of (1). **AEON PURGED** (re-pullable `sakamakismile/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4`). Full lesson `docs/pfi/model-quantization-playbook.md` ยง3.8. (Superseded as primary by `absolute-heresy` 2026-08-17.) โš  **pinned nightly is bleeding-edge โ€” move to a stable release once #51113 ships in one (the standing follow-up).** 7 aliases (gen/gen-reasoning/summarizer/-large/classifier/image-judge/qwen-image-bench) all route here. Seat carries `--default-chat-template-kwargs '{"reasoning_effort":"medium"}'` (per-request overridable, affects gen-reasoning only). Commits `d28a371`,`2f2bbce`,`2185964`.