memory: gemma4 CoT test on v0.27.1 blocked by config incompatibility (reverted)

Tried serving the MeroMero NVFP4A16 quant on vllm/vllm-openai:v0.27.1 to test the
per-request enable_thinking hypothesis. Couldn't even load it:

  - v0.27.1's stricter transformers raised AmbiguousGlobalPerLayerAttributeError
    on the Gemma-4 config's head_dim (now a per-layer attribute).
  - Setting allow_global_per_layer_attribute_access on text_config downgraded that
    to a warning, but then gemma4.py load_weights asserted:
    "Attempted to load weight (512) into parameter (256)". Gemma-4-31B is
    genuinely heterogeneous -- some layers have head_dim 512, not a uniform 256 --
    so forcing the global value built wrong-shaped parameters. The guard was right.

So the MeroMero quant's config was authored for v0.24.0's Gemma4 loader and cannot
load on v0.27.x without a config migration or a re-quant against newer transformers.
The per-request-kwargs hypothesis was never reachable -- the model wouldn't load.

Consequence worth carrying: the eventual gen-seat move to v0.27.2 stable must
re-verify config-compat for any Gemma-4 seat of this vintage.

Fully reverted: config.json restored (flags removed), image back to latest
(v0.24.0), gateway char-rp-reasoning removed, char-rp prose verified. char-rp stays
on v0.24.0; MeroMero CoT remains undelivered. For RP-with-CoT use gen-reasoning, or
re-quant MeroMero against v0.27.x (real work, unproven payoff).

persistent-memory.md

@@ -122,6 +122,8 @@ _As of 2026-08-21 00:35 — **the Heretic-300 session, and its reversal** (see t
 
 - **🟢 HOMEPAGE — cleaned + themed (Australis Skyfall).** Three real defects fixed (UltraSeedbox on all tabs, Uptime Kuma double-rendered, fiction column counts), AI tab reordered by clickability, then themed from the operator's Skyfall handoff bundle with a background generated by **Arbo** (`t2i-ui-background`, job `13f0891f4e42`). ⚠ **After any recreate the tab bar/wallpaper/i18n vanish for up to ~an hour and then return on their own — do not chase it.** ⚠ CSS is served per-request: a theme change needs a **reload**, not a recreate, and candidate CSS can be injected live via Playwright for seconds-long iteration. → `persistent-memory.d/2026-08-19-homepage-skyfall-theme.md`
 
+- **⛔ RESULT 2026-08-21 — the gemma4 CoT test on a STABLE (v0.27.1) is BLOCKED by a config incompatibility, NOT the parser.** Tried serving the MeroMero NVFP4A16 quant on `vllm/vllm-openai:v0.27.1`. Two-stage failure: (1) v0.27.1's stricter transformers raised `AmbiguousGlobalPerLayerAttributeError: 'head_dim' is per-layer` on the Gemma-4 config; setting `allow_global_per_layer_attribute_access:true` on `text_config` downgraded it to a warning BUT (2) then `gemma4.py load_weights` crashed with **`AssertionError: load weight (512) into parameter (256)`** — **Gemma-4-31B is genuinely HETEROGENEOUS (some layers head_dim 512, not a uniform 256)**, so forcing the global value built wrong-shaped params. The transformers guard was CORRECT; there is no safe override. **The MeroMero quant's config was authored for v0.24.0's Gemma4 loader and cannot load on v0.27.x without a config migration (proper per_layer_config) or a re-quant against the newer transformers.** ⚠ **This also means the eventual gen-seat move to v0.27.2 stable must re-verify any Gemma-4 seat's config-compat** — the transformers heterogeneity change affects all Gemma-4 quants of this vintage. **FULLY REVERTED:** config.json restored (flags removed), compose + image back to `latest` (v0.24.0), gateway char-rp-reasoning removed, char-rp prose verified on v0.24.0. Net: char-rp stays pinned to v0.24.0; MeroMero CoT remains undelivered. **The per-request-kwargs hypothesis was never even reachable** — couldn't load the model to test it. **For RP-with-CoT: gen-reasoning (works now) or a re-quant of MeroMero against v0.27.x transformers (real work, unproven payoff).**
+
 - **⚠️ CORRECTED 2026-08-21 — MeroMero-v2 CoT: NOT a hard wall, and NOT MeroMero-specific. My first conclusion ("gemma4 parser is process-wide") was WRONG.** Read the actual code, not the stale compose comment. **The real mechanism (gemma4-GENERAL, applies to any gemma4 finetune on this template family):** thinking is a **per-request** template toggle — `chat_template.jinja:347-352` emits the generation prompt `<|turn>model\n`, and **only when `enable_thinking` is false** does it prefill an empty `<|channel>thought\n<channel|>` to SUPPRESS thinking; `enable_thinking:true` omits the prefill so the model is free to open a real `<|channel>thought…<channel|>` block. The vLLM parser (`vllm/reasoning/gemma4_utils.py:parse_thinking_output`) **splits on `<|channel>`/`<channel|>` tag PRESENCE — "works with or without enable_thinking," NOT a process-wide flag.** The stale compose comment I trusted cited an OLD parser API (`vllm/parser/gemma4.py:439`) that this container does not run. **So there is no architectural blocker; the two-served-name gen pattern SHOULD work.** **What actually failed my test:** meromero runs `vllm/vllm-openai:latest` (v0.24.0); per-request `chat_template_kwargs.enable_thinking:true` produced no thinking on it, whereas the **gen seat's pinned nightly demonstrably applies per-request `chat_template_kwargs`** (gen-reasoning works). So the practical block is a **vLLM-version / per-request-plumbing issue on v0.24.0**, not the model and not the architecture — and it would hit ANY gemma4 finetune served on that image the same way. **UNVERIFIED FIX (needs a GPU window): re-serve meromero on the nightly image + no process default + per-request enable_thinking; likely yields clean split CoT.** Currently REVERTED to known-good (char-rp prose, process default false, single served-name). ⚠ Kept `MEROMERO_GPU_MEM_UTIL` 0.52→0.51 (0.52 no longer boots next to the bigger orcarouter gen; free 49.02 < 49.38 GiB; 0.51 = KV 2.00× @ 262K).
 
 - **🟢 PEN-TEST SEAT — M.O.G.-SEC LIVE on ana-ml2 GPU1 :8019 (2026-08-21 ~02:35, autonomous overnight run under operator's "full autonomy" grant).** `stacks/mog-sec` serves `/tank/aimodels/mog-sec-27b-nvfp4-mixed` (23.4 GB, our mixed NVFP4+FP8 quant of `Blackfrost-Research/M.O.G.-SEC-27B-1M-CTX-BF16`, rev `deede677`) in the **retired fablefusion-charrp-probe slot**. Aliases: **`mog-sec`** (non-thinking, presence_penalty 0.0 for code) + **`mog-sec-reasoning`** (thinking); served-names `mog-sec-27b`/`-thinking`. **Gates all pass:** KV **418,218 tok / 1.60× @ 262K**; surface **6/6**; MTP **55.3% @ 111.7 tok/s**; format screen **1.11e-05** (rank 196, no `<think>` leak); **vision 7/3/1**; **capability 4/4** (SQL-inj exfil, pentest phasing, port scanner, Kerberoasting — it delivers, doesn't refuse). **★ Chose the BF16 + self-quant over the pre-made ModelOpt NVFP4**, which was disqualified on W4A4 (4-bit activations, AEON mode) + zero MTP + the self-defeating W4A4-on-1M-context. **⚠ CONTEXT IS 262K NATIVE, NOT the card's 1M** — 1M needs YaRN (absent from config) + the SGLang/DFlash2 path the repo ships, neither our vLLM surface; a real 1M seat is a separate SGLang project. **MTP head is a base graft** (identical to stock), not in-band abliterated like orcarouter's. Provenance `mog-sec-27b-nvfp4-mixed.PROVENANCE.txt`; eval `docs/pfi/gen-seat-candidate-eval-2026-08-21.md`. **RETIRED with it:** `char-rp-reasoning` + `char-rp-fable` LiteLLM aliases (zero traffic, pointed at the dead fable :8019; now 404). RP-reasoning's real home is `darkscarlett-charrp-reasoning` (:8018, compose-down, weights intact) if ever wanted. ⚠ brand-new model (0 downloads at pull, org-name mismatch Research-vs-AI) — card claims are claims.