From 14ff4a3f570a846280757f3feb6644aeb24c53e3 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Mon, 24 Aug 2026 13:52:00 -0700 Subject: [PATCH] docs(gemma4-charrp): stage two abliterated trainee bases; record the endemic stale-template trap MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The operator directed that the ERP/RP trainee base be a low-damage abliterated instruct build rather than the stock checkpoint. Two are now staged under /tank/aimodels/, both BF16, both unquantized, both matching upstream's 51.61 GB / 25.8B shape with only transformers_version differing in config: gemma4-26b-a4b-it-heretic-bf16 llmfan46, Heretic v1.2.0 ARA, KL 0.1237, refusals 3/100 gemma4-26b-a4b-it-abliterated-bf16 TrevorJS, KL 0.09, 1/100 effective and 5/686 cross-dataset "Low damage" was treated as a measurable claim rather than a description: the field spreads from KL 0.09 to 0.4118 and the table is in the README so the next choice is made on numbers. Fleet anchor for reading them — our own abliteration work found Heretic at KL 0.12 preserved the MTP head at 83.7% acceptance, so both staged builds sit inside an already-validated band rather than past it. huihui-ai is rejected despite its reputation: no published metrics, its own card calls the method a crude proof-of-concept, it abliterates both thinking and non-thinking modes, and its parameter count runs ~738M over upstream. The operator's independent read matched. The more durable finding is the chat template. NOT ONE third-party Gemma-4 derivative pulled here ships upstream's — three independent repos carry the identical stale 266-line file (sha 58c66fdee4afa297), llmfan46 carries a third 365-line variant, and only the RedHatAI NVFP4 build matches upstream's 6a1015c47ccfcfa6. It propagated through the ecosystem rather than one packager slipping, and it is now recorded as a class rather than as the single incident that surfaced it during the A16 control staging. That matters twice over and silently both times: serving a mismatched template renders a different prompt, which is why production pins it; and training through `base/chat_template.jinja` means training on a different prompt format than production serves — train/serve skew with no error, presenting as a tuning failure. brokkr-smithy-dev has been warned on the training side while the harness contract is still early enough to amend. --- stacks/gemma4-charrp/README.md | 57 +++++++++++++++++++++++++++++++++- 1 file changed, 56 insertions(+), 1 deletion(-) diff --git a/stacks/gemma4-charrp/README.md b/stacks/gemma4-charrp/README.md index 445492c..6380a8d 100644 --- a/stacks/gemma4-charrp/README.md +++ b/stacks/gemma4-charrp/README.md @@ -15,13 +15,68 @@ up, and it cost a peer a mis-attributed benchmark before it was noticed. | path | size | what it is | |---|---|---| -| `gemma4-26b-a4b-it-bf16` | 49 GB | **QLoRA tuning base.** BF16, unquantized. Cannot be served here — 48.10 GiB of weights against ~49 GiB of free GPU0 leaves nothing for KV cache. | +| `gemma4-26b-a4b-it-bf16` | 49 GB | **Stock BF16.** Unquantized. Cannot be served here — 48.10 GiB of weights against ~49 GiB of free GPU0 leaves nothing for KV cache. Its `chat_template.jinja` is the canonical upstream one; see below. | +| `gemma4-26b-a4b-it-heretic-bf16` | 49 GB | **QLoRA trainee base** (operator's pick, 2026-08-24). llmfan46, Heretic v1.2.0 ARA, KL 0.1237, refusals 3/100. | +| `gemma4-26b-a4b-it-abliterated-bf16` | 49 GB | **Trainee alternate.** TrevorJS, KL 0.09, refusals 1/100 effective and 5/686 cross-dataset — lowest measured damage of the field. | | `gemma4-26b-a4b-it-nvfp4` | 16 GB | **What is served.** RedHatAI, compressed-tensors, W4**A4**. | | `gemma4-26b-a4b-it-nvfp4a16` | 17 GB | **Activation-axis control**, for benching only. prithivMLmods, compressed-tensors, W4**A16**. | All under `/tank/aimodels/`, pulled by the `gemma4-26b-*-dl.py` scripts beside them with revisions pinned. +### ⚠ Third-party Gemma-4 builds ship STALE CHAT TEMPLATES — this is endemic + +Verified by hash on 2026-08-24 across every third-party derivative pulled here. +**Not one of them ships upstream's template:** + +| build | lines | sha256 (normalised) | +|---|---|---| +| upstream `google/gemma-4-26B-A4B-it` | 390 | `6a1015c47ccfcfa6` | +| RedHatAI NVFP4 (what is served) | 389 | `6a1015c47ccfcfa6` — the only match | +| llmfan46 heretic | 365 | `0a52be69cda5ab8a` | +| TrevorJS abliterated | 266 | `58c66fdee4afa297` | +| jenerallee78 abliterated | 266 | `58c66fdee4afa297` | +| prithivMLmods NVFP4A16 | 266 | `58c66fdee4afa297` | + +Three independent repos carrying the identical stale 266-line file means it +propagated through the ecosystem, not that one packager slipped. + +**Consequences differ by use and both are silent:** + +- **Serving** — a different template renders a different prompt. This is why the + production compose pins the template explicitly. +- **Training** — if the harness renders examples through `base/chat_template.jinja`, + you train on a different prompt format than production serves. Train/serve + skew, no error, and it presents as a tuning failure. + +For both, point at the upstream file: +`/tank/aimodels/gemma4-26b-a4b-it-bf16/chat_template.jinja`. + +### Choosing an abliterated base — compare on published damage, not on names + +"Low damage" has a measurable proxy and the field spreads widely on it: + +| build | method | KL | refusals | +|---|---|---|---| +| TrevorJS | ARA-family | **0.09** | 1/100 effective, 5/686 cross-dataset, manually audited | +| llmfan46 | Heretic v1.2.0 ARA | 0.1237 | 3/100 | +| jenerallee78 | ARA 2-pass | 0.1299 | 7.7% StrongREJECT | +| huihui-ai | remove-refusals-with-transformers | none published | none published | +| trohrbaugh/heretic-ara | — | 0.2999 | 31.4% | +| coder3101/heretic | — | 0.4118 | 15.8% | + +Fleet anchor for reading those numbers: our own abliteration work found **Heretic +at KL 0.12 preserved the MTP head at 83.7% acceptance**. Both staged builds sit +at or below that, so neither is an extrapolation past what has been measured +here. + +huihui-ai is rejected on this stack despite being the best-known abliteration +house: no published metrics, its card describes the method as "a crude, +proof-of-concept implementation", it states both thinking and non-thinking modes +were "completely abliterated", and its parameter count is 26,544,131,376 against +upstream's 25,805,936,206 — roughly 738M unexplained extra. The operator's +independent read is the same ("huihui produces garbage"). + ## GPU0 is shared and the budgets must sum under ~0.92 `vllm-gen` runs at `--gpu-memory-utilization 0.43` but actually holds ~45.6 GiB