vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(gemma4-charrp): stage two abliterated trainee bases; record the endemic stale-template trap

Browse Source 
The operator directed that the ERP/RP trainee base be a low-damage abliterated
instruct build rather than the stock checkpoint. Two are now staged under
/tank/aimodels/, both BF16, both unquantized, both matching upstream's 51.61 GB
/ 25.8B shape with only transformers_version differing in config:

  gemma4-26b-a4b-it-heretic-bf16       llmfan46, Heretic v1.2.0 ARA, KL 0.1237, refusals 3/100
  gemma4-26b-a4b-it-abliterated-bf16   TrevorJS, KL 0.09, 1/100 effective and 5/686 cross-dataset

"Low damage" was treated as a measurable claim rather than a description: the
field spreads from KL 0.09 to 0.4118 and the table is in the README so the next
choice is made on numbers. Fleet anchor for reading them — our own abliteration
work found Heretic at KL 0.12 preserved the MTP head at 83.7% acceptance, so
both staged builds sit inside an already-validated band rather than past it.
huihui-ai is rejected despite its reputation: no published metrics, its own card
calls the method a crude proof-of-concept, it abliterates both thinking and
non-thinking modes, and its parameter count runs ~738M over upstream. The
operator's independent read matched.

The more durable finding is the chat template. NOT ONE third-party Gemma-4
derivative pulled here ships upstream's — three independent repos carry the
identical stale 266-line file (sha 58c66fdee4afa297), llmfan46 carries a third
365-line variant, and only the RedHatAI NVFP4 build matches upstream's
6a1015c47ccfcfa6. It propagated through the ecosystem rather than one packager
slipping, and it is now recorded as a class rather than as the single incident
that surfaced it during the A16 control staging.

That matters twice over and silently both times: serving a mismatched template
renders a different prompt, which is why production pins it; and training
through `base/chat_template.jinja` means training on a different prompt format
than production serves — train/serve skew with no error, presenting as a tuning
failure. brokkr-smithy-dev has been warned on the training side while the
harness contract is still early enough to amend.
This commit is contained in:
vh
2026-08-24 13:52:00 -07:00
parent 8d6a9390de
commit 14ff4a3f57
1 changed files with 56 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
stacks/gemma4-charrp/README.md
+56 -1
View File
@@ -15,13 +15,68 @@ up, and it cost a peer a mis-attributed benchmark before it was noticed.
| path | size | what it is |
|---|---|---|
| `gemma4-26b-a4b-it-bf16` | 49 GB | **QLoRA tuning base.** BF16, unquantized. Cannot be served here — 48.10 GiB of weights against ~49 GiB of free GPU0 leaves nothing for KV cache. |
| `gemma4-26b-a4b-it-bf16` | 49 GB | **Stock BF16.** Unquantized. Cannot be served here — 48.10 GiB of weights against ~49 GiB of free GPU0 leaves nothing for KV cache. Its `chat_template.jinja` is the canonical upstream one; see below. |
| `gemma4-26b-a4b-it-heretic-bf16` | 49 GB | **QLoRA trainee base** (operator's pick, 2026-08-24). llmfan46, Heretic v1.2.0 ARA, KL 0.1237, refusals 3/100. |
| `gemma4-26b-a4b-it-abliterated-bf16` | 49 GB | **Trainee alternate.** TrevorJS, KL 0.09, refusals 1/100 effective and 5/686 cross-dataset — lowest measured damage of the field. |
| `gemma4-26b-a4b-it-nvfp4` | 16 GB | **What is served.** RedHatAI, compressed-tensors, W4**A4**. |
| `gemma4-26b-a4b-it-nvfp4a16` | 17 GB | **Activation-axis control**, for benching only. prithivMLmods, compressed-tensors, W4**A16**. |
All under `/tank/aimodels/`, pulled by the `gemma4-26b-*-dl.py` scripts beside
them with revisions pinned.
### ⚠ Third-party Gemma-4 builds ship STALE CHAT TEMPLATES — this is endemic
Verified by hash on 2026-08-24 across every third-party derivative pulled here.
**Not one of them ships upstream's template:**
| build | lines | sha256 (normalised) |
|---|---|---|
| upstream `google/gemma-4-26B-A4B-it` | 390 | `6a1015c47ccfcfa6` |
| RedHatAI NVFP4 (what is served) | 389 | `6a1015c47ccfcfa6` — the only match |
| llmfan46 heretic | 365 | `0a52be69cda5ab8a` |
| TrevorJS abliterated | 266 | `58c66fdee4afa297` |
| jenerallee78 abliterated | 266 | `58c66fdee4afa297` |
| prithivMLmods NVFP4A16 | 266 | `58c66fdee4afa297` |
Three independent repos carrying the identical stale 266-line file means it
propagated through the ecosystem, not that one packager slipped.
**Consequences differ by use and both are silent:**
- **Serving** — a different template renders a different prompt. This is why the
production compose pins the template explicitly.
- **Training** — if the harness renders examples through `base/chat_template.jinja`,
you train on a different prompt format than production serves. Train/serve
skew, no error, and it presents as a tuning failure.
For both, point at the upstream file:
`/tank/aimodels/gemma4-26b-a4b-it-bf16/chat_template.jinja`.
### Choosing an abliterated base — compare on published damage, not on names
"Low damage" has a measurable proxy and the field spreads widely on it:
| build | method | KL | refusals |
|---|---|---|---|
| TrevorJS | ARA-family | **0.09** | 1/100 effective, 5/686 cross-dataset, manually audited |
| llmfan46 | Heretic v1.2.0 ARA | 0.1237 | 3/100 |
| jenerallee78 | ARA 2-pass | 0.1299 | 7.7% StrongREJECT |
| huihui-ai | remove-refusals-with-transformers | none published | none published |
| trohrbaugh/heretic-ara | — | 0.2999 | 31.4% |
| coder3101/heretic | — | 0.4118 | 15.8% |
Fleet anchor for reading those numbers: our own abliteration work found **Heretic
at KL 0.12 preserved the MTP head at 83.7% acceptance**. Both staged builds sit
at or below that, so neither is an extrapolation past what has been measured
here.
huihui-ai is rejected on this stack despite being the best-known abliteration
house: no published metrics, its card describes the method as "a crude,
proof-of-concept implementation", it states both thinking and non-thinking modes
were "completely abliterated", and its parameter count is 26,544,131,376 against
upstream's 25,805,936,206 — roughly 738M unexplained extra. The operator's
independent read is the same ("huihui produces garbage").
## GPU0 is shared and the budgets must sum under ~0.92
`vllm-gen` runs at `--gpu-memory-utilization 0.43` but actually holds ~45.6 GiB