esh-pfi-infrastructure/stacks/gemma4-charrp/README.md

# gemma4-charrp — the char-rp seat (ana-ml2 GPU0)

`google/gemma-4-26B-A4B-it`, NVFP4, serving both halves of the char-rp pair on
`:8016`. Replaced the dense `G4-MeroMero-v2-31B-NVFP4A16` seat on 2026-08-24.

```
char-rp            non-thinking   ->  http://10.250.50.54:8016/v1
char-rp-reasoning  thinking       ->  http://10.250.50.54:8016/v1
```

Two LiteLLM aliases, **one backend**. They are not two seats — this trips people
up, and it cost a peer a mis-attributed benchmark before it was noticed.

## Three model directories, and they are not interchangeable

| path | size | what it is |
|---|---|---|
| `gemma4-26b-a4b-it-bf16` | 49 GB | **Stock BF16.** Unquantized. Cannot be served here — 48.10 GiB of weights against ~49 GiB of free GPU0 leaves nothing for KV cache. Its `chat_template.jinja` is the canonical upstream one; see below. |
| `gemma4-26b-a4b-it-heretic-bf16` | 49 GB | **QLoRA trainee base** (operator's pick, 2026-08-24). llmfan46, Heretic v1.2.0 ARA, KL 0.1237, refusals 3/100. |
| `gemma4-26b-a4b-it-abliterated-bf16` | 49 GB | **Trainee alternate.** TrevorJS, KL 0.09, refusals 1/100 effective and 5/686 cross-dataset — lowest measured damage of the field. |
| `gemma4-26b-a4b-it-nvfp4` | 16 GB | **What is served.** RedHatAI, compressed-tensors, W4**A4**. |
| `gemma4-26b-a4b-it-nvfp4a16` | 17 GB | **Activation-axis control**, for benching only. prithivMLmods, compressed-tensors, W4**A16**. |

All under `/tank/aimodels/`, pulled by the `gemma4-26b-*-dl.py` scripts beside
them with revisions pinned.

### ⚠ Third-party Gemma-4 builds ship STALE CHAT TEMPLATES — this is endemic

Verified by hash on 2026-08-24 across every third-party derivative pulled here.
**Not one of them ships upstream's template:**

| build | lines | sha256 (normalised) |
|---|---|---|
| upstream `google/gemma-4-26B-A4B-it` | 390 | `6a1015c47ccfcfa6` |
| RedHatAI NVFP4 (what is served) | 389 | `6a1015c47ccfcfa6` — the only match |
| llmfan46 heretic | 365 | `0a52be69cda5ab8a` |
| TrevorJS abliterated | 266 | `58c66fdee4afa297` |
| jenerallee78 abliterated | 266 | `58c66fdee4afa297` |
| prithivMLmods NVFP4A16 | 266 | `58c66fdee4afa297` |

Three independent repos carrying the identical stale 266-line file means it
propagated through the ecosystem, not that one packager slipped.

**Consequences differ by use and both are silent:**

- **Serving** — a different template renders a different prompt. This is why the
  production compose pins the template explicitly.
- **Training** — if the harness renders examples through `base/chat_template.jinja`,
  you train on a different prompt format than production serves. Train/serve
  skew, no error, and it presents as a tuning failure.

For both, point at the upstream file:
`/tank/aimodels/gemma4-26b-a4b-it-bf16/chat_template.jinja`.

### Measured: abliteration is close to free on this base (2026-08-24)

Isolated properly — stock BF16 against llmfan46 BF16, same precision, same
pinned upstream template, same 192 items, CoT off. **Abliteration was the only
axis that moved.**

| task | stock BF16 | heretic BF16 | items |
|---|---|---|---|
| T1 state / T3 constraint / T4 long-context / T5 control | 100% | 100% | 0 |
| T2 contradiction | 75% | 59% | **−5** |
| T6 spatial | 75% | 88% | **+4** |
| **core** | **90.0%** | **89.4%** | −0.6 pts |

**Net cost 0.6 points — but it MOVED capability rather than removing it.** Five
items lost on contradiction detection, four gained on spatial composition,
nearly cancelling. Nobody predicted a gain anywhere, least of all that
direction.

Consequence for the base choice: **llmfan46 stands.** There is no case for
re-staging on TrevorJS at KL 0.09 over a 0.6-point net difference — the KL gap
between the two builds is smaller than the gap this measurement failed to find.

⚠ Read those as **~5 items and ~4 items at n=32**, not as −15.6/+12.5 percent.
The percentages read more precisely than the measurement supports, and only
marginals were run — no paired per-item analysis.

⚠ **This says nothing about quantization.** Stock BF16 scores T2 75% where stock
NVFP4 scored 94%, but those runs were n=32 and n=16 — different item counts mean
different item sets, and the extra items are not guaranteed equally easy. That
comparison is n-confounded and is not being made.

### Choosing an abliterated base — compare on published damage, not on names

"Low damage" has a measurable proxy and the field spreads widely on it:

| build | method | KL | refusals |
|---|---|---|---|
| TrevorJS | ARA-family | **0.09** | 1/100 effective, 5/686 cross-dataset, manually audited |
| llmfan46 | Heretic v1.2.0 ARA | 0.1237 | 3/100 |
| jenerallee78 | ARA 2-pass | 0.1299 | 7.7% StrongREJECT |
| huihui-ai | remove-refusals-with-transformers | none published | none published |
| trohrbaugh/heretic-ara | — | 0.2999 | 31.4% |
| coder3101/heretic | — | 0.4118 | 15.8% |

Fleet anchor for reading those numbers: our own abliteration work found **Heretic
at KL 0.12 preserved the MTP head at 83.7% acceptance**. Both staged builds sit
at or below that, so neither is an extrapolation past what has been measured
here.

huihui-ai is rejected on this stack despite being the best-known abliteration
house: no published metrics, its card describes the method as "a crude,
proof-of-concept implementation", it states both thinking and non-thinking modes
were "completely abliterated", and its parameter count is 26,544,131,376 against
upstream's 25,805,936,206 — roughly 738M unexplained extra. The operator's
independent read is the same ("huihui produces garbage").

## GPU0 is shared and the budgets must sum under ~0.92

`vllm-gen` runs at `--gpu-memory-utilization 0.43` but actually holds ~45.6 GiB
of the 94.97 GiB card — that flag sizes the KV cache and does **not** cover CUDA
context, graphs and non-torch overhead. The predecessor seat sat at 0.51, the
pair summed to 0.94, and on 2026-08-24 it stopped fitting and crash-looped 13
times with `torch.OutOfMemoryError: ... 195.19 MiB is free`.

This seat runs 0.47. Raising it means lowering gen's in the same change, and
**check the real numbers, not the flags**:

```bash
nvidia-smi --query-compute-apps=pid,used_memory --format=csv
```

There is no room for a second concurrent seat on this card. That is why the A16
control below is a swap rather than a parallel deployment.

## Running the A16 activation-axis control

Why it was run, and what it actually settled: a 2026-08-24 battery appeared to
show a large contradiction-detection deficit with CoT off, which had the exact
shape 4-bit input activations would produce on the most reasoning-dense task.
The control ran and found activation precision **close to free** — every other
task identical across the two builds. That quantization result stands.

⚠ The *deficit* it was chasing does not — the benchmark item was ill-posed. See
"Superseded claims" at the end of this file. The procedure below is kept because
the flip is a genuinely useful capability, not because the original question was
sound.

⚠ **The A16 build ships a STALE CHAT TEMPLATE and the run MUST override it.**
Verified by hash 2026-08-24: upstream `google/gemma-4-26B-A4B-it` is 390 lines,
the A4 build's is 389 and byte-identical to upstream once trailing newlines are
normalised, the A16 build's is 266 and is not. Upstream and A4 open the thinking
path with `{%- set enable_thinking = enable_thinking | default(false) -%}`; the
A16 template has no such set. Its `tokenizer_config.json` `response_schema` also
lacks the `thinking` property. Served with its own template the two arms render
**different prompts**, and a score delta could be the template rather than the
activations.

Overriding is safe: the tokenizers agree — vocab identical at 262,144 entries,
`added_tokens` identical — so the same template over the same vocab renders the
same token ids. brokkr independently diffed every non-quantization config field
of both builds against the upstream BF16 and found only `transformers_version`
differing, which is a save-time library version rather than a model property.
Residual risk stated honestly on both sides: config identity is not weight
identity, and nobody has done a dequantization pass.

The procedure, ~2 minutes each way:

1. Add the template flag to `compose.yaml` (a **no-op for production** — the A4
   build ships this exact file, so pinning it explicitly changes nothing and
   guards against precisely the staleness above):

   ```yaml
   - --chat-template
   - ${GEMMA4_CHAT_TEMPLATE:-/tank/aimodels/gemma4-26b-a4b-it-nvfp4/chat_template.jinja}
   ```

   ⚠ If `GEMMA4_MODEL` is ever pointed at a *different checkpoint*, this default
   must move with it. A pinned template is only correct for the checkpoint it
   came from — that is the mistake the outgoing MeroMero seat's hand-patched
   template was warning about, inverted.

2. `GEMMA4_MODEL=/tank/aimodels/gemma4-26b-a4b-it-nvfp4a16` in the host `.env`,
   then `docker compose up -d --force-recreate vllm-gemma4-charrp`.
3. Tell brokkr; they fire one CoT-off arm, 96 items, under a minute of wall
   clock, and report back.
4. Revert `GEMMA4_MODEL` to the `nvfp4` path and recreate.

Only the **CoT-off** arm is worth running. With thinking on the model already
scores 100% on every task it completes, and a ceiling cannot move.

⚠ Displacing this seat is an operator decision, not a routine one. It is live.

## Rollback to MeroMero-v2

`stacks/meromero-charrp/` is retained stopped in `created` state, labelled
`AI - Dormant`. Both stacks bind `:8016`, so rollback is **stop-then-start**:

```bash
cd /opt/docker/compose/gemma4-charrp  && docker compose stop vllm-gemma4-charrp
cd /opt/docker/compose/meromero-charrp && docker compose up -d vllm-meromero-rp
```

## Gemma-4 flags that are load-bearing

Carried over from the MeroMero seat because they are architecture-level, not
checkpoint-level:

- `--tool-call-parser gemma4` + `--enable-auto-tool-choice` — Gemma-4 emits its
  own native tool syntax, not the qwen3_coder XML the other seats use. Without
  these, any tools-bearing request 400s outright.
- `--reasoning-parser gemma4` — absorbs the `<|channel>` thought markers; without
  it they leak into `content` verbatim on the post-tool turn (vllm #45834).
- `--default-chat-template-kwargs '{"enable_thinking": false}'` — **mandatory
  companion to the reasoning parser.** The parser defaults `enable_thinking` to
  TRUE, which pre-initialises the engine to REASONING, so all plain prose lands
  in `reasoning_content` with a null `content` and every char-rp consumer breaks.
  Production runs thinking off; the thinking route reaches it per-request.

## Known defect, not ours

With thinking on, the model does not reliably terminate on constraint-following.
brokkr measured **32 of 96 calls truncating at a 12,000-token cap, with all 16
constraint items among them**, and inspected the traces: the reasoning is sound
right up to the point it fails to stop. A non-termination defect, not a
wrong-answer one. `VLLM_USE_V2_MODEL_RUNNER=0` enables
`thinking_token_budget` and is the first lever to try if a usable thinking seat
is ever wanted — deliberately **not** applied here, since it costs the faster
model runner for a mode production does not serve.

## Superseded claims

Following the repo convention for quant work: when a recorded claim turns out
wrong, it gets a dated row rather than a silent edit, so older notes elsewhere
stop misleading people.

| date | claim, as recorded | status |
|---|---|---|
| 2026-08-24 | "Gemma-4-26B-A4B-it scores 12% on contradiction detection with CoT off, against gen's 81% — the model owns the deficit." | **RETRACTED same day.** The benchmark item was ill-posed: it presented two mutually contradicting statements and asked for "the contradicting statement", but **contradiction is symmetric** — neither was more the contradicting one, and the model consistently named the absolute claim, a defensible reading the labelling scored wrong every time. |
| 2026-08-24 | "Domain tuning costs 43 points of contradiction detection." | **RETRACTED.** Rested entirely on the same item. On the corrected instrument the effect does not shrink, it **reverses**. |
| 2026-08-24 | All pre-fix T2 (contradiction) numbers for Gemma-4, MeroMero-v2, `sec` and `gen`. | **VOID.** |

**The tell, worth internalising: the score was BELOW CHANCE.** 12% on a
five-option task is under the 20% floor. A below-chance score indicts the
instrument before it indicts the model, and that should be the first reaction
rather than a late one. Neither side caught it until the individual items were
read.

A second defect surfaced while fixing the first: all generators shared one RNG,
so rewriting one task reshuffled every task after it. Each task now seeds from
its own name.

**What survived, and it is not nothing:** the A16 control result holds —
activation precision is close to free on this workload, with every other task
identical across the W4A4 and W4A16 builds. The two staging confounds caught
before the run (the misnamed A16 repos, the stale chat template) were real and
independent of the item defect. The corrected picture, CoT off, n=96:

| arm | core | T2 | T6 | median latency |
|---|---|---|---|---|
| `char-rp` (Gemma-4) | 92.5% | **94%** | 69% | 0.24s |
| `sec` | 92.5% | 81% | 81% | 1.11s |
| `gen` | 86.2% | 50% | 81% | 0.33s |

Gemma leads on the very axis it was suspected of failing. Its actual weak axis
with thinking off is **T6, spatial composition at 69%** — and an earlier run
showed CoT-on takes T6 to 100%, which nothing else benched has managed.