vh
91bda3c480
The box is physically at Fountain Valley, renamed, renumbered onto 10.251/16, and serving inference again. This lands the repo half of that. Host: hostname ana-ml2 -> fv-ml1, pinned to 10.251.50.54 by a dnsmasq reservation so the address the runbook, DNS and LiteLLM all assume is the address it actually has. Its headscale node is renamed too. The sweep ran from scripts/fv-ml1-rename-sweep.sh, whose allowlist is the reason this diff touches current-state files and not the record. Dated persistent-memory entries, archival-memory and incident notes still say ana-ml2 in 31 and 62 places respectively, because that is what the box was when those things happened. Rewriting them would make the history lie. LiteLLM was the load-bearing piece and needed more than the api_base sed the runbook describes. Twenty api_base entries repointed, but a grep-and-verify pass also caught a LIVE pass_through_endpoints target for the scalar-judge reward route still on the old address -- an api_base-only substitution would have left it dead. Four prose references describing current state were repointed as well; one historical note recording where a hand-test was run is deliberately left pointing at 10.250.50.54. Two facts in the server tables were wrong and are corrected here. The site is Fountain Valley, not Anaheim. And the box has FOUR RTX PRO 6000 Blackwell Max-Q, not two -- verified by nvidia-smi -L and independently by PCI enumeration of four GB202GL devices. That is 391 GB of VRAM rather than 196, which changes what fits on it. DNS: fv-ml1, fv-ml1-bmc and fv-gw added under the fv site via the piggyback approach, scriberr re-homed, and the ana-ml2 records removed. Applied to all three resolvers. The BMC record carries a warning that its 802.1q VLAN tag must stay disabled -- it shipped tagging VLAN 250 into an untagged port, which made it invisible to every network-side diagnostic and is the reason it appeared dead through several cable changes. Verified end to end: summarizer and sec both answer through the Anaheim gateway across the mesh to FV seats on different ports.
261 lines
13 KiB
Markdown
261 lines
13 KiB
Markdown
# gemma4-charrp — the char-rp seat (fv-ml1 GPU0)
|
||
|
||
`google/gemma-4-26B-A4B-it`, NVFP4, serving both halves of the char-rp pair on
|
||
`:8016`. Replaced the dense `G4-MeroMero-v2-31B-NVFP4A16` seat on 2026-08-24.
|
||
|
||
```
|
||
char-rp non-thinking -> http://10.251.50.54:8016/v1
|
||
char-rp-reasoning thinking -> http://10.251.50.54:8016/v1
|
||
```
|
||
|
||
Two LiteLLM aliases, **one backend**. They are not two seats — this trips people
|
||
up, and it cost a peer a mis-attributed benchmark before it was noticed.
|
||
|
||
## Three model directories, and they are not interchangeable
|
||
|
||
| path | size | what it is |
|
||
|---|---|---|
|
||
| `gemma4-26b-a4b-it-bf16` | 49 GB | **Stock BF16.** Unquantized. Cannot be served here — 48.10 GiB of weights against ~49 GiB of free GPU0 leaves nothing for KV cache. Its `chat_template.jinja` is the canonical upstream one; see below. |
|
||
| `gemma4-26b-a4b-it-heretic-bf16` | 49 GB | **QLoRA trainee base** (operator's pick, 2026-08-24). llmfan46, Heretic v1.2.0 ARA, KL 0.1237, refusals 3/100. |
|
||
| `gemma4-26b-a4b-it-abliterated-bf16` | 49 GB | **Trainee alternate.** TrevorJS, KL 0.09, refusals 1/100 effective and 5/686 cross-dataset — lowest measured damage of the field. |
|
||
| `gemma4-26b-a4b-it-nvfp4` | 16 GB | **What is served.** RedHatAI, compressed-tensors, W4**A4**. |
|
||
| `gemma4-26b-a4b-it-nvfp4a16` | 17 GB | **Activation-axis control**, for benching only. prithivMLmods, compressed-tensors, W4**A16**. |
|
||
|
||
All under `/tank/aimodels/`, pulled by the `gemma4-26b-*-dl.py` scripts beside
|
||
them with revisions pinned.
|
||
|
||
### ⚠ Third-party Gemma-4 builds ship STALE CHAT TEMPLATES — this is endemic
|
||
|
||
Verified by hash on 2026-08-24 across every third-party derivative pulled here.
|
||
**Not one of them ships upstream's template:**
|
||
|
||
| build | lines | sha256 (normalised) |
|
||
|---|---|---|
|
||
| upstream `google/gemma-4-26B-A4B-it` | 390 | `6a1015c47ccfcfa6` |
|
||
| RedHatAI NVFP4 (what is served) | 389 | `6a1015c47ccfcfa6` — the only match |
|
||
| llmfan46 heretic | 365 | `0a52be69cda5ab8a` |
|
||
| TrevorJS abliterated | 266 | `58c66fdee4afa297` |
|
||
| jenerallee78 abliterated | 266 | `58c66fdee4afa297` |
|
||
| prithivMLmods NVFP4A16 | 266 | `58c66fdee4afa297` |
|
||
|
||
Three independent repos carrying the identical stale 266-line file means it
|
||
propagated through the ecosystem, not that one packager slipped.
|
||
|
||
**Consequences differ by use and both are silent:**
|
||
|
||
- **Serving** — a different template renders a different prompt. This is why the
|
||
production compose pins the template explicitly.
|
||
- **Training** — if the harness renders examples through `base/chat_template.jinja`,
|
||
you train on a different prompt format than production serves. Train/serve
|
||
skew, no error, and it presents as a tuning failure.
|
||
|
||
For both, point at the upstream file:
|
||
`/tank/aimodels/gemma4-26b-a4b-it-bf16/chat_template.jinja`.
|
||
|
||
### Measured: abliteration is close to free on this base (2026-08-24)
|
||
|
||
Isolated properly — stock BF16 against llmfan46 BF16, same precision, same
|
||
pinned upstream template, same 192 items, CoT off. **Abliteration was the only
|
||
axis that moved.**
|
||
|
||
| task | stock BF16 | heretic BF16 | items |
|
||
|---|---|---|---|
|
||
| T1 state / T3 constraint / T4 long-context / T5 control | 100% | 100% | 0 |
|
||
| T2 contradiction | 75% | 59% | **−5** |
|
||
| T6 spatial | 75% | 88% | **+4** |
|
||
| **core** | **90.0%** | **89.4%** | −0.6 pts |
|
||
|
||
**Net cost 0.6 points — but it MOVED capability rather than removing it.** Five
|
||
items lost on contradiction detection, four gained on spatial composition,
|
||
nearly cancelling. Nobody predicted a gain anywhere, least of all that
|
||
direction.
|
||
|
||
Consequence for the base choice: **llmfan46 stands.** There is no case for
|
||
re-staging on TrevorJS at KL 0.09 over a 0.6-point net difference — the KL gap
|
||
between the two builds is smaller than the gap this measurement failed to find.
|
||
|
||
⚠ Read those as **~5 items and ~4 items at n=32**, not as −15.6/+12.5 percent.
|
||
The percentages read more precisely than the measurement supports, and only
|
||
marginals were run — no paired per-item analysis.
|
||
|
||
⚠ **This says nothing about quantization.** Stock BF16 scores T2 75% where stock
|
||
NVFP4 scored 94%, but those runs were n=32 and n=16 — different item counts mean
|
||
different item sets, and the extra items are not guaranteed equally easy. That
|
||
comparison is n-confounded and is not being made.
|
||
|
||
### Choosing an abliterated base — compare on published damage, not on names
|
||
|
||
"Low damage" has a measurable proxy and the field spreads widely on it:
|
||
|
||
| build | method | KL | refusals |
|
||
|---|---|---|---|
|
||
| TrevorJS | ARA-family | **0.09** | 1/100 effective, 5/686 cross-dataset, manually audited |
|
||
| llmfan46 | Heretic v1.2.0 ARA | 0.1237 | 3/100 |
|
||
| jenerallee78 | ARA 2-pass | 0.1299 | 7.7% StrongREJECT |
|
||
| huihui-ai | remove-refusals-with-transformers | none published | none published |
|
||
| trohrbaugh/heretic-ara | — | 0.2999 | 31.4% |
|
||
| coder3101/heretic | — | 0.4118 | 15.8% |
|
||
|
||
Fleet anchor for reading those numbers: our own abliteration work found **Heretic
|
||
at KL 0.12 preserved the MTP head at 83.7% acceptance**. Both staged builds sit
|
||
at or below that, so neither is an extrapolation past what has been measured
|
||
here.
|
||
|
||
huihui-ai is rejected on this stack despite being the best-known abliteration
|
||
house: no published metrics, its card describes the method as "a crude,
|
||
proof-of-concept implementation", it states both thinking and non-thinking modes
|
||
were "completely abliterated", and its parameter count is 26,544,131,376 against
|
||
upstream's 25,805,936,206 — roughly 738M unexplained extra. The operator's
|
||
independent read is the same ("huihui produces garbage").
|
||
|
||
## GPU0 is shared and the budgets must sum under ~0.92
|
||
|
||
`vllm-gen` runs at `--gpu-memory-utilization 0.43` but actually holds ~45.6 GiB
|
||
of the 94.97 GiB card — that flag sizes the KV cache and does **not** cover CUDA
|
||
context, graphs and non-torch overhead. The predecessor seat sat at 0.51, the
|
||
pair summed to 0.94, and on 2026-08-24 it stopped fitting and crash-looped 13
|
||
times with `torch.OutOfMemoryError: ... 195.19 MiB is free`.
|
||
|
||
This seat runs 0.47. Raising it means lowering gen's in the same change, and
|
||
**check the real numbers, not the flags**:
|
||
|
||
```bash
|
||
nvidia-smi --query-compute-apps=pid,used_memory --format=csv
|
||
```
|
||
|
||
There is no room for a second concurrent seat on this card. That is why the A16
|
||
control below is a swap rather than a parallel deployment.
|
||
|
||
## Running the A16 activation-axis control
|
||
|
||
Why it was run, and what it actually settled: a 2026-08-24 battery appeared to
|
||
show a large contradiction-detection deficit with CoT off, which had the exact
|
||
shape 4-bit input activations would produce on the most reasoning-dense task.
|
||
The control ran and found activation precision **close to free** — every other
|
||
task identical across the two builds. That quantization result stands.
|
||
|
||
⚠ The *deficit* it was chasing does not — the benchmark item was ill-posed. See
|
||
"Superseded claims" at the end of this file. The procedure below is kept because
|
||
the flip is a genuinely useful capability, not because the original question was
|
||
sound.
|
||
|
||
⚠ **The A16 build ships a STALE CHAT TEMPLATE and the run MUST override it.**
|
||
Verified by hash 2026-08-24: upstream `google/gemma-4-26B-A4B-it` is 390 lines,
|
||
the A4 build's is 389 and byte-identical to upstream once trailing newlines are
|
||
normalised, the A16 build's is 266 and is not. Upstream and A4 open the thinking
|
||
path with `{%- set enable_thinking = enable_thinking | default(false) -%}`; the
|
||
A16 template has no such set. Its `tokenizer_config.json` `response_schema` also
|
||
lacks the `thinking` property. Served with its own template the two arms render
|
||
**different prompts**, and a score delta could be the template rather than the
|
||
activations.
|
||
|
||
Overriding is safe: the tokenizers agree — vocab identical at 262,144 entries,
|
||
`added_tokens` identical — so the same template over the same vocab renders the
|
||
same token ids. brokkr independently diffed every non-quantization config field
|
||
of both builds against the upstream BF16 and found only `transformers_version`
|
||
differing, which is a save-time library version rather than a model property.
|
||
Residual risk stated honestly on both sides: config identity is not weight
|
||
identity, and nobody has done a dequantization pass.
|
||
|
||
The procedure, ~2 minutes each way:
|
||
|
||
1. Add the template flag to `compose.yaml` (a **no-op for production** — the A4
|
||
build ships this exact file, so pinning it explicitly changes nothing and
|
||
guards against precisely the staleness above):
|
||
|
||
```yaml
|
||
- --chat-template
|
||
- ${GEMMA4_CHAT_TEMPLATE:-/tank/aimodels/gemma4-26b-a4b-it-nvfp4/chat_template.jinja}
|
||
```
|
||
|
||
⚠ If `GEMMA4_MODEL` is ever pointed at a *different checkpoint*, this default
|
||
must move with it. A pinned template is only correct for the checkpoint it
|
||
came from — that is the mistake the outgoing MeroMero seat's hand-patched
|
||
template was warning about, inverted.
|
||
|
||
2. `GEMMA4_MODEL=/tank/aimodels/gemma4-26b-a4b-it-nvfp4a16` in the host `.env`,
|
||
then `docker compose up -d --force-recreate vllm-gemma4-charrp`.
|
||
3. Tell brokkr; they fire one CoT-off arm, 96 items, under a minute of wall
|
||
clock, and report back.
|
||
4. Revert `GEMMA4_MODEL` to the `nvfp4` path and recreate.
|
||
|
||
Only the **CoT-off** arm is worth running. With thinking on the model already
|
||
scores 100% on every task it completes, and a ceiling cannot move.
|
||
|
||
⚠ Displacing this seat is an operator decision, not a routine one. It is live.
|
||
|
||
## Rollback to MeroMero-v2
|
||
|
||
`stacks/meromero-charrp/` is retained stopped in `created` state, labelled
|
||
`AI - Dormant`. Both stacks bind `:8016`, so rollback is **stop-then-start**:
|
||
|
||
```bash
|
||
cd /opt/docker/compose/gemma4-charrp && docker compose stop vllm-gemma4-charrp
|
||
cd /opt/docker/compose/meromero-charrp && docker compose up -d vllm-meromero-rp
|
||
```
|
||
|
||
## Gemma-4 flags that are load-bearing
|
||
|
||
Carried over from the MeroMero seat because they are architecture-level, not
|
||
checkpoint-level:
|
||
|
||
- `--tool-call-parser gemma4` + `--enable-auto-tool-choice` — Gemma-4 emits its
|
||
own native tool syntax, not the qwen3_coder XML the other seats use. Without
|
||
these, any tools-bearing request 400s outright.
|
||
- `--reasoning-parser gemma4` — absorbs the `<|channel>` thought markers; without
|
||
it they leak into `content` verbatim on the post-tool turn (vllm #45834).
|
||
- `--default-chat-template-kwargs '{"enable_thinking": false}'` — **mandatory
|
||
companion to the reasoning parser.** The parser defaults `enable_thinking` to
|
||
TRUE, which pre-initialises the engine to REASONING, so all plain prose lands
|
||
in `reasoning_content` with a null `content` and every char-rp consumer breaks.
|
||
Production runs thinking off; the thinking route reaches it per-request.
|
||
|
||
## Known defect, not ours
|
||
|
||
With thinking on, the model does not reliably terminate on constraint-following.
|
||
brokkr measured **32 of 96 calls truncating at a 12,000-token cap, with all 16
|
||
constraint items among them**, and inspected the traces: the reasoning is sound
|
||
right up to the point it fails to stop. A non-termination defect, not a
|
||
wrong-answer one. `VLLM_USE_V2_MODEL_RUNNER=0` enables
|
||
`thinking_token_budget` and is the first lever to try if a usable thinking seat
|
||
is ever wanted — deliberately **not** applied here, since it costs the faster
|
||
model runner for a mode production does not serve.
|
||
|
||
## Superseded claims
|
||
|
||
Following the repo convention for quant work: when a recorded claim turns out
|
||
wrong, it gets a dated row rather than a silent edit, so older notes elsewhere
|
||
stop misleading people.
|
||
|
||
| date | claim, as recorded | status |
|
||
|---|---|---|
|
||
| 2026-08-24 | "Gemma-4-26B-A4B-it scores 12% on contradiction detection with CoT off, against gen's 81% — the model owns the deficit." | **RETRACTED same day.** The benchmark item was ill-posed: it presented two mutually contradicting statements and asked for "the contradicting statement", but **contradiction is symmetric** — neither was more the contradicting one, and the model consistently named the absolute claim, a defensible reading the labelling scored wrong every time. |
|
||
| 2026-08-24 | "Domain tuning costs 43 points of contradiction detection." | **RETRACTED.** Rested entirely on the same item. On the corrected instrument the effect does not shrink, it **reverses**. |
|
||
| 2026-08-24 | All pre-fix T2 (contradiction) numbers for Gemma-4, MeroMero-v2, `sec` and `gen`. | **VOID.** |
|
||
|
||
**The tell, worth internalising: the score was BELOW CHANCE.** 12% on a
|
||
five-option task is under the 20% floor. A below-chance score indicts the
|
||
instrument before it indicts the model, and that should be the first reaction
|
||
rather than a late one. Neither side caught it until the individual items were
|
||
read.
|
||
|
||
A second defect surfaced while fixing the first: all generators shared one RNG,
|
||
so rewriting one task reshuffled every task after it. Each task now seeds from
|
||
its own name.
|
||
|
||
**What survived, and it is not nothing:** the A16 control result holds —
|
||
activation precision is close to free on this workload, with every other task
|
||
identical across the W4A4 and W4A16 builds. The two staging confounds caught
|
||
before the run (the misnamed A16 repos, the stale chat template) were real and
|
||
independent of the item defect. The corrected picture, CoT off, n=96:
|
||
|
||
| arm | core | T2 | T6 | median latency |
|
||
|---|---|---|---|---|
|
||
| `char-rp` (Gemma-4) | 92.5% | **94%** | 69% | 0.24s |
|
||
| `sec` | 92.5% | 81% | 81% | 1.11s |
|
||
| `gen` | 86.2% | 50% | 81% | 0.33s |
|
||
|
||
Gemma leads on the very axis it was suspected of failing. Its actual weak axis
|
||
with thinking off is **T6, spatial composition at 69%** — and an earlier run
|
||
showed CoT-on takes T6 to 100%, which nothing else benched has managed.
|