vh
8d6a9390de
Supersedes what commit 3446367 recorded. That message stated the A16 control
"settles a question" and quoted 12% contradiction detection against gen's 81%.
The quantization half of it stands; the deficit it was measuring does not exist.
brokkr-smithy-dev retracted the finding after the operator asked to see the
individual items. The task presented two mutually contradicting statements and
asked for "the contradicting statement" — but CONTRADICTION IS SYMMETRIC.
Neither statement was more the contradicting one, the model had no way to know
which had been inserted, and it consistently named the absolute claim: a
defensible reading that the labelling scored wrong on every single item.
The tell was there and both of us walked past it: 12% on a five-option task is
BELOW THE 20% CHANCE FLOOR. A below-chance score indicts the instrument before
it indicts the model. That should be the first reaction to a below-chance
result, not a late one, and it is now written into the README as such.
Retracted: "the model owns the contradiction deficit"; "domain tuning costs 43
points of contradiction detection" (which on a sound instrument does not shrink
but REVERSES); and every pre-fix T2 number for Gemma-4, MeroMero-v2, sec and
gen. A second defect surfaced during the fix — all generators shared one RNG, so
rewriting one task reshuffled every task after it.
What survives is real and worth separating out: the A16 control result holds.
Activation precision is close to free on this workload, every other task
identical across the W4A4 and W4A16 builds. The two staging confounds caught
before the run — the two Hub repos named NVFP4A16 that declare 4-bit
activations, and the stale chat template — were independent of the item defect
and remain load-bearing. On the corrected instrument Gemma leads the very axis
it was suspected of failing (94% against sec's 81% and gen's 50%); its actual
weak axis with thinking off is spatial composition at 69%.
Recorded as a dated superseded-claims table rather than a silent edit, per the
repo's quant-work convention, so notes elsewhere stop misleading people.
67 lines
3.5 KiB
Bash
67 lines
3.5 KiB
Bash
# gemma4-charrp — char-rp seat on ana-ml2 GPU0. Real .env lives on the host.
|
|
|
|
# ⚠ GPU0 IS SHARED WITH `vllm-gen`. gen runs at --gpu-memory-utilization 0.43
|
|
# but actually holds ~45.6 GiB of the 94.97 GiB card — that flag sizes the KV
|
|
# cache and does NOT cover CUDA context, graphs and non-torch overhead. The
|
|
# predecessor seat sat at 0.51, the pair summed to 0.94, and on 2026-08-24 it
|
|
# stopped fitting and crash-looped 13 times with
|
|
# `torch.OutOfMemoryError: ... 195.19 MiB is free`.
|
|
#
|
|
# 0.47 keeps ~4.8 GiB of real margin. This model's weights are only ~15.3 GiB
|
|
# (NVFP4) against the predecessor's ~19.5 GiB, so the same budget buys MORE KV
|
|
# cache than before, not less. Raising this means lowering gen's in the same
|
|
# change — and check the real numbers, not the flags:
|
|
# nvidia-smi --query-compute-apps=pid,used_memory --format=csv
|
|
GEMMA4_GPU_MEM_UTIL=0.47
|
|
|
|
# Native context. config.json declares max_position_embeddings 262144, same as
|
|
# the outgoing seat, so this is a straight-across swap on context too.
|
|
GEMMA4_MAX_MODEL_LEN=262144
|
|
GEMMA4_MAX_NUM_SEQS=32
|
|
|
|
# ⚠ THE NVFP4 QUANT, NOT THE BF16. /tank/aimodels/gemma4-26b-a4b-it-bf16 is the
|
|
# QLoRA tuning base and is 48.10 GiB of weights — it does not fit beside gen.
|
|
GEMMA4_MODEL=/tank/aimodels/gemma4-26b-a4b-it-nvfp4
|
|
#
|
|
# ACTIVATION-AXIS CONTROL, for one-off benching — swap this line, recreate:
|
|
# GEMMA4_MODEL=/tank/aimodels/gemma4-26b-a4b-it-nvfp4a16
|
|
# Same NVFP4 weights and the same compressed-tensors loader, but 16-bit
|
|
# activations.
|
|
#
|
|
# ⚠ THE ORIGINAL MOTIVATION FOR THIS CONTROL WAS RETRACTED — see the README's
|
|
# "Superseded claims" section. The A16 run itself HELD and is the reason this
|
|
# path is documented: activation precision is close to free on this battery,
|
|
# every other task identical across builds. But the "12% contradiction
|
|
# detection" number that prompted it was an artifact of an ill-posed benchmark
|
|
# item, not a model property. Keep the path; do not repeat the reasoning.
|
|
# Two minutes down each way; port and both aliases are unchanged, so no
|
|
# consumer config moves.
|
|
#
|
|
# ⚠ Only prithivMLmods/gemma-4-26B-A4B-it-NVFP4A16 is genuinely A16.
|
|
# bg-digitalservices and ManniX-ITA both publish repos NAMED "NVFP4A16" whose
|
|
# config.json declares input_activations num_bits 4. Check the field, not the
|
|
# name, before ever substituting a different repo.
|
|
#
|
|
# ⚠⚠ THE A16 BUILD ALSO SHIPS A STALE CHAT TEMPLATE — the control run MUST
|
|
# override it or it moves two axes instead of one:
|
|
# --chat-template /tank/aimodels/gemma4-26b-a4b-it-nvfp4/chat_template.jinja
|
|
# Verified by hash on 2026-08-24: upstream google/gemma-4-26B-A4B-it is 390
|
|
# lines, the A4 build's is 389 and byte-identical to it once trailing newlines
|
|
# are normalised, and the A16 build's is 266 and is NOT. The thinking machinery
|
|
# is built differently too — upstream and A4 set
|
|
# `enable_thinking | default(false)` at line 186, the A16 template has no such
|
|
# set — and its tokenizer_config response_schema has no `thinking` property.
|
|
# It was quantized from an older revision. Served with its own template the two
|
|
# arms would render DIFFERENT PROMPTS, and a score delta could be the template
|
|
# rather than the activations.
|
|
# Safe to override because the tokenizers agree: vocab identical at 262,144
|
|
# entries, added_tokens identical. Same template over the same vocab renders
|
|
# the same token ids.
|
|
|
|
GEMMA4_PORT=8016
|
|
GEMMA4_GPU_ID=0
|
|
GEMMA4_CONTAINER=vllm-gemma4-charrp
|
|
|
|
# Same value as every other vLLM seat on this host — the gateway presents it.
|
|
API_KEY=
|